Scrapinghub Platform
The most advanced platform for deploying and running web crawlers
Scrapy Cloud
The most powerful platform to run your web crawlers
Scrapy Cloud is a battle-tested platform for running web crawlers (aka. spiders). Your spiders run in the cloud and scale on demand, from thousands to billion of pages. Think of it as a Heroku for web crawling.
Features
Code your spiders
Write your spiders using Scrapy, the most powerful open source web crawling framework
Deploy to the cloud
With a single command or the push of a button. No servers involved
Full API access
To integrate and build your apps with Scrapinghub data
Manage your spiders
Unearth actionable insights. We’re able to filter, normalize, augment, analyze, and aggregate your data.
Watch them run
Watch spiders as they run and scrape data, compare and annotate the data scraped
Manage resources wisely
Run heavy jobs with more memory and lighter jobs with more concurrency
Download the data
Dowload your data in JSON, CSV or XML formats
...or share with the world
You can choose to share the data by publishing a dataset
Pricing
Scrapy cloud is forever free
- Unlimited team members
- Unlimited projects
- Unlimited requests
- 1 concurrent crawl
- 7 day data retention
- No credit card required
Elastic pricing
One of Scrapy Cloud key features is its elastic capacity. You can purchase capacity units (essentially, 1 GB RAM each) when you need to scale up. You can distribute these units intelligenly, by having heavy spiders require more units to run and lighter spiders less. This way, the capacity will be optimized to the size of your spiders.
When you purchase any unit, you also get:
- Unlimited job runtime
- 120 days of data retention (vs 7 days in the free tier)
- Personalized support
