at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial pipelines file. tutorial/settings.py: the project’s settings file. tutorial/spiders/: a directory where you’ll later put your spiders. Defining our Item Items are containers that will be loaded with the

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial 2.3. Scrapy Tutorial 11 Scrapy Documentation, Release 0.14.4 • tutorial/spiders/: a directory where you’ll later put your spiders. 2.3.2 Defining our Item Items are containers that will be loaded

org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: /tor/\d+. For extracting data we’ll use XPath to select the part of the document where the data system site-packages directory like this: ln -s /path/to/scrapy-trunk/scrapy SITE-PACKAGES/scrapy Where SITE-PACKAGES is the location of your system site-packages directory. To find this out execute the Creating a project Before start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: python scrapy-ctl.py startproject dmoz This will create

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: For extracting data we’ll use XPath [http://www.w3.org/TR/xpath] to select the part of the document where the data is to be extracted. Let’s take one of those torrent pages: http://www.mininova.org/tor/2657665 system site-packages directory like this: ln -s /path/to/scrapy-trunk/scrapy SITE-PACKAGES/scrapy Where SITE-PACKAGES is the location of your system site-packages directory. To find this out execute the

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial pipelines file. • tutorial/settings.py: the project’s settings file. • tutorial/spiders/: a directory where you’ll later put your spiders. 2.3.2 Defining our Item Items are containers that will be loaded

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject dmoz This will create a dmoz directory project’s pipelines file. • dmoz/settings.py: the project’s settings file. • dmoz/spiders/: a directory where you’ll later put your spiders. 2.3. Scrapy Tutorial 11 Scrapy Documentation, Release 0.12.0 2.3

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject dmoz This will create a dmoz directory project’s pipelines file. dmoz/settings.py: the project’s settings file. dmoz/spiders/: a directory where you’ll later put your spiders. Defining our Item Items are containers that will be loaded with the

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial pipelines file. tutorial/settings.py: the project’s settings file. tutorial/spiders/: a directory where you’ll later put your spiders. Defining our Item Items are containers that will be loaded with the

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial Release 0.20.2 • tutorial/settings.py: the project’s settings file. • tutorial/spiders/: a directory where you’ll later put your spiders. 2.3.2 Defining our Item Items are containers that will be loaded

at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer. We’ll use that to construct the regular expression for the links to follow: project Before you start scraping, you will have set up a new Scrapy project. Enter a directory where you’d like to store your code and then run: scrapy startproject tutorial This will create a tutorial Release 0.18.4 • tutorial/settings.py: the project’s settings file. • tutorial/spiders/: a directory where you’ll later put your spiders. 2.3.2 Defining our Item Items are containers that will be loaded