follow- ing links and the rules for extracting the data from pages. If we take a look at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer interesting, as our parse method instructs, two files have been created: Books and Resources, with the content of both URLs. What just happened under the hood? Scrapy creates scrapy.Request objects for each . html content here ... ] $ scrapy fetch --nolog --headers http://www.example.com/ {'Accept-Ranges': ['bytes'], 'Age': ['1263 '], 'Connection': ['close '], 'Content-Length': ['596'], 'Content-Type': ['text/html;

JSON format, containing the title, link, number of upvotes, a list of the tags and the question content in HTML, looking like this (reformatted for easier reading): [{ "body": "... LONG HTML HERE .. directory. You should notice two new files have been created: Books.html and Resources.html, with the content for the respective URLs, as our parse method instructs. 12 Chapter 2. First steps Scrapy Documentation However, XPath offers more power because besides navigating the structure, it can also look at the content: you’re able to select things like: the link that contains the text ‘Next Page’. Because of this

following links and the rules for extracting the data from pages. If we take a look at that page content we’ll see that all torrent URLs are like http://www.mininova.org/tor/NUMBER where NUMBER is an integer interesting, as our parse method instructs, two files have been created: Books and Resources, with the content of both URLs. What just happened under the hood? Scrapy creates scrapy.Request objects for each html content here ... ] $ scrapy fetch --nolog --headers http://www.example.com/ {'Accept-Ranges': ['bytes'], 'Age': ['1263 '], 'Connection': ['close '], 'Content-Length': ['596'], 'Content-Type':

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it. The parse() method usually parses the response, extracting You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. 12 Chapter 2. First steps Scrapy Documentation

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it. 12 Chapter 2. First steps Scrapy Documentation, Release You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. Note: If you are wondering why we haven’t

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful 2.3. Scrapy Tutorial 11 Scrapy Documentation, Release 1.1.3 methods to You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. Note: If you are wondering why we haven’t

JSON format, containing the title, link, number of upvotes, a list of the tags and the question content in HTML, looking like this (reformatted for easier reading): [{ "body": "... LONG HTML HERE directory. You should notice two new files have been created: Books.html and Resources.html, with the content for the respective URLs, as our parse method instructs. What just happened under the hood? Scrapy However, XPath offers more power because besides navigating the structure, it can also look at the content: you’re able to select things like: the link that contains the text ‘Next Page’. Because of this

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it. The parse() method usually parses the response, extracting You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. Note: If you are wondering why we haven’t

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it. The parse() method usually parses the response, extracting You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. Note: If you are wondering why we haven’t

requests to make, optionally how to follow links in the pages, and how to parse the downloaded page content to extract data. This is the code for our first Spider. Save it in a file named quotes_spider.py of the requests made. The response parameter is an instance of TextResponse that holds the page content and has further helpful methods to handle it. The parse() method usually parses the response, extracting You should notice that two new files have been created: quotes-1.html and quotes-2.html, with the content for the respective URLs, as our parse method instructs. 12 Chapter 2. First steps Scrapy Documentation