Services [http://aws.amazon.com/associates/]) or as a general purpose web crawler. The purpose of this document is to introduce you to the concepts behind Scrapy so you can get an idea of how it works and decide /tor/\d+. For extracting data we’ll use XPath [http://www.w3.org/TR/xpath] to select the part of the document where the data is to be extracted. Let’s take one of those torrent pages: http://www.mininova.org/tor/2657665 using pickle [http://docs.python.org/library/pickle.html]: import pickle class StoreItemPipeline(object): def process_item(self, spider, item): torrent_id = item['url'].split('/')[-1]

(such as Amazon Associates Web Services) or as a general purpose web crawler. The purpose of this document is to introduce you to the concepts behind Scrapy so you can get an idea of how it works and decide for the links to follow: /tor/\d+. For extracting data we’ll use XPath to select the part of the document where the data is to be extracted. Let’s take one of those torrent pages: http://www.mininova.org/tor/2657665 serializes and stores the extracted item into a file using pickle: import pickle class StoreItemPipeline(object): def process_item(self, spider, item): torrent_id = item['url'].split('/')[-1] f = open("torrent-%s

(such as Amazon Associates Web Services) or as a general purpose web crawler. The purpose of this document is to introduce you to the concepts behind Scrapy so you can get an idea of how it works and decide start URLs. • parse() is a method of the spider, which will be called with the downloaded Response object of each start URL. The response is passed to the method as the first and only argument. This method meanings: • /html/head/title: selects the