called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item extract()) print ’Link number %d points to url %s and image %s’ % args Link number 0 points to url [u’image1.html’] and image [u’image1_thumb.jpg’] Link number 1 points to url [u’image2.html’] and image jpg’] Link number 2 points to url [u’image3.html’] and image [u’image3_thumb.jpg’] Link number 3 points to url [u’image4.html’] and image [u’image4_thumb.jpg’] Link number 4 points to url [u’image5.html’]

called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item extract()) print ’Link number %d points to url %s and image %s’ % args Link number 0 points to url [u’image1.html’] and image [u’image1_thumb.jpg’] Link number 1 points to url [u’image2.html’] and image jpg’] Link number 2 points to url [u’image3.html’] and image [u’image3_thumb.jpg’] Link number 3 points to url [u’image4.html’] and image [u’image4_thumb.jpg’] Link number 4 points to url [u’image5.html’]

called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item extract()) print 'Link number %d points to url %s and image %s' % args Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg'] Link number 1 points to url [u'image2.html'] and image jpg'] Link number 2 points to url [u'image3.html'] and image [u'image3_thumb.jpg'] Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg'] Link number 4 points to url [u'image5.html']

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a Scrapy provides Selector class and convenient shortcuts to avoid instantiating selectors yourself every time you need to select something from a response. You can see selectors as objects that represent nodes setup.py entry points Note: This is an experimental feature, use with caution. You can also add Scrapy commands from an external library by adding a scrapy.commands section in the entry points of the library

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a overwriting its contents. If you run this command twice without removing the file before the second time, you’ll end up with a broken JSON file. You can also used other formats, like JSON Lines: scrapy examples and patterns Here is another spider that illustrates callbacks and following links, this time for scraping author information: import scrapy class AuthorSpider(scrapy.Spider): name = 'author'

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a overwriting its contents. If you run this command twice without removing the file before the second time, you’ll end up with a broken JSON file. You can also used other formats, like JSON Lines: scrapy examples and patterns Here is another spider that illustrates callbacks and following links, this time for scraping author information: import scrapy class AuthorSpider(scrapy.Spider): name = 'author'

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a overwriting its contents. If you run this command twice without removing the file before the second time, you’ll end up with a broken JSON file. You can also used other formats, like JSON Lines: scrapy examples and patterns Here is another spider that illustrates callbacks and following links, this time for scraping author information: import scrapy class AuthorSpider(scrapy.Spider): name = 'author'

called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item extract()) print 'Link number %d points to url %s and image %s' % args Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg'] Link number 1 points to url [u'image2.html'] and image jpg'] Link number 2 points to url [u'image3.html'] and image [u'image3_thumb.jpg'] Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg'] Link number 4 points to url [u'image5.html']

called for each result (item or request) returned by the spider, and it’s intended to perform any last time processing required before returning the results to the framework core, for example setting the item extract()) print 'Link number %d points to url %s and image %s' % args Link number 0 points to url [u'image1.html'] and image [u'image1_thumb.jpg'] Link number 1 points to url [u'image2.html'] and image jpg'] Link number 2 points to url [u'image3.html'] and image [u'image3_thumb.jpg'] Link number 3 points to url [u'image4.html'] and image [u'image4_thumb.jpg'] Link number 4 points to url [u'image5.html']

While this enables you to do very fast crawls (sending multiple concurrent requests at the same time, in a fault-tolerant way) Scrapy also gives you control over the politeness of the crawl through a Scrapy provides Selector class and convenient shortcuts to avoid instantiating selectors yourself every time you need to select something from a response. You can see selectors as objects that represent nodes setup.py entry points Note This is an experimental feature, use with caution. You can also add Scrapy commands from an external library by adding a scrapy.commands section in the entry points of the library