though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth print the response’s HTTP headers instead of the response’s body • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them) Usage examples: $ scrapy fetch --nolog http://www.example.com/some/page

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth print the response’s HTTP headers instead of the response’s body • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them) Usage examples: $ scrapy fetch --nolog http://www.example.com/some/page

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth spider’s attributes. Note: Even if an HTTPS URL is specified, the protocol used in start_urls is always HTTP. This is a known issue: issue 3553. Usage example: $ scrapy genspider -l Available templates: basic

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth spider’s attributes. Note: Even if an HTTPS URL is specified, the protocol used in start_urls is always HTTP. This is a known issue: issue 3553. Usage example: $ scrapy genspider -l Available templates: basic

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth print the response’s HTTP headers instead of the response’s body • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them) Usage examples: $ scrapy fetch --nolog http://www.example.com/some/page

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth print the response’s HTTP headers instead of the response’s body • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them) Usage examples: $ scrapy fetch --nolog http://www.example.com/some/page

though Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through Wide range of built-in extensions and middlewares for handling: – cookies and session handling – HTTP features like compression, authentication, caching – user-agent spoofing – robots.txt – crawl depth print the response’s HTTP headers instead of the response’s body • --no-redirect: do not follow HTTP 3xx redirects (default is to follow them) Usage examples: $ scrapy fetch --nolog http://www.example.com/some/page

different formats and storages. Requests and Responses Understand the classes used to represent HTTP requests and responses. Link Extractors Convenient classes to extract links to follow from pages scraping [https://en.wikipedia.org/wiki/Web_scraping], it can also be used to extract data using APIs (such as Amazon Associates Web Services [https://affiliate- program.amazon.com/gp/advertising/api/detail/main pipelines). Wide range of built-in extensions and middlewares for handling: cookies and session handling HTTP features like compression, authentication, caching user-agent spoofing robots.txt crawl depth restriction

different formats and storages. Requests and Responses Understand the classes used to represent HTTP requests and responses. Link Extractors Convenient classes to extract links to follow from pages scraping [https://en.wikipedia.org/wiki/Web_scraping], it can also be used to extract data using APIs (such as Amazon Associates Web Services [https://affiliate- program.amazon.com/gp/advertising/api/detail/main pipelines). Wide range of built-in extensions and middlewares for handling: cookies and session handling HTTP features like compression, authentication, caching user-agent spoofing robots.txt crawl depth restriction

different formats and storages. Requests and Responses Understand the classes used to represent HTTP requests and responses. Link Extractors Convenient classes to extract links to follow from pages scraping [https://en.wikipedia.org/wiki/Web_scraping], it can also be used to extract data using APIs (such as Amazon Associates Web Services [https://affiliate- program.amazon.com/gp/advertising/api/detail/main pipelines). Wide range of built-in extensions and middlewares for handling: cookies and session handling HTTP features like compression, authentication, caching user-agent spoofing robots.txt crawl depth restriction