. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.3 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RS"]["https"] = MyHandler 240 Chapter 6. Extending Scrapy Scrapy Documentation, Release 2.10.1 6.3 Downloader Middleware The downloader middleware is a framework of hooks into Scrapy’s request/response should either: return None, return a Response object, return a Request object, or raise IgnoreRequest. 6.3. Downloader Middleware 241 Scrapy Documentation, Release 2.10.1 If it returns None, Scrapy will continue

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.3 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RS"]["https"] = MyHandler 240 Chapter 6. Extending Scrapy Scrapy Documentation, Release 2.11.1 6.3 Downloader Middleware The downloader middleware is a framework of hooks into Scrapy’s request/response should either: return None, return a Response object, return a Request object, or raise IgnoreRequest. 6.3. Downloader Middleware 241 Scrapy Documentation, Release 2.11.1 If it returns None, Scrapy will continue

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.3 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RS"]["https"] = MyHandler 240 Chapter 6. Extending Scrapy Scrapy Documentation, Release 2.11.1 6.3 Downloader Middleware The downloader middleware is a framework of hooks into Scrapy’s request/response should either: return None, return a Response object, return a Request object, or raise IgnoreRequest. 6.3. Downloader Middleware 241 Scrapy Documentation, Release 2.11.1 If it returns None, Scrapy will continue

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238 6.3 Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . RS"]["https"] = MyHandler 240 Chapter 6. Extending Scrapy Scrapy Documentation, Release 2.11.1 6.3 Downloader Middleware The downloader middleware is a framework of hooks into Scrapy’s request/response should either: return None, return a Response object, return a Request object, or raise IgnoreRequest. 6.3. Downloader Middleware 241 Scrapy Documentation, Release 2.11.1 If it returns None, Scrapy will continue

Downloader Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In order for a spider to override the default user agent, its user_agent attribute must be set. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism response being processed • spider (BaseSpider object) – the spider for which this response is intended 6.3. Spider Middleware 95 Scrapy Documentation, Release 0.9 process_spider_output(response, result, spider)

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HTTPPROXY_AUTH_ENCODING Default: "latin-1" The default encoding for proxy authentication on HttpProxyMiddleware. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism action and your middleware could depend on some previous (or subsequent) middleware being applied. 6.3. Spider Middleware 187 Scrapy Documentation, Release 1.5.2 If you want to disable a builtin middleware

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . HTTPPROXY_AUTH_ENCODING Default: "latin-1" The default encoding for proxy authentication on HttpProxyMiddleware. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism action and your middleware could depend on some previous (or subsequent) middleware being applied. 6.3. Spider Middleware 193 Scrapy Documentation, Release 1.6.0 If you want to disable a builtin middleware

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In order for a spider to override the default user agent, its user_agent attribute must be set. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism action and your middleware could depend on some previous (or subsequent) middleware being applied. 6.3. Spider Middleware 115 Scrapy Documentation, Release 0.12.0 If you want to disable a builtin middleware

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111 i 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In order for a spider to override the default user agent, its user_agent attribute must be set. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism response being processed • spider (BaseSpider object) – the spider for which this response is intended 6.3. Spider Middleware 119 Scrapy Documentation, Release 0.14.4 process_spider_output(response, result

Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121 6.3 Spider Middleware . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . In order for a spider to override the default user agent, its user_agent attribute must be set. 6.3 Spider Middleware The spider middleware is a framework of hooks into Scrapy’s spider processing mechanism need to be enabled through a particular setting. See each middleware documentation for more info. 6.3. Spider Middleware 129 Scrapy Documentation, Release 0.16.5 6.3.2 Writing your own spider middleware