Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 4.2. Stats Collection

Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

Firebug for scraping Learn how to scrape efficiently using Firebug. Debugging memory leaks Learn how to find and get rid of memory leaks in your crawler. Downloading Item Images Download static images associated must load all DOM in memory which could be a problem for big feeds 'xml' - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83 5.4 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87 5.5 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem _scraped', value) Set global stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get global stat value: >>> stats.get_value('spiders_crawled') 8 Get all global

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96 5.9 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value('max_items_scraped', value) Set stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get stat value: >>> stats.get_value('pages_crawled') 8 Get all stats: >>> stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 5.9 Downloading and must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value(’max_items_scraped’, value) Set stat value only if lower than previous: stats.min_value(’min_free_memory_percent’, value) Get stat value: >>> stats.get_value(’pages_crawled’) 8 Get all stats: >>> stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93 5.9 Downloading must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses XmlXPathSelector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value('max_items_scraped', value) Set stat value only if lower than previous: stats.min_value('min_free_memory_percent', value) Get stat value: >>> stats.get_value('pages_crawled') 8 Get all stats: >>> stats

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 5.8 Debugging memory leaks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 5.9 Downloading and must load all DOM in memory which could be a problem for big feeds •’xml’ - an iterator which uses Selector. Keep in mind this uses DOM parsing and must load all DOM in memory which could be a problem max_value(’max_items_scraped’, value) Set stat value only if lower than previous: stats.min_value(’min_free_memory_percent’, value) Get stat value: >>> stats.get_value(’pages_crawled’) 8 Get all stats: >>> stats