. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index 335 Index 337 ii Scrapy Documentation, Release 2.4.1 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It CHAPTER TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index 333 Index 335 ii Scrapy Documentation, Release 2.3.0 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It CHAPTER TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index 365 Index 367 ii Scrapy Documentation, Release 2.6.3 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy (/skrepa/) is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index 289 Index 291 ii Scrapy Documentation, Release 1.7.4 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Index 347 Index 349 ii Scrapy Documentation, Release 2.5.1 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It CHAPTER TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like

2.4 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl websites and Scrapy project. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Items Telnet Console Inspect a running crawler using a built-in Python console. Web Service Monitor and control a crawler using a web service. Solving specific problems Frequently Asked Questions Get answers

1.7 documentation Scrapy is a fast high-level web crawling [https://en.wikipedia.org/wiki/Web_crawler] and web scraping [https://en.wikipedia.org/wiki/Web_scraping] framework, used to crawl websites and Scrapy project. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Items Telnet Console Inspect a running crawler using a built-in Python console. Web Service Monitor and control a crawler using a web service. Solving specific problems Frequently Asked Questions Get answers

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like for screen scraping (more precisely, web scraping), it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. The purpose of this document

Index 399 Index 401 ii Scrapy Documentation, Release 2.10.1 Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It TWO FIRST STEPS 2.1 Scrapy at a glance Scrapy (/skrepa/) is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like Scrapy was originally designed for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70 4.5 Web Service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . help CHAPTER 2 First steps 2.1 Scrapy at a glance Scrapy is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like for screen scraping (more precisely, web scraping), it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. The purpose of this document