. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 and XPath expressions, with helper methods to extract using regular expressions. • An interactive shell console (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell): echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc – Reload .bashrc to ensure

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46 3.7 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 for monitoring the performance of your spiders and detecting when they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service in the Shell To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended Python console) installed on your system. To start a shell, you must

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56 for monitoring the performance of your spiders and detecting when they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service in the Shell To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended Python console) installed on your system. To start a shell, you must

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 3.7 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 for monitoring the performance of your spiders and detecting when they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service in the Shell To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended Python console) installed on your system. To start a shell, you must

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 for monitoring the performance of your spiders and detecting when they get broken • An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders • A System service in the Shell To illustrate the use of Selectors we’re going to use the built-in Scrapy shell, which also requires IPython (an extended Python console) installed on your system. To start a shell, you must

the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Item Loaders Populate your items with functionality Reference Command line tool Learn about the command-line tool and see all available commands. Requests and Responses Understand the classes used to represent HTTP requests and responses. for monitoring the performance of your spiders and detecting when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 and XPath expressions, with helper methods to extract using regular expressions. • An interactive shell console (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell): echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc – Reload .bashrc to ensure

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74 and XPath expressions, with helper methods to extract using regular expressions. • An interactive shell console (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell): echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc – Reload .bashrc to ensure

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54 3.6 Scrapy shell . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64 and XPath expressions, with helper methods to extract using regular expressions. • An interactive shell console (IPython aware) for trying out the CSS and XPath expressions to scrape data, very useful when used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell): echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc – Reload .bashrc to ensure

the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy shell Test your extraction code in an interactive environment. Item Loaders Populate your items with functionality Reference Command line tool Learn about the command-line tool and see all available commands. Requests and Responses Understand the classes used to represent HTTP requests and responses. for monitoring the performance of your spiders and detecting when they get broken An Interactive shell console for trying XPaths, very useful for writing and debugging your spiders A System service designed