. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Basic concepts 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 17 Scrapy Documentation, Release 0.22.0 18 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 21 Scrapy Documentation, Release 0.22.0 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Basic concepts 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 17 Scrapy Documentation, Release 0.20.2 18 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 21 Scrapy Documentation, Release 0.20.2 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Basic concepts 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 17 Scrapy Documentation, Release 0.24.6 18 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 21 Scrapy Documentation, Release 0.24.6 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Scraping basics 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tutorial 17 Scrapy Documentation, Release 0.12.0 18 Chapter 2. First steps CHAPTER 3 Scraping basics 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 21 Scrapy Documentation, Release 0.12.0 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 3 Basic concepts 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 17 Scrapy Documentation, Release 0.14.4 18 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool spiders. You can just create the spider source code files yourself, instead of using this command. 3.1. Command line tool 21 Scrapy Documentation, Release 0.14.4 Usage example: $ scrapy genspider -l

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Basic concepts 19 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Examples 17 Scrapy Documentation, Release 0.18.4 18 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 21 Scrapy Documentation, Release 0.18.4 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 3 Basic concepts 17 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . by playing with a pre-made Scrapy project. 16 Chapter 2. First steps CHAPTER 3 Basic concepts 3.1 Command line tool New in version 0.10. Scrapy is controlled through the scrapy command-line tool myproject genspider • Syntax: scrapy genspider [-t template] • Requires project: yes 3.1. Command line tool 19 Scrapy Documentation, Release 0.16.5 Create a new spider in the current project

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 3 Scraping basics 19 3.1 Items . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Tutorial Write your first Scrapy project. 18 Chapter 2. First steps CHAPTER 3 Scraping basics 3.1 Items The main goal in scraping is to extract structured data from unstructured sources, typically Field(default=0) last_updated = Field() Note: Those familiar with Django will notice that Scrapy Items are declared similar to Django Models, except that Scrapy Items are much simpler as there is no concept

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 3 Basic concepts 23 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . commands: • startproject • genspider • settings • runspider • shell • fetch • view • version 3.1. Command line tool 25 Scrapy Documentation, Release 1.3.3 Project-only commands: • crawl • check common case, the developer is of course free to choose any tool or IDE to write and debug his spiders. 3.1. Command line tool 27 Scrapy Documentation, Release 1.3.3 Usage example: $ scrapy edit spider1

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18 3 Basic concepts 21 3.1 Command line tool . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . startproject • settings • runspider • shell • fetch • view • version Project-only commands: • crawl 3.1. Command line tool 23 Scrapy Documentation, Release 1.0.7 • check • list • edit • parse • genspider example: $ scrapy list spider1 spider2 edit • Syntax: scrapy edit • Requires project: yes 3.1. Command line tool 25 Scrapy Documentation, Release 1.0.7 Edit the given spider using the editor