format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation Documentation, Release 1.6.0 (continued from previous page) with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

how it can help you. Installation guide Get Scrapy installed on your computer. Scrapy Tutorial Write your first Scrapy project. Examples Learn more by playing with a pre-made Scrapy project. Basic Command line tool Learn about the command-line tool used to manage your Scrapy project. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy or the storage backend (FTP or Amazon S3 [https://aws.amazon.com/s3/], for example). You can also write an item pipeline to store the items in a database. What else? You’ve seen how to extract and store

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation Documentation, Release 1.7.4 (continued from previous page) with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 2.1.2 What else? You’ve seen how to extract and url.split("/")[-2] filename = f'quotes-{page}.html' with open(filename, 'wb') as f: f.write(response.body) self.log(f'Saved file {filename}') As you can see, our Spider subclasses scrapy.Spider and defines Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

format (XML or CSV, for example) or the storage backend (FTP or Amazon S3, for example). You can also write an item pipeline to store the items in a database. 6 Chapter 2. First steps Scrapy Documentation url.split("/")[-2] filename = 'quotes-%s.html' % page with open(filename, 'wb') as f: f.write(response.body) self.log('Saved file %s' % filename) As you can see, our Spider subclasses scrapy.Spider and Spiders. • start_requests(): must return an iterable of Requests (you can return a list of requests or write a generator function) which the Spider will begin to crawl from. Subsequent requests will be generated

how it can help you. Installation guide Get Scrapy installed on your computer. Scrapy Tutorial Write your first Scrapy project. Examples Learn more by playing with a pre-made Scrapy project. Basic Command line tool Learn about the command-line tool used to manage your Scrapy project. Spiders Write the rules to crawl your websites. Selectors Extract the data from web pages using XPath. Scrapy or the storage backend (FTP or Amazon S3 [https://aws.amazon.com/s3/], for example). You can also write an item pipeline to store the items in a database. What else? You’ve seen how to extract and store