scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list • Syntax: scrapy list • Requires project: yes List all ['http://www.example.com/categories/%s' % category] # ... Spider arguments can also be passed through the schedule.json API. 28 Chapter 3. Basic concepts Scrapy Documentation, Release 0.16.5 3.3.2 Built-in spiders crawl: curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d

scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list Syntax: scrapy list Requires project: yes List all available com/categories/%s' % category] # ... Spider arguments can also be passed through the schedule.json API. Built-in spiders reference Scrapy comes with some useful generic spiders that you can crawl: curl http://scrapy1.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1 -d part=1 curl http://scrapy2.mycompany.com:6800/schedule.json -d project=myproject -d spider=spider1

scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list Syntax: scrapy list Requires project: yes List all available managing processes, Scrapyd provides a JSON web service to upload new project versions (as eggs) and schedule spiders. This feature is optional and can be disabled if you want to implement your own custom Scrapyd org/library/tempfile.html] for temporary files. Scheduling a spider run To schedule a spider run: $ curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2 {"status": "ok", "jobid":

scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list • Syntax: scrapy list • Requires project: yes List all sorted(get_commands().items()): print " ", func.__doc__ def cmd_run(args, opts): """run - schedule spider for running""" jsonrpc_call(opts, 'crawler/queue', 'append_spider_name', args[0]) def cmd_stop(args managing processes, Scrapyd provides a JSON web service to upload new project versions (as eggs) and schedule spiders. This feature is optional and can be disabled if you want to implement your own custom Scrapyd

scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list Syntax: scrapy list Requires project: yes List all available .items()): print " ", func.__doc__ def cmd_run(args, opts): """run - schedule spider for running""" jsonrpc_call(opts, 'crawler/queue', 'append_spider_name', args[0]) def managing processes, Scrapyd provides a JSON web service to upload new project versions (as eggs) and schedule spiders. This feature is optional and can be disabled if you want to implement your own custom Scrapyd

scrapy server [ ... scrapyd starts and stays idle waiting for spiders to get scheduled ... ] To schedule spiders, use the Scrapyd JSON API. list • Syntax: scrapy list 22 Chapter 3. Basic concepts Scrapy managing processes, Scrapyd provides a JSON web service to upload new project versions (as eggs) and schedule spiders. This feature is optional and can be disabled if you want to implement your own custom Scrapyd use tempfile for temporary files. 5.7.7 Scheduling a spider run To schedule a spider run: $ curl http://localhost:6800/schedule.json -d project=myproject -d spider=spider2 {"status": "ok", "jobid":

yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the main advantages Scrapy’s mechanism of following links: when you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes example.com/categories/%s' % category] # ... Spider arguments can also be passed through the Scrapyd schedule.json API. See Scrapyd documentation. Generic Spiders Scrapy comes with some useful generic spiders

yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the main advantages Scrapy’s mechanism of following links: when you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes example.com/categories/%s' % category] # ... Spider arguments can also be passed through the Scrapyd schedule.json API. See Scrapyd documentation. Generic Spiders Scrapy comes with some useful generic spiders

yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the main advantages Scrapy’s mechanism of following links: when you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes http_pass=mypassword -a user_agent=mybot Spider arguments can also be passed through the Scrapyd schedule.json API. See Scrapyd documentation. Generic Spiders Scrapy comes with some useful generic spiders

yield a Python dict with the extracted quote text and author, look for a link to the next page and schedule another request using the same parse method as callback. Here you notice one of the main advantages Scrapy’s mechanism of following links: when you yield a Request in a callback method, Scrapy will schedule that request to be sent and register a callback method to be executed when that request finishes http_pass=mypassword -a user_agent=mybot Spider arguments can also be passed through the Scrapyd schedule.json API. See Scrapyd documentation. 40 Chapter 3. Basic concepts Scrapy Documentation, Release