extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) © Copyright 2008-2013, Scrapy developers. Last updated would be a spider argument. I’m scraping a XML document and my XPath selector doesn’t return any items You may need to remove namespaces. See Removing namespaces. I’m getting an error: “cannot import

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) Settings The Scrapy settings allows you to customize settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) Settings The Scrapy settings allows you to customize settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) Settings The Scrapy settings allows you to customize settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) Settings The Scrapy settings allows you to customize settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

be repeated) --callback or -c: spider method to use as callback for parsing the response --meta or -m: additional request meta that will be passed to the callback request. This must be a valid json string extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) strip (boolean) – whether to strip whitespaces from

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) strip (boolean) – whether to strip whitespaces from settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context following function in process_value: def process_value(value): m = re.search("javascript:goToPage\('(.*?)'", value) if m: return m.group(1) strip (boolean) – whether to strip whitespaces from settings (less precedence) The population of these settings sources is taken care of internally, but a manual handling is possible using API calls. See the Settings API topic for reference. These mechanisms

be repeated) --callback or -c: spider method to use as callback for parsing the response --meta or -m: additional request meta that will be passed to the callback request. This must be a valid json string entries): for entry in entries: date_time = datetime.strptime(entry['lastmod'], '%Y- %m-%d') if date_time.year >= 2005: yield entry This would retrieve only entries extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context

be repeated) --callback or -c: spider method to use as callback for parsing the response --meta or -m: additional request meta that will be passed to the callback request. This must be a valid json string entries): for entry in entries: date_time = datetime.strptime(entry['lastmod'], '%Y- %m-%d') if date_time.year >= 2005: yield entry This would retrieve only entries extracts a length from it: def parse_length(text, loader_context): unit = loader_context.get('unit', 'm') # ... length parsing code goes here ... return parsed_length By accepting a loader_context