. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150 4 Built-in services 153 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore >= 1.4.87 The AWS credentials can be passed as user/password in the

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4 Built-in services 137 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show stored on Amazon S3. • URI scheme: s3 • Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore (Python 2 and Python

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122 4 Built-in services 125 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore (Python 2 and Python 3) or boto (Python 2 only) The AWS credentials can be passed as

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 4 Built-in services 135 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134 4 Built-in services 137 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 4 Built-in services 141 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4 Built-in services 143 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore >= 1.4.87 The AWS credentials can be passed as user/password in the

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 4 Built-in services 143 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show Example URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore The AWS credentials can be passed as user/password in the URI

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160 4 Built-in services 163 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: boto3 >= 1.20.0 The AWS credentials can be passed as user/password in the

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 4 Built-in services 157 4.1 Logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . for web scraping, it can also be used to extract data using APIs (such as Amazon Associates Web Services) or as a general purpose web crawler. 2.1.1 Walk-through of an example spider In order to show URIs: – s3://mybucket/path/to/export.csv – s3://aws_key:aws_secret@mybucket/path/to/export.csv • Required external libraries: botocore >= 1.4.87 The AWS credentials can be passed as user/password in the