How to Get Started with S3 on Polygon.io

Polygon.io is proud to offer extensive financial datasets, easily accessible thanks to S3. This guide provides detailed steps on setting up various S3 clients to efficiently retrieve the data you need.

Pre-requisites:

A valid flat files subscription with Polygon.io, after which you will receive unique authentication keys. We have standardized the endpoint and bucket details for all users as follows:

Endpoint: https://files.polygon.io Bucket: flatfiles

Supported S3 Clients

Polygon.io rigorously tests and supports four widely recognized S3 clients to ensure compatibility and user convenience. These include:

1. AWS S3 CLI
2. Rclone
3. MinIO
4. Python Boto3 SDK

Below, you’ll find the setup process for each client.

AWS S3 CLI

The AWS S3 Command Line Interface (CLI) is a powerful tool designed to interact with Amazon S3.

Here’s how to get started:

• Install the AWS S3 CLI from the official website.
• Run aws configure in your command line and enter your Access Key ID, Secret Access Key, and default region when prompted.
• To interact with Polygon.io's S3 files, use the following structured command:

aws s3 ls s3://flatfiles/ --endpoint-url https://files.polygon.io

Rclone

Rclone facilitates data synchronization between your local system and S3 storage. Follow these steps to configure Rclone with Polygon.io:

• Download and install Rclone from here.
• Initiate a new remote storage configuration with rclone config.
• Here's an example configuration is provided below for reference:

[s3]
type = s3
env_auth = false
access_key_id = [ACCESS_KEY_ID]
secret_access_key = [SECRET_ACCESS_KEY]
endpoint = https://files.polygon.io

d. After configuration, you can interact with your storage using commands like rclone ls s3:flatfiles.

MinIO

MinIO client works seamlessly with any S3 compatible cloud storage. Here’s how to configure it with Polygon.io:

• If you haven't already, download and set up MinIO from their documentation.
• Modify your .mc/config.json file with the configuration shown below:

   {
           "version": "10",
           "aliases": {
                   "s3": {
                           "url": "https://files.polygon.io",
                           "accessKey": "YourAccessKey",
                           "secretKey": "YourSecretKey",
                           "api": "S3v4",
                           "path": "auto"
                   }
           }
   }

• Your client is now ready for use with commands specific to MinIO.

Python Boto3 SDK

Boto3 is the Amazon Web Services (AWS) SDK for Python, which enables Python developers to write software that makes use of Amazon services like S3. To set it up, follow these steps:

• Install Boto3 by running pip install boto3 if you haven't already.
• Utilize the following script to interact with Polygon.io data:

   import boto3
   from botocore.client import Config

   # Initialize a session using your credentials
   session = boto3.Session(
       aws_access_key_id='YOUR_ACCESS_KEY',
       aws_secret_access_key='YOUR_SECRET_KEY',
   )

   # Create a client with your session and specify the endpoint
   s3 = session.client(
       's3',
       endpoint_url='https://files.polygon.io',
       config=Config(signature_version='s3v4'),
   )

   # List buckets, objects, or perform any other operations
   for key in s3.list_objects(Bucket='flatfiles')['Contents']:
       print(key['Key'])

c. This script initiates a session with your credentials, establishes a client connection, and lists all files available in the specified bucket.

Next Steps

By following the steps outlined for each client, you can efficiently set up your systems to access extensive financial datasets from Polygon.io via S3. or further assistance or more advanced setup options, please refer to the respective client’s official documentation.