Introducing

Introducing Flat Files: Bulk Data Simplified

Nov 7, 2023

Meet Flat Files, our latest innovation designed for hassle free self-serve access to large financial market datasets. Leveraging the S3 protocol, this feature offers a quick and efficient way to download large datasets across Stocks, Options, Forex, and Crypto, all packaged conveniently in a CSV format.

Flat Files eliminates the need for repetitive API calls and greatly simplifies integration with existing financial applications. It streamlines your data retrieval workflow and complements our existing platform, which continues to excel in delivering real-time and historical market data via API endpoints and WebSocket feeds.

Coverage for Stocks, Options, Forex, and Crypto

We have aggregated countless data points into easily accessible gzipped CSV files, saving you hours of time and countless API calls. With just a single command, dive into meticulously curated datasets, perfect for analysis, back-testing, and research. Below, we'll walk you through each asset class, offering snapshots of the wealth of information you can harness.

Stocks

Easily navigate the complexities of all major U.S. exchanges and dark pools, including NYSE, Nasdaq, Cboe, Finra with Flat Files. Our datasets provide tick-level granularity for every available ticker. From trade and quote data to minute and daily candlesticks, our files serve as a bedrock for robust market analysis, strategy development, and trend identification, catering to professionals and enthusiasts alike.

Trades: Tick-level trades with nanosecond timestamps

ticker,conditions,correction,exchange,id,participant_timestamp,price,sequence_number,sip_timestamp,size,tape,trf_id,trf_timestamp
A,"12,37",0,8,52983525028937,1697532069025572000,111.52,3817,1697532069025839872,2,1,0,0
A,"12,37",0,8,52983525028938,1697532069025572000,111.51,3818,1697532069025857792,10,1,0,0
A,"12,37",0,8,52983525030761,1697540401144206000,112.01,4831,1697540401144454144,4,1,0,0
...

Quotes: Top of book quotes with nanosecond timestamps

ticker,ask_exchange,ask_price,ask_size,bid_exchange,bid_price,bid_size,conditions,indicators,participant_timestamp,sequence_number,sip_timestamp,tape,trf_timestamp
A,8,0,0,8,0,0,"1,81",,1697529540085795000,102,1697529540086096384,1,0
A,19,0,0,19,0,0,"1,81",,1697529540117112000,282,1697529540117744128,1,0
A,11,118.69,2,11,0,0,"1,81",,1697529600011330144,495,1697529600011438592,1,0
...

Minute Candlesticks: Open, high, low, close, and volume at per minute granularity

ticker,volume,open,close,high,low,window_start,transactions
A,9794,111.33,110.82,111.33,110.82,1697635800000000000,26
A,995,111.01,111.09,111.115,110.99,1697635860000000000,40
A,2154,110.78,111.175,111.18,110.76,1697635920000000000,39
...

Daily Candlesticks: Open, high, low, close, and volume at per day granularity

ticker,volume,open,close,high,low,window_start,transactions
A,1803500,111.33,108.61,111.33,108.535,1697601600000000000,31518
AA,6489364,27.1,26.52,27.43,26.5,1697601600000000000,54944
AAA,1025,24.845,24.845,24.845,24.8188,1697601600000000000,13
...

Options

Unlock the world of options trading with Flat Files, offering detailed insight from all U.S. options markets including CBOE, NYSE, and NASDAQ. Our data encompasses essential information, from trades and quotes data to minute and daily candlesticks, offering a complete picture that's indispensable for traders seeking to analyze strategies, assess risks, and identify market opportunities.

Trades: Tick-level trades with nanosecond timestamps

ticker,conditions,correction,exchange,price,sip_timestamp,size
O:A231020C00110000,209,0,315,2.3,1697552097181000000,8
O:A231020C00110000,227,0,302,2.21,1697552097281000000,7
O:A231020C00115000,209,0,325,0.38,1697549416345000000,1
...

Quotes: Top of book quotes with nanosecond timestamps

ticker,ask_exchange,ask_price,ask_size,bid_exchange,bid_price,bid_size,sequence_number,sip_timestamp
O:AMZN231020C00045000,308,86.1,1,308,84.6,1,1423266,1697549400071915776
O:AMZN231020C00045000,323,86.8,10,323,83.8,10,1427165,1697549400148152576
O:AMZN231020C00045000,323,87.55,1,319,83.1,1,1445286,1697549400475110912
...

Minute Candlesticks: Open, high, low, close, and volume at per minute granularity

ticker,volume,open,close,high,low,window_start,transactions
O:R231020C00095000,1,10.5,10.5,10.5,10.5,1697554500000000000,1
O:R231020C00100000,1,5.5,5.5,5.5,5.5,1697554680000000000,1
O:R231020C00100000,2,6.95,6.97,6.97,6.95,1697559600000000000,2
...

Daily Candlesticks: Open, high, low, close, and volume at per day granularity

ticker,volume,open,close,high,low,window_start,transactions
O:NAAS231020P00002500,100,0.05,0.05,0.05,0.05,1697515200000000000,10
O:NAAS231117C00002500,1,0.9,0.9,0.9,0.9,1697515200000000000,1
O:NAAS231215C00005000,1,0.05,0.05,0.05,0.05,1697515200000000000,1
...

Forex

Immerse yourself in the global foreign exchange market for over 1,750+ currency pairs. From quote data to minute and daily candlesticks, capturing the pulse of monetary market dynamics. Ideal for traders and analysts, our data assists in monitoring currency fluctuations, analyzing Forex market trends, and making informed decisions in this highly liquid market.

Quotes: Global foreign exchange rate quotes

ticker,ask_exchange,ask_price,bid_exchange,bid_price,participant_timestamp
X:EUR-USD,48,1.08066,48,1.08063,1679961600000000000
X:EUR-USD,48,1.08066,48,1.08059,1679961600000000000
X:EUR-USD,48,1.08066,48,1.08063,1679961600000000000
...

Minute Candlesticks: Open, high, low, close, and volume at per minute granularity

ticker,volume,open,close,high,low,window_start,transactions
C:EURUSD,120,1.08063,1.08033,1.0807,1.0803,1679961600000,120
C:EURUSD,120,1.08034,1.08055,1.0806,1.0803,1679961660000,120
C:EURUSD,83,1.0804,1.08066,1.08068,1.0803,1679961720000,83
...

Daily Candlesticks: Open, high, low, close, and volume at per day granularity

ticker,volume,open,close,high,low,window_start,transactions
C:THBZAR,65503,0.5315684,0.5291029,0.5344774,0.5282698,1680047999999,65503
C:AUDCAD,151816,0.90929,0.9116132,0.9136431,0.9082,1680047999999,151816
C:USDIDR,4467,15106.5,15061,15114.8,15027.9,1680047999999,4467
...

Crypto

Enter the digital currency frontier with Flat Files. Our datasets offer in-depth views across an expansive set of crypto pairings, including trade data and minute and daily candlesticks. This information is critical for anyone looking to dive into market analysis in this rapidly evolving digital economy.

Trades: Tick-level trades with nanosecond timestamp

ticker,conditions,exchange,id,participant_timestamp,price,size
X:1INCH-USD,1,23,156336,1694390820946705920,0.239,23
X:1INCH-USD,1,23,156337,1694390820946891776,0.239,28
X:1INCH-USD,1,2,1415721852,1694391203251000064,0.23949,3
...

Minute Candlesticks: Open, high, low, close, and volume at per minute granularity

ticker,volume,open,close,high,low,window_start,transactions
X:1INCH-USD,52.569048,0.239,0.239,0.239,0.239,1694390820000000000,2
X:1INCH-USD,3.44838089,0.23949,0.23949,0.23949,0.23949,1694391180000000000,1
X:1INCH-USD,500,0.239,0.239,0.239,0.239,1694391840000000000,1
...

Daily Candlesticks: Open, high, low, close, and volume at per day granularity

ticker,volume,open,close,high,low,window_start,transactions
X:1INCH-USD,549272.6971407501,0.239,0.225,0.24,0.219,1694390400000000000,822
X:AAVE-USD,23376.0799031201,53.98,52.13,54.18,51.36,1694390400000000000,7666
X:ACH-USD,37678619.4463618,0.013575,0.01351,0.013993,0.0132,1694390400000000000,2739
...

These enhancements are not just improvements; they represent a paradigm shift in self-serve financial market data accessibility. By empowering you with this depth of data, we're ensuring you’re equipped to make informed decisions, backed by comprehensive market insights.

Getting Started with S3

Now that you are familiar with the datasets available, let's look at how you can actually access these datasets. Flat Files leverages S3 technology, making data retrieval a hassle-free experience since there is a wide range of supported clients. We have tested and can verify support for four popular S3 clients, these include the AWS S3 CLI, Rclone, MinIO, and the Python boto3 SDK. Each client requires a simple configuration process and for detailed setup instructions, please see our knowledge base article.

For the examples below, we are going to be using the MinIO client, after you download and install the client, the configuration looks something like what's shown below in your

.mc/config.json

file:

{
        "version": "10",
        "aliases": {
                "s3": {
                        "url": "https://files.polygon.io",
                        "accessKey": "YourAccessKey",
                        "secretKey": "YourSecretKey",
                        "api": "S3v4",
                        "path": "auto"
                }
        }
}

Now that you have the client configured, let’s explore the top-level buckets or directories to see what is available. Listing operations work for anyone, even without specific subscription access to the Flat Files products, but to actually download a dataset you will need an active subscription.

$ mc ls s3/flatfiles/
[2023-10-18 17:44:05 PDT]     0B global_crypto/
[2023-10-18 17:44:05 PDT]     0B global_forex/
[2023-10-18 17:44:05 PDT]     0B us_options_opra/
[2023-10-18 17:44:05 PDT]     0B us_stocks_sip/

For instance, if you want to look at the stock trades directory, you can list the files in that specific path as shown below:

$ mc ls s3/flatfiles/us_stocks_sip/trades_v1/2023/08/
[2023-09-03 22:05:52 PDT] 1.4GiB STANDARD 2023-08-01.csv.gz
[2023-09-03 22:06:47 PDT] 1.5GiB STANDARD 2023-08-02.csv.gz
…
[2023-09-03 22:04:17 PDT] 1.2GiB STANDARD 2023-08-30.csv.gz
[2023-09-03 22:04:05 PDT] 1.2GiB STANDARD 2023-08-31.csv.gz

You can preview the contents of a file without downloading it which can be very useful to learn the format of these files and what to expect. This command displays the beginning of a file, providing a quick look at the data structure and content, which is especially useful for very large files:

$ mc cat s3/flatfiles/us_stocks_sip/trades_v1/2023/08/2023-08-31.csv.gz | gzcat | head -4

ticker,conditions,correction,exchange,id,participant_timestamp,price,sequence_number,sip_timestamp,size,tape,trf_id,trf_timestamp
A,"7,12,37",0,4,71675224470353,1693481448303145000,122.68,12100,1693483335106606848,76,1,12,1693483335106247634
A,"7,12,37",0,4,71675224478274,1693481427798534000,122.68,12285,1693483356815538944,76,1,12,1693483356815177096
A,"12,37",0,8,52983525029460,1693484817590913000,123,14584,1693484817591178752,63,1,0,0

If you decide to use a file for your analysis, simply download it directly to your local machine using the following command:

$ mc cp polygon/flatfiles/us_stocks_sip/trades_v1/2023/08/2023-08-31.csv.gz .

With just a few easy commands shown in these examples, you are all set to leverage these extensive datasets to their full potential.

Next Steps

The Flat Files feature is a game changer for your workflow if you download large datasets, because it eliminates the need for countless API calls, and the hassle of aggregating large amounts of data yourself. This means not only do you save time, but you also gain immediate access to comprehensive datasets with just a simple download. This is the epitome of efficiency and simplicity combined. Whether you are conducting deep research, integrating vast datasets, or back-testing trading strategies, Flat Files ensures you can focus on your core tasks without getting bogged down.

Happy exploring!

• 

  Team Polygon

  editor

  Read More