Introducing Flat Files: Bulk Data Simplified
Meet Flat Files, our latest innovation designed for hassle free self-serve access to large financial market datasets.

editor
Introducing
May 26, 2020
The conditions that are taken into consideration when creating aggregate bars are quite complex. Users trying to recreate the aggregate Open, High, Low, Close and Volume values from our tick data API often get confused about why their values do not match ours or other popular data sets like Yahoo Finance, Bloomberg, etc.. This can also cause confusion for users seeing trades come in through the real-time WebSocket connection that appear to be higher or lower than the High and Low values made available through our Aggregates or Snapshot endpoints.
There are specific ‘rules’ that revolve around the ‘Sale Conditions’ defined in the UTP and CTA plan specifications that must be followed to calculate these aggregate values correctly.
Here, we’ll attempt to clear up any confusion that these highly technical documentations stir up.
CTA and UTP? What’s that? Great Question! We plan on making a separate write up because this can be just as confusing, but for now, let’s stick to the basics: The CTA and UTP feeds are Security Information Processors (SIPs) that every US stock exchange must report their trades and quotes to. These SIPs serve as the sources of truth for all US equities price data.
The CTA plan specification thoroughly describes every aspect of how price data is disseminated in a not so short 93 pages. To save time, we’ve outlined some of the most vital information:
These sale conditions are the raw codes, not what we send out in our real-time and historical ticks APIs.
The Consolidated Update guidelines should be used if you're looking to create aggregates representative of the entire market. The Participant Update guidelines should be used if you're planning on creating aggregates representative of each individual exchange.
Four notes can deviate the way calculations are made for the OHLC Values. Think of these as conditions for the conditions.
- According to the table, it will only be applied to the Market Center Opening Trade. So, this will determine the eligibility for the condition to update the price of the Last executed qualifying trade.
- So, this will determine the eligibility for the condition to update the price of the Last executed qualifying trade.
- Same as the first two Notes, but this one is only applied to the Sold Last trades (trades that are reported late)
- This will determine the eligibility for the condition to update the Open Price
- In instances of multiple Market Center Opening Trades, the latest trade takes precedence.
Many conditions may apply to each trade, so what if the eligibility of these conditions contradict each other?
That is the TLDR for the CTA’s specification regarding Sale Conditions and their eligibility to be calculated into the OHLC aggregates. If you still yearn for more information, check out pages 62-64 of the following document: https://www.ctaplan.com/publicdocs/ctaplan/notifications/trader-update/CTS_BINARY_OUTPUT_SPECIFICATION.pdf
Like the CTA specification, the UTP plan has its own rules and regulations regarding the eligibility of trades to update aggregates. While the CTA Plan specification pertains to the updates of the OHLC values, the UTP plan uses these conditions to determine when to update the HLCV or High, Low, Close, and Volume values.
The UTP Sale condition matrix is available on our Conditions and Indicators page. Most questions asked about the eligibility for trades to update the volume value is denoted by this spec.
Similar to the CTA matrix, this table represents the eligibility of each value to get updated according to two guidelines; the Consolidated Processing Guidelines and the Market Center Processing Guidelines. The Consolidated processing guidelines refers to the consolidated tape, in this case the UTP SIP feed, while the Market Center Processing Guidelines are specific to the individual participants (exchanges). Again, if you're looking to build aggregates to represent toal market volume, use the consolidated processing guidelines. If you're looking to build aggregates to represent each exchange, use the market center processing guidelines.
Much like the CTA plan, there are conditions applied to the conditions. These superscript values appear in a few of the table cells and are defined under the table.
- This is just saying that anybody receiving this data should only update the field if the trade is the first or last eligible transaction of the business day.
- Data recipients should only update the last sale if the trade is received prior to the EOLS control message. This message signals the closing of the trade-reporting window for consolidated last sale calculation eligibility.
- Under the UTP plan, participants (exchanges) are required to report trade transactions that may impact the last sale price within 10 seconds of the normal market session close. This message will be generated 10 seconds after the market close message from the primary market. (Page 34)
- This one states that if the data recipient is tracking the participant’s open values, they should use the condition code ‘Market Center Official Open’ and adhere to those rules.
Other notes to consider when making these adjustments are:
For more information, refer to pages 42 - 47 of the following document: https://www.utpplan.com/DOC/UtpBinaryOutputSpec.pdf
Each trade condition, their definitions, update rules, and CTA/UTP mappings can now be found in our Conditions API endpoint:
{ "asset_class": "stocks", "data_types": [ "trade" ], "id": 2, "name": "Average Price Trade", "sip_mapping": { "CTA": "B", "UTP": "W" }, "type": "condition", "update_rules": { "consolidated": { "updates_high_low": false, "updates_open_close": false, "updates_volume": true }, "market_center": { "updates_high_low": false, "updates_open_close": false, "updates_volume": true }
Polygon.io takes both of these specifications into consideration when calculating the aggregate bars that we provide. We follow the consolidated update guidelines of both the CTA and UTP plans. If you’re planning to recreate bars from our tick data yourself, these are the guidelines you will need to follow. We have carefully created a system that takes every condition that is applied to a trade into consideration to curate the most accurate data possible. Polygon normalizes the conditions between the SIPs into one uniform list. The condition codes you will need to follow are on the Conditions and Indicators page, and are also available through our Condition Mappings endpoint.
If you ever have any additional questions or concerns, please don’t hesitate to contact us. We’re always happy to help any way we can.
Jack Bell
jack
See what's happening at polygon.io
Meet Flat Files, our latest innovation designed for hassle free self-serve access to large financial market datasets.
editor
Discover the newly launched Universal Snapshots API, a powerful tool providing streamlined and consistent market data across Stocks, Options, Indices, Forex, and Crypto assets, all in one API call - enhancing your market analysis and decision-making process.
editor
Introducing the Polygon plugin for ChatGPT, a powerful tool that simplifies financial research by providing easy access to real-time and historical market data for Stocks, Options, Indices, Forex, and Crypto.
editor