Introducing

Introducing New API Pagination Patterns Using Cursors and Query Filter Extensions

Apr 28, 2021

As we have been writing new APIs that bring you more and better reference data, we have also been thinking of ways to improve the user experience of our APIs. Starting with our Reference Tickers API, we are introducing our new pagination model and query filter extensions.  The goals of introducing this pattern are to maintain the quality of our API response times, provide a consistent interface for pagination between APIs, allow for incremental performance improvements without introducing breaking changes, and to provide a mechanism for fetching pages concurrently for those who need it.

Pagination

Several of our APIs have pagination capabilities that allow users to make large requests and page through the different results. Currently, not all APIs share the same query interface, so we are introducing a single standard for paginating Polygon.io APIs.

Next/Previous URL

An API that supports our standard pagination will return top level attributes in its response body if there are more pages of data to retrieve. Those attributes will be

next_url
or
previous_url
.
next_url
signals that there is another page of data to retrieve while
previous_url
signals that you are in the middle of paginating through responses and we are able to show you the previous page. If
next_url
is not present in the response then there are no more pages to query over.

Not every API will have the ability to page backwards (

previous_url
) but those that do will be noted in our API docs.

Cursor

Our APIs will use an opaque query parameter (`cursor`) to know how to fetch the next or previous page. This allows us to obscure a particular API’s low-level paging mechanism from the user’s request so that we can make future improvements to pagination queries without breaking the API’s public contract.

Authentication Params

As with all of our APIs, we use the apiKey query param to authenticate requests. In order to not make our response payloads contain sensitive information we will exclude the API key from the values in next/previous_url, requiring the user to append that information to the value of next/previous_url. In the future, we plan to allow API Keys to be sent in the Authorization Header which will render this additional step unnecessary.

Example

A request to our Reference Tickers API at

https://api.polygon.io/vX/reference/tickers?limit=2
could get the following response

{
  "results": [
    {
      "ticker": "A",
      "name": "Agilent Technologies Inc."
    },
        {
      "ticker": "AA",
      "name": "Alcoa Corporation"
    }
  ],
  "status": "OK",
  "request_id": "491513491b2bdd4c1647cd98bc8bc6df",
  "count": 2,
  "next_url": "https://api.polygon.io/vX/reference/tickers?cursor=YWN0aXZlP..."
}

To get the next page of data, you would make an HTTP GET on the value of

next_url
appending the API key query param:
https://api.polygon.io/vX/reference/tickers?cursor=YWN0aXZlP...&apiKey=...
.

When the `cursor` query param is present, only the apiKey query param will be respected by the API, all others will be discarded.

Query Filter Extensions

We are also introducing a new query pattern to allow for more granular searches. We are extending our model with selectors of the form <field>.<operator>=<value>. We will start by offering the following operators on certain fields:

  • lt
    less than
  • lte
    less than or equal
  • gt
    greater than
  • gte
    greater than or equal

For example:

  • ?ticker.gte=A
  • ?ticker.gte=A&ticker.lt=B
  • ?last_updated_utc.gt=2021-03-30T00:00:00Z

Note:

The notation

<field>=<value
> is still a valid query and still represents an equality check.
?ticker=AAPL
will only return results where the
ticker
attribute equals exactly
AAPL
.

All fields that support query extensions will be noted in our REST Docs.  For an example, check out our recently updated Ticker News API.

Examples and Usage

This new query pattern opens the door for some interesting use cases. We've been getting a lot of feedback asking for a way to be able to page through the new Tickers API with parallel workers. With these new filter extensions, you can break up your request into discrete parts and do just that. For example, these 3 requests could be followed in parallel to get tickers starting with A, B, and C:

GET https://api.polygon.io/vX/reference/tickers?sort=ticker&ticker.gte=A&ticker.lt=B

GET https://api.polygon.io/vX/reference/tickers?sort=ticker&ticker.gte=B&ticker.lt=C

GET https://api.polygon.io/vX/reference/tickers?sort=ticker&ticker.gte=C&ticker.lt=D

This pattern also naturally supports filter results by the non-sort field. For example, this request gets news articles that were published since April 22, 2021:

https://api.polygon.io/v2/reference/news?limit=2&sort=ticker&published_utc.gt=2021-04-22T00:00:00-04:00

Rollout Plan

All new APIs going forward will have this standard pagination approach. Any new list APIs will take advantage of our query filter extension pattern as well. Where applicable we will rollout these patterns to existing APIs while maintaining backwards compatibility.

Summary

We are introducing a new pagination approach that will become the standard across our APIs going forward and query filter extensions that will increase the capabilities of our APIs to allow for more powerful searches. We are excited for you to start using our new API enhancements and we hope they make finding market data more accessible.

From the blog

See what's happening at polygon.io

polygon plugin for chatgpt Feature Image
announcement

Announcing the Polygon ChatGPT Plugin

Introducing the Polygon plugin for ChatGPT, a powerful tool that simplifies financial research by providing easy access to real-time and historical market data for Stocks, Options, Indices, Forex, and Crypto.

Team Polygon Profile Photo

editor

Team Polygon

indices data has arrived Feature Image
announcement

Indices Data Has Arrived

We are excited to announce the launch of the Indices API endpoints, offering real-time and historical market data for over 11,400+ Indices, including the S&P 500, Dow Jones Industrial Average, Nasdaq-100, and the VIX. With access to comprehensive market data, businesses and individual investors can make informed decisions in today's rapidly evolving financial markets. We have brought the same low-latency and developer-centric API design of our

Team Polygon Profile Photo

editor

Team Polygon