Introducing

Introducing New API Pagination Patterns Using Cursors and Query Filter Extensions

Apr 28, 2021

As we have been writing new APIs that bring you more and better reference data, we have also been thinking of ways to improve the user experience of our APIs. Starting with our Reference Tickers API, we are introducing our new pagination model and query filter extensions.  The goals of introducing this pattern are to maintain the quality of our API response times, provide a consistent interface for pagination between APIs, allow for incremental performance improvements without introducing breaking changes, and to provide a mechanism for fetching pages concurrently for those who need it.

Pagination

Several of our APIs have pagination capabilities that allow users to make large requests and page through the different results. Currently, not all APIs share the same query interface, so we are introducing a single standard for paginating Polygon.io APIs.

Next/Previous URL

An API that supports our standard pagination will return top level attributes in its response body if there are more pages of data to retrieve. Those attributes will be

or . signals that there is another page of data to retrieve while signals that you are in the middle of paginating through responses and we are able to show you the previous page. If is not present in the response then there are no more pages to query over.

Not every API will have the ability to page backwards (

) but those that do will be noted in our API docs.

Cursor

Our APIs will use an opaque query parameter (`cursor`) to know how to fetch the next or previous page. This allows us to obscure a particular API’s low-level paging mechanism from the user’s request so that we can make future improvements to pagination queries without breaking the API’s public contract.

Authentication Params

As with all of our APIs, we use the apiKey query param to authenticate requests. In order to not make our response payloads contain sensitive information we will exclude the API key from the values in next/previous_url, requiring the user to append that information to the value of next/previous_url. In the future, we plan to allow API Keys to be sent in the Authorization Header which will render this additional step unnecessary.

Example

A request to our Reference Tickers API at

could get the following response

{
  "results": [
    {
      "ticker": "A",
      "name": "Agilent Technologies Inc."
    },
        {
      "ticker": "AA",
      "name": "Alcoa Corporation"
    }
  ],
  "status": "OK",
  "request_id": "491513491b2bdd4c1647cd98bc8bc6df",
  "count": 2,
  "next_url": "https://api.polygon.io/vX/reference/tickers?cursor=YWN0aXZlP..."
}

To get the next page of data, you would make an HTTP GET on the value of

appending the API key query param: .

When the `cursor` query param is present, only the apiKey query param will be respected by the API, all others will be discarded.

Query Filter Extensions

We are also introducing a new query pattern to allow for more granular searches. We are extending our model with selectors of the form <field>.<operator>=<value>. We will start by offering the following operators on certain fields:

• lt
  less than
• lte
  less than or equal
• gt
  greater than
• gte
  greater than or equal

For example:

• ?ticker.gte=A
• ?ticker.gte=A&ticker.lt=B
• ?last_updated_utc.gt=2021-03-30T00:00:00Z

Note:

The notation

> is still a valid query and still represents an equality check. will only return results where the attribute equals exactly .

All fields that support query extensions will be noted in our REST Docs.  For an example, check out our recently updated Ticker News API.

Examples and Usage

This new query pattern opens the door for some interesting use cases. We've been getting a lot of feedback asking for a way to be able to page through the new Tickers API with parallel workers. With these new filter extensions, you can break up your request into discrete parts and do just that. For example, these 3 requests could be followed in parallel to get tickers starting with A, B, and C:

This pattern also naturally supports filter results by the non-sort field. For example, this request gets news articles that were published since April 22, 2021:

Rollout Plan

All new APIs going forward will have this standard pagination approach. Any new list APIs will take advantage of our query filter extension pattern as well. Where applicable we will rollout these patterns to existing APIs while maintaining backwards compatibility.

Summary

We are introducing a new pagination approach that will become the standard across our APIs going forward and query filter extensions that will increase the capabilities of our APIs to allow for more powerful searches. We are excited for you to start using our new API enhancements and we hope they make finding market data more accessible.

• 

  Ricky Barillas

  ricky

  Read More