Introducing New API Pagination Patterns Using Cursors and Query Filter Extensions
Apr 28, 2021
As we have been writing new APIs that bring you more and better reference data, we have also been thinking of ways to improve the user experience of our APIs. Starting with our Reference Tickers API, we are introducing our new pagination model and query filter extensions. The goals of introducing this pattern are to maintain the quality of our API response times, provide a consistent interface for pagination between APIs, allow for incremental performance improvements without introducing breaking changes, and to provide a mechanism for fetching pages concurrently for those who need it.
Several of our APIs have pagination capabilities that allow users to make large requests and page through the different results. Currently, not all APIs share the same query interface, so we are introducing a single standard for paginating Polygon.io APIs.
An API that supports our standard pagination will return top level attributes in its response body if there are more pages of data to retrieve. Those attributes will be
signals that there is another page of data to retrieve while
signals that you are in the middle of paginating through responses and
we are able to show you the previous page. If
is not present in the response then there are no more pages to query over.
Not every API will have the ability to page backwards (
) but those that do will be noted in our API docs.
Our APIs will use an opaque query parameter (`cursor`) to know how to fetch the next or previous page. This allows us to obscure a particular API’s low-level paging mechanism from the user’s request so that we can make future improvements to pagination queries without breaking the API’s public contract.
As with all of our APIs, we use the apiKey query param to authenticate requests. In order to not make our response payloads contain sensitive information we will exclude the API key from the values in next/previous_url, requiring the user to append that information to the value of next/previous_url. In the future, we plan to allow API Keys to be sent in the Authorization Header which will render this additional step unnecessary.
A request to our Reference Tickers API at
could get the following response
"name": "Agilent Technologies Inc."
"name": "Alcoa Corporation"
To get the next page of data, you would make an HTTP GET on the value of
appending the API key query param:
When the `cursor` query param is present, only the apiKey query param will be respected by the API, all others will be discarded.
Query Filter Extensions
We are also introducing a new query pattern to allow for more granular searches. We are extending our model with selectors of the form <field>.<operator>=<value>. We will start by offering the following operators on certain fields:
less than or equal
greater than or equal
> is still a valid query and still represents an equality check.
will only return results where the
attribute equals exactly
All fields that support query extensions will be noted in our REST Docs. For an example, check out our recently updated Ticker News API.
Examples and Usage
This new query pattern opens the door for some interesting use cases. We've been getting a lot of feedback asking for a way to be able to page through the new Tickers API with parallel workers. With these new filter extensions, you can break up your request into discrete parts and do just that. For example, these 3 requests could be followed in parallel to get tickers starting with A, B, and C:
This pattern also naturally supports filter results by the non-sort field. For example, this request gets news articles that were published since April 22, 2021:
All new APIs going forward will have this standard pagination approach. Any new list APIs will take advantage of our query filter extension pattern as well. Where applicable we will rollout these patterns to existing APIs while maintaining backwards compatibility.
We are introducing a new pagination approach that will become the standard across our APIs going forward and query filter extensions that will increase the capabilities of our APIs to allow for more powerful searches. We are excited for you to start using our new API enhancements and we hope they make finding market data more accessible.