Pagination - Handling Large Data Sets

The customizable nature of GraphQL makes it possible to fetch vast amounts of data in a single query. While this is a useful feature, there are instances in which it is not possible or desirable to receive all the data at once. Pagination allows you to fetch data in manageable batches.

Recommended Batch Size

There is a limit to how many records you can request in a single batch. We strongly recommend getting 1000 records with each API request to prevent hitting this limit. If you attempt to request more than the limit, you will receive a warning indicating that the amount of records returned has been reduced to the limit.

Cursor Based Pagination

Our GraphQL API implements cursor-based pagination. This type of pagination allows you to specify how many pieces of data you want to read, and where you want to start reading from. Each record has a cursor, which is an opaque string identifying the record's place in the greater dataset.

Queries that offer pagination will have the following two parameters that you can pass in:

• first: Specifies how many records to fetch.
• after: Used to request the next result set from the query beginning with the data record directly following the cursor you put here.

Notice in the example below, an object called pageInfo within the query. Here you can utilize the fields endCursor and hasNextPage.

The endCursor field is the cursor associated with the last record fetched from your query. Referencing the example seen below we are asking for the first 10 records. The pageInfo.endCursor will have the cursor for that 10th record that you would want to use as the value for the after parameter, to fetch the next 10 records.

The hasNextPage field is a boolean field that lets you know if there are more data records you could fetch after the current endCursor. The endCursor will still be populated if hasNextPage is false.

Another valuable field found below is the totalCount. This shows you the total number of data records available for this request and helps you know how many batches you can fetch. The totalCount can differ in each request when you are paging through batches of data.

Let's walk through an example with the channels query, fetching 10 records at a time. We start with the following query, and get the corresponding result.

query {
	channels(first: 10) {
	  totalCount
	  pageInfo {
	    endCursor
	    hasNextPage
	  }
	  nodes {
	    id
	  }
	}
}

{
  "data": {
    "channels": {
      "totalCount": 1710,
      "pageInfo": {
        "endCursor": "MTcxOTA3",
        "hasNextPage": true
      },
      "nodes": [
        {
          "id": "51427947-12d9-4bbf-aae8-5258c596cdd7"
        },
        ...9 more records
      ]
    }
  }
}

Notice the pageInfo.endCursor returned in the response, as this becomes the after of our next query.

query {
	channels(
	  first: 10, 
	  after: "MTcxOTA3") 
	{
	  totalCount
	  pageInfo {
	    endCursor
	    hasNextPage
	  }
	  nodes {
	    id
	  }
	}
}

{
  "data": {
    "channels": {
      "totalCount": 1710,
      "pageInfo": {
        "endCursor": "MTcxMzIx",
        "hasNextPage": true
      },
      "nodes": [
        {
          "id": "9c641b7a-3931-4e2f-bbf4-0c57a7582c66"
        },
        ...9 more records
      ]
    }
  }
}

Edges

Most use cases are well handled by just using the pageInfo.endCursor to paginate through the records, however if you need to start your next batch of data at a record that is different than the last record fetched you will need the cursor field.

Each record has a cursor field that can be used in the after query parameter, allowing you to paginate over data that has already been returned. In order to view this field, you will need to utilize the edges object. Using edges, the nodes that are returned, get individually wrapped with a cursor field.

The edges object has the following fields that can be fetched:

• cursor: The cursor associated with that data record.
• node: The object where you specify the fields you would like to fetch about this data record.

The node object will contain the same fields that you would have used in the nodes object.

In the following example, you will notice another request to the channels query and that the nodes object has been replaced by edges and displays the cursor and the requested field information with the node object.

query {
	channels(first: 10) {
	  totalCount
	  pageInfo {
	    endCursor
	    hasNextPage
	  }
	  edges {
	    cursor
	    node{
	      id
	    }
	  }
	}
}

{
  "data": {
    "channels": {
      "totalCount": 1710,
      "pageInfo": {
        "endCursor": "MTcxOTA3",
        "hasNextPage": true
      },
      "edges": [
        {
          "cursor": "MTcx9DA7"
          "node": {
            "id": "51427947-12d9-4bbf-aae8-5258c596cdd7"
          }
        },
        ...9 more records
      ]
    }
  }
}

Additional Resources

(See Complete Connection Model)