Paginating with async generators

Many APIs limit how many results they return for a single call. For example, the functions of the AWS SDK that return lists are paginated operations. That means that you get a limited number of elements in one call along with a token to get the next batch. This can be a problem if you are not aware of it, as you might get all the elements during development but your function might break in the future.

But while getting the first batch is just await lambda.listFunctions().promise(), paginating until all the pages are retrieved requires more work.

Let's see how to make an await-able structure that reliable gets all the elements from an AWS API!

Pagination

In the case of Lambda functions, the lambda.listFunctions call returns a structure with a list of all your lambdas if you don't have too many of them (at most 50):

const functions = await lambda.listFunctions().promise();

Result:

{
  Functions: [...]
}

To simulate a paginated case, you can set the MaxItems parameter:

const functions = await lambda.listFunctions({
  MaxItems: 1
}).promise();

This time a NextMarker is also returned, indicating there are more items:

{
  Functions: [...],
  NextMarker: ...
}

To get the next batch, provide the last marker:

const functions = await lambda.listFunctions({
  MaxItems: 1,
  Marker: functions.NextMarker,
}).promise();

Then do so until no NextMarker is returned.

A solution with async generators

Async generators are a relatively new feature of JavaScript. They are like traditional generator functions, but they are async, meaning you can await inside them.

To collect all the Lambda functions no matter how many calls are needed, use: