Async functions with map

The map is the easiest and most common collection function. It runs each element through an iteratee function and returns an array with the results.

The synchronous version that adds one to each element:

const arr = [1, 2, 3];

const syncRes = arr.map((i) => {
  return i + 1;
});

console.log(syncRes);
// 2,3,4

An async version needs to do two things. First, it needs to map every item to a Promise with the new value, which is what adding async before the function does.

And second, it needs to wait for all the Promises then collect the results in an Array. Fortunately, the Promise.all built-in call is exactly what we need for step 2.

This makes the general pattern of an async map to be Promise.all(arr.map(async (...) => ...)).

An async implementation doing the same as the sync one:

const arr = [1, 2, 3];

const asyncRes = await Promise.all(arr.map(async (i) => {
  await sleep(10);
  return i + 1;
}));

console.log(asyncRes);
// 2,3,4

Concurrency

The above implementation runs the iteratee function in parallel for each element of the array. This is usually fine, but in some cases, it might consume too much resources. This can happen when the async function hits an API or consumes too much RAM that it's not feasible to run too many at once.

While an async map is easy to write, adding concurrency controls is more involved. In the next few examples, we'll look into different solutions.

Batch processing

The easiest way is to group elements and process the groups one by one. This gives you control of the maximum amount of parallel tasks that can run at once. But since one group has to finish before the next one starts, the slowest element in each group becomes the limiting factor.

To make groups, the example below uses the groupBy implementation from Underscore.js. Many libraries provide an implementation and they are mostly interchangeable. The exception is Lodash, as its groupBy does not pass the index of the item.