Easy Concurrent Promises with Async Await


You don't need heavyweight dependencies like async-promises or highland to throttle concurrent promises. These 9 lines of magic form an easy reusable pattern.

While packages like async-promises or highland are great, sometimes it's better to use your own pattern to solve your specific problem.

The following is a pattern I've been using for a while, it's clean, concise, and you can easily bend it to your will.

const series = async (workers) => {
  while (workers.length) await workers.pop()()
}
const concurrent = (workers, concurrency) => {
  const split = []
  const count = Math.ceil(workers.length / concurrency)
  while (workers.length) split.push(workers.splice(0, count))
  return Promise.all(split.map((item) => series(item)))
}

The series function is pretty self explanatory, pass it an array of workers and it will work through them in series.

The concurrent function simply splits an array of workers into separate series.

You could call it like this:

const records = getRecordsToSave()
concurrent(records.map((record) => dbTable.save(record)), 2)

The key here is that a worker should be a function which returns a promise when called.

This very simple structure does have some shortcomings, for example there's no error handling, and there's no way to collect results. This kind of functionality can be implemented within your worker itself.

const records = getRecordsToSave()
const results = []
concurrent(records.map((record) => {
  return dbTable.save(record)
  .catch((e) => handleError(e))
  .then((res) => results.push(res)
}), 2)