Skip to content

Conversation

@dvisztempacct
Copy link

Wasted a good deal of my time trying to figure out why parallel was not performing as efficiently as I expected.

Hopefully this will save others from the same tar pit!

@richardscarrott
Copy link

@dvisztempacct I wonder if you could share any more insight into the difference between parallel and merge?... I have a scenario where I don't care about order, but I do want to constrain the amount of parallelism. e.g.

_(nodeReadableStream)
   .map(wrapAsync(async (obj) => {
          const transformedObj = await transformObj(obj);
          return transformedObj;
   }))
   .parallel(100) // I want to limit the concurrent reads to 100, but would rather it was pushed to the writable stream ASAP
   .pipe(nodeWritableStream); // Doesn't care what order the `transformedObj`s are written in

I was wondering if Highland.prototype.merge would:

a) honour backpressure, or is it like turning on the tap full blast?
b) prevent Highland.prototype.map from being called over n times in parallel?

@richardscarrott
Copy link

I think Highland.prototype.mergeWithLimit is exactly what I'm after -- I had glossed over it for some reason 🤷‍♂️

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants