is flowControl.maxMessages option honored in 0.14.x to limit calls to handler

[evilpacket]

Environment details

• OS: Linux (ubuntu)
• Node.js version: 8.5.0
• npm version: 5.3.0
• google-cloud-node version: @google-cloud/pubsub 0.14.2

Steps to reproduce

Is it possible to get or limit the inflight messages using the new 0.14.x pubsub library?

I'm calling something like the following
this.subscription = this.topic.subscription(this.options.name, this.options.subscription);

Even if I get a topic.subscription with the following options it appears to call the my handler with as many messages in in the queue.

{
  flowControl: {
    maxMessages: 1
  }
}

[callmehiphop]

This happens because the PubSub API doesn't allow us to limit how many results it sends us, so we use those numbers as an indicator to stop making requests to the API if they are hit/exceeded. Unfortunately in this case its easy to exceed.

@lukesneeringer should we only emit the specified number of messages listed in maxMessages and store the rest in memory while modifying the ack deadline along the way?

[callmehiphop]

/cc @jganetsk

[evilpacket]

Thanks @callmehiphop seems like this is a behavior change from the 0.13 client where I could specifically do 1 thing at a time if I wanted. For my specific use case I only can do one thing at a time due to the intensity of the jobs I'm doing.

[jganetsk]

should we only emit the specified number of messages listed in maxMessages and store the rest in memory while modifying the ack deadline along the way?

We should not do this. The point of flow control is to prevent the process from dying because it ran out of memory. Storing messages in memory defeats that purpose, and it's not very useful to the user to have a message in memory but not to emit it.

Is it possible to get or limit the inflight messages using the new 0.14.x pubsub library?

It sounds like the user wants flow control to be on, not off. But it's supposed to be on by default, so is there a bug in flow control?

[callmehiphop]

It sounds like the user wants flow control to be on, not off. But it's supposed to be on by default, so is there a bug in flow control?

Maybe? @jganetsk I'm a little lost on how we are supposed to process payloads with message sizes that exceed flow control limits. As it is today all the flowControl knobs do are control when we pause/resume connections. If we don't want overflow messages in memory should we be nacking them?

[jganetsk]

We have a lower bound on flow control limits, iirc, can you take a look at what those are? I think it's ok to overflow by 1 message (which means we can only overflow by up to 10MB, since that's the max message size in Pub/Sub), but beyond that we probably should nack any additional overflowing messages.

[callmehiphop]

I don't think there are bounds on any of the flow control limits. We have defaults of

• maxBytes - 20% of available memory
• maxMessages - Infinity

But I do not believe that there were any bounds specified in the design doc.

[jganetsk]

I might have been thinking about limits on batching settings. Well regardless, my point about overflowing 1 message still holds and nacking otherwise still holds.

[callmehiphop]

@jganetsk sounds good, thanks for the input!

[evilpacket]

@jganetsk thanks for taking the time to comment on this issue. I'm not sure I follow completely with what you and @callmehiphop were discussing so maybe it's going to be resolved.... I wanted to clarify what i wanted to happen. I want maxMessages on and set to emit 1 at a time. It doesn't appear to honor that.

Thank you again.

[callmehiphop]

@evilpacket I just cut a new release (0.14.4) that should make maxMessages behave as you intend. Thanks for reporting this!