PubSub: Unable to create a pull subscription to a topic where acknowledged messages are retained

[braxtonb]

Environment details

• OS: macOS Sierra version 10.12.2
• Node.js version: v8.2.1
• npm version: 4.6.1
• google-cloud-node version: 0.14.2

Steps to reproduce

1. require google-cloud
2. initialize pubsubClient
3. create subscription to a topic with config option retainAckedMessages set to true (see code below)
4. push 5 messages to a topic
5. pull and acknowledge the 5 messages from the topic successfully the first time
6. in a second attempt to pull the 5 messages again, 0 are retrieved
7. get the metadata for the subscription created to the topic in step 3, and retainAckedMessages is set to false (and messageRetentionDuration is set to null even though it should default to 7 days)

Below is the code I use to create a subscription

const PubSub = require('@google-cloud/pubsub');

const Keys = require('./json/pubsub-editor-keys');

const projectId = 'my-project-id';
let pubsubClient;
const subscriptionName = 'my-subscription-name';
const topicName = 'my-topic';

const initializePubSubClient = (client_email, private_key) => {
  const options = {
    projectId,
    credentials: {
      client_email,
      private_key
    }
  };
  pubsubClient = PubSub(options);
}

const createSubscription = (topic, name) => {
  const options = {
    retainAckedMessages: true
  };

  console.log('createSubscription called');

  pubsubClient
    .createSubscription(topic, name, options)
    .then((data) => {
      const subscription = data[0];
      const apiResponse = data[1];
      console.log('subscription', subscription);
      console.log('apiResponse', apiResponse);
    })
    .catch((err) => {
      console.log('error', err);
    });
}

initializePubSubClient(Keys.client_email, Keys.private_key);
createSubscription(topicName, subscriptionName);

• I would like to create a pull subscription to a topic where acknowledged messages are retained for a set amount of time.
• I have tried accomplishing this by setting retainAckedMessages to true and leaving messageRetentionDuration as the default 7 days; however, I have not had success in initializing a subscription with those option values.
• Regardless of the options I use to initialize the subscription, acknowledged messages are removed from the subscribed topic's stored messages.
• Related to pubsub: implement message retention #2240

[callmehiphop]

@braxtonb Out of curiosity when you log out the subscription and response upon creation, do you see retainAckedMessages set initially? I ask because in my local testing it seems that the API is honoring the option.

[braxtonb]

@callmehiphop Yes, when I log out the subscription and response upon creation I see retainAckedMessages set initially in the response. However, it is set to false.

Here's a sample response I see upon creation:

{
  "metadata": {
    "name": "projects/my-project-id/subscriptions/my-subscription-name",
    "topic": "projects/my-project-id/topics/my-topic",
    "pushConfig": { "pushEndpoint": "", "attributes": {} },
    "ackDeadlineSeconds": 10,
    "retainAckedMessages": false,
    "messageRetentionDuration": null,
    "labels": {} 
  }
}

[callmehiphop]

I can't actually reproduce this locally, could you try testing against the latest version (0.14.3) and see if your issue still persists?

Also, after creating your subscription have you tried setting retainAckedMessages using setMetadata? Do you get any errors or is it the same result?

[braxtonb]

I have tested against the latest version (0.14.3) and the issue persists.

After creating the subscription I have tried setting retainAckedMessages using setMetadata and I get errors.

Below is the code I use to set the metadata

const PubSub = require('@google-cloud/pubsub');

const Keys = require('./json/pubsub-editor-keys');

const projectId = 'my-project-id';
let pubsubClient;
const subscriptionName = 'my-subscription-name';
const topicName = 'my-topic';

const initializePubSubClient = (client_email, private_key) => {
  const options = {
    projectId,
    credentials: {
      client_email,
      private_key
    }
  };
  pubsubClient = PubSub(options);
}

const setMetadata = (name) => {
  const metadata = {
    retainAckedMessages: true
  };

  pubsubClient
    .subscription(name)
    .setMetadata(metadata)
    .then(res => console.log('res', res))
    .catch(err => console.log('error', err));
}

initializePubSubClient(Keys.client_email, Keys.private_key);
setMetadata(subscriptionName);

And the errors I get are

error Error: Illegal value for subscriptionvalue element of type message: string (object expected)
    at fail (/path/to/node_modules/protobufjs/dist/protobuf.js:1817:23)
    at Element.ProtoBuf.Reflect.ElementPrototype.verifyValue (/path/to/node_modules/protobufjs/dist/protobuf.js:1913:25)
    at Field.ProtoBuf.Reflect.FieldPrototype.verifyValue (/path/to/node_modules/protobufjs/dist/protobuf.js:3499:33)
    at MessagePrototype.set (/path/to/node_modules/protobufjs/dist/protobuf.js:2493:59)
    at MessagePrototype.set (/path/to/node_modules/protobufjs/dist/protobuf.js:2483:38)
    at Message (/path/to/node_modules/protobufjs/dist/protobuf.js:2411:34)
    at serialize (/path/to/node_modules/grpc/src/node/src/protobuf_js_5_common.js:66:23)
    at ServiceClient.Client.makeUnaryRequest (/path/to/node_modules/grpc/src/node/src/client.js:515:17)
    at apply (/path/to/node_modules/lodash/lodash.js:499:17)
    at ServiceClient.wrapper [as updateSubscription] (/path/to/node_modules/lodash/lodash.js:5356:16)

[callmehiphop]

@jganetsk is retainAckedMessages still honored? It's documented in our local proto files. However I've noticed it is missing from the upstream docs.

[callmehiphop]

@braxtonb I've just been informed that retainAckedMessages is still an Alpha feature and is not yet generally available. So what you are experiencing appears to be the expected behavior. I'm going to close this issue, but I will comment on it if I learn any new info.