Are notifications supported?

[sebelik]

According to the API reference, a Job configuration may specify one or more notifications to be sent when the job or task states change, however I wasn't able to get it to work.

The Job in question, based on a single container runnable, gets created (jobs.create returns 200) and runs successfully until completion but does not produce any Pub/Sub messages despite having notifications for all possible combinations of type/state configured:

{
    "taskGroups": [
        ...
    ],
    "allocationPolicy": {
        "instances": [
            {
                "instanceTemplate": "..."
            }
        ]
    },
    "logsPolicy": {
        "destination": "CLOUD_LOGGING"
    },
    "notifications": [
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "SCHEDULED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "QUEUED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "RUNNING"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "SUCCEEDED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "FAILED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "DELETION_IN_PROGRESS"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "TASK_STATE_CHANGED",
                "newTaskState": "ASSIGNED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "TASK_STATE_CHANGED",
                "newTaskState": "PENDING"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "TASK_STATE_CHANGED",
                "newTaskState": "RUNNING"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "TASK_STATE_CHANGED",
                "newTaskState": "SUCCEEDED"
            }
        },
        {
            "pubsubTopic": "...",
            "message": {
                "type": "TASK_STATE_CHANGED",
                "newTaskState": "FAILED"
            }
        }
    ]
}

The documentation is not clear on whether the messages will be published under the worker node's service account, the default Compute Engine service account or some other SA (possibly the Batch service's own SA?) but even with all audit logs enabled, no attempts to publish any messages are recorded, so this does not seem to be an IAM issue. Job/agent logs don't show any errors either.

Unfortunately, no examples - either here or on the product page - include notification configuration and with the service being in preview, I'm wondering if this feature is currently supported or if I've missed something?

[chapmanjacobd]

Are you using the full URL to define the pubsub topic? projects/{project}/topics/{topic}

I'm sure you are, but it would be more clear if you don't obfuscate unnecessarily

Also, it looks like both states are required to be specified:

        new_job_state = proto.Field(
            proto.ENUM,
            number=2,
            enum="JobStatus.State",
        )
        new_task_state = proto.Field(
            proto.ENUM,
            number=3,
            enum=task.TaskStatus.State,
        )

but it does seem kinda weird. I'm sure 40%+ of the combinations are edge-cases or impossible to reach. If this isn't an instance of mis-typing then it would be helpful if they publish a table of possible and recommended combinations.

            "message": {
                "type": "JOB_STATE_CHANGED",
                "newJobState": "RUNNING",
                "newTaskState": "RUNNING"
            }

[chapmanjacobd]

I'm mistaken. it's not possible to tell what counts as a well-formed message without external documentation

https://stackoverflow.com/questions/31801257/why-required-and-optional-is-removed-in-protocol-buffers-3