[Bug] Signal caused `condition` to fail with `CancelledFailure` on `1.14.0`

[justinadkins]

What are you really trying to do?

We bumped our Temporal dependencies to 1.14.0 today from 1.13.1 and upon deploying started seeing workflows fail due to the following error:

CancelledFailure: Workflow cancelled
    at RootCancellationScope.cancel (/tmp/build_d69c691a/node_modules/.pnpm/@temporalio+workflow@1.14.0/node_modules/@temporalio/workflow/src/cancellation-scope.ts:264:16)
    at /tmp/build_d69c691a/node_modules/.pnpm/@temporalio+workflow@1.14.0/node_modules/@temporalio/workflow/src/workflow.ts:1171:36

Rolling back to 1.13.1 resolves the issue. We've reset the effected workflows to the signal event and they now run correctly. A race condition seems to have been introduced since we do observe some of our workflows succeeding with a signal and condition.

Describe the bug

We started seeing phantom CancelledFailure errors surface in our workflows after updating to 1.14.0 that seem to be coming from condition. Nothing else about our code has changed. An example of the code where this appears to be failing:

Minimal Reproduction

Unable to reproduce locally so far. We have over 30 workflows in production that have experienced this.

Environment/Versions

• OS and processor: Linux
• Temporal Version: 1.14.0
• Are you using Docker or Kubernetes or building Temporal from source? From source

[yosefrev]

We’re experiencing this as well ➕

[yardenli]

We also started experiencing it as well with updates.
We have an update with timeout, and experienced it in some workflows right after the update was accepted, and in some after the update reached it's timeout.
It is important to mention that it happened in some workflows while other succeeded, and we couldn't find a clear pattern.

[chris-olszewski]

For those affected by this:

• Have you observed this with 1.13.2 or exclusively with 1.14.0?
• Do you have interceptors setup such as @temporalio/interceptors-opentelemetry?

[yardenli]

@chris-olszewski we observed it exclusively with 1.14.0. After going back to 1.13.2 we observed no issues.
We do have interceptor set up using @temporalio/interceptors-opentelemetry (we are using OpenTelemetryWorkflowClientInterceptor).

[justinadkins]

@chris-olszewski We only observe this on 1.14.0. We are not using an interceptor. However, for activities we have this custom wrapper used for creating spans with Datadog.

[Stereobit]

Same issue here, no interceptors