Skip to content

[WIP] Fix exception when Continue-as-New and Termination happens at the same time #901

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
davidmrdavid wants to merge 6 commits into from
Closed

[WIP] Fix exception when Continue-as-New and Termination happens at the same time #901

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
10b0194
Fix exception when Continue-as-New and Termination happens at the sam…
fuifuiyong May 5, 2023
d955a72
add private version suffix
davidmrdavid May 5, 2023
97eac9a
add temporary version suffixes to allow private packages
davidmrdavid May 5, 2023
11c3dae
change version suffix
davidmrdavid May 11, 2023
db648c4
increase version
davidmrdavid May 16, 2023
ad52467
Merge branch 'main' into dajusto/patch-continue-as-new-and-terminate-…
davidmrdavid Aug 3, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions src/DurableTask.AzureServiceFabric/DurableTask.AzureServiceFabric.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,11 +41,14 @@
<PackageReference Include="Microsoft.ServiceFabric.Services" Version="3.3.617" />
<PackageReference Include="Microsoft.ServiceFabric" Version="6.4.617" />
<PackageReference Include="System.Collections.Immutable" Version="1.2.0" />
<PackageReference Include="Microsoft.Azure.DurableTask.Core" Version="2.13.0" />
</ItemGroup>

<!--
<ItemGroup>
<ProjectReference Include="..\DurableTask.Core\DurableTask.Core.csproj" />
</ItemGroup>
-->

<ItemGroup Condition="'$(Configuration)'=='Release'">
<Content Include="..\..\_manifest\**">
Expand Down
2 changes: 1 addition & 1 deletion src/DurableTask.AzureStorage/DurableTask.AzureStorage.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
<!-- The assembly version is only the major/minor pair, making it easier to do in-place upgrades -->
<AssemblyVersion>$(MajorVersion).$(MinorVersion).0.0</AssemblyVersion>
<!-- This version is used as the nuget package version -->
<Version>$(VersionPrefix)</Version>
<Version>$(VersionPrefix)-cntasnew-1</Version>
</PropertyGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net462'">
Expand Down
4 changes: 2 additions & 2 deletions src/DurableTask.Core/DurableTask.Core.csproj
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
<PropertyGroup>
<MajorVersion>2</MajorVersion>
<MinorVersion>13</MinorVersion>
<PatchVersion>0</PatchVersion>
<PatchVersion>1</PatchVersion>

<VersionPrefix>$(MajorVersion).$(MinorVersion).$(PatchVersion)</VersionPrefix>
<FileVersion>$(VersionPrefix).0</FileVersion>
Expand All @@ -27,7 +27,7 @@
<!-- The assembly version is only the major/minor pair, making it easier to do in-place upgrades -->
<AssemblyVersion>$(MajorVersion).$(MinorVersion).0.0</AssemblyVersion>
<!-- This version is used as the nuget package version -->
<Version>$(VersionPrefix)</Version>
<Version>$(VersionPrefix)-cntasnew-1</Version>
</PropertyGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net462'">
Expand Down
9 changes: 6 additions & 3 deletions src/DurableTask.Core/TaskOrchestrationDispatcher.cs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -805,7 +805,7 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem)
out bool continuedAsNew)
{
ExecutionCompletedEvent executionCompletedEvent;
continuedAsNew = (completeOrchestratorAction.OrchestrationStatus == OrchestrationStatus.ContinuedAsNew);
continuedAsNew = (completeOrchestratorAction.OrchestrationStatus == OrchestrationStatus.ContinuedAsNew) && (runtimeState.OrchestrationStatus != OrchestrationStatus.Terminated);
Copy link
Member

@cgillum cgillum Sep 5, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of making a change here, I think it would be better to make a change further upstream in OrchestrationRuntimeState.SetMarkerEvents. Instead of always throwing if we see that ExecutionCompletedEvent == null, we could add an additional check that allows transitioning from a ContinuedAsNew state to the Terminated state. I feel like this is a reasonable thing to allow since termination should be considered forceful.

I'd be fine going even further than this too. I don't see a strong reason to throw an exception if multiple completion events are found as long as we can make the expected outcome predictable. Ideally we'd simply log a warning if we encounter multiple completion events. I feel like failing with an exception is only useful when we don't have a reasonable way to procede.

if (completeOrchestratorAction.OrchestrationStatus == OrchestrationStatus.ContinuedAsNew)
{
executionCompletedEvent = new ContinueAsNewEvent(completeOrchestratorAction.Id,
Expand All @@ -819,7 +819,10 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem)
completeOrchestratorAction.FailureDetails);
}

runtimeState.AddEvent(executionCompletedEvent);
if (runtimeState.OrchestrationStatus != OrchestrationStatus.Terminated)
{
runtimeState.AddEvent(executionCompletedEvent);
}

this.logHelper.OrchestrationCompleted(runtimeState, completeOrchestratorAction);
TraceHelper.TraceInstance(
Expand Down Expand Up @@ -1076,4 +1079,4 @@ public void Release()
}
}
}
}
}