diff --git a/README.md b/README.md index dc4360635..0451cdda2 100644 --- a/README.md +++ b/README.md @@ -25,7 +25,13 @@ The core programming model for the Durable Task Framework is contained in the [D ## Learning more -The associated wiki contains more details about the framework and how it can be used: https://github.com/Azure/durabletask/wiki. You can also find great information in [this blog series](https://abhikmitra.github.io/blog/durable-task/). In some cases, the [Durable Functions documentation](https://docs.microsoft.com/en-us/azure/azure-functions/durable/) can actually be useful in learning things about the underlying framework, although not everything will apply. Lastly, you can watch a video with some of the original maintainers in [this Channel 9 video](https://channel9.msdn.com/Shows/On-NET/Building-workflows-with-the-Durable-Task-Framework). +There are several places where you can learn more about this framework. Note that some are external and not owned by Microsoft: + +- [This repo's wiki](https://github.com/Azure/durabletask/wiki), which contains more details about the framework and how it can be used. +- The following blog series contains useful information: https://abhikmitra.github.io/blog/durable-task/ +- Several useful samples are available here: https://github.com/kaushiksk/durabletask-samples +- You can watch a video with some of the original maintainers in [Building Workflows with the Durable Task Framework](https://learn.microsoft.com/shows/on-net/building-workflows-with-the-durable-task-framework). +- In some cases, the [Azure Durable Functions documentation](https://learn.microsoft.com/azure/azure-functions/durable/) can actually be useful in learning things about the underlying framework, although not everything will apply. ## Development Notes @@ -35,6 +41,8 @@ Unit tests also require [Azure Storage Emulator](https://docs.microsoft.com/azur > Note: While it's possible to use in tests a real Azure Storage account it is not recommended to do so because many tests will fail with a 409 Conflict error. This is because tests delete and quickly recreate the same storage tables, and Azure Storage doesn't do well in these conditions. If you really want to change Azure Storage connection string you can do so via the **StorageConnectionString** app.config value in the test project, or by defining a **DurableTaskTestStorageConnectionString** environment variable. + diff --git a/azure-pipelines-release.yml b/azure-pipelines-release.yml index aa2f0cfec..f17bd9320 100644 --- a/azure-pipelines-release.yml +++ b/azure-pipelines-release.yml @@ -1,4 +1,6 @@ -trigger: none +trigger: +- main + pr: none pool: @@ -6,6 +8,21 @@ pool: demands: - ImageOverride -equals MMS2022TLS +variables: + + - name: VersionSuffix + # The `Build.Reason` env var gets populated with `IndividualCI` on an automatic run of the CI, + # such as when a commit is made to `main`. If the CI is manually run, it will get populated with + # "Manual". For more details on these `Build.X` vars, see: https://learn.microsoft.com/en-us/azure/devops/pipelines/build/variables?view=azure-devops&tabs=yaml#build-variables-devops-services + ${{ if eq(variables['Build.Reason'], 'IndividualCI') }}: + # The `Build.BuildNumber` env var is an automatically-generated "build ID" + # for any given ADO run. It is usually the date format yyddmm. where + # `yyddmm` is a date formatter, and is a daily counter in case of multiple + # builds on the same date. + value: 'ci.$(Build.BuildNumber)' + ${{ else }}: + value: '' + steps: # Start by restoring all the dependencies. This needs to be its own task # from what I can tell. We specifically only target DurableTask.AzureStorage diff --git a/docs/telemetry/traces/getting-started.md b/docs/telemetry/traces/getting-started.md new file mode 100644 index 000000000..6ba72e786 --- /dev/null +++ b/docs/telemetry/traces/getting-started.md @@ -0,0 +1,26 @@ +# Getting Started - Distributed Tracing + +> ⚠ Important: durable task distributed tracing is currently [experimental](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/document-status.md). The schema is subject to changes until it is marked as stable. These changes may occur in any package update. + +> ⚠ Important: this guide only applies DurableTask users. For Durable Functions, please see [here](https://github.com/Azure/azure-functions-durable-extension/blob/dev/samples/distributed-tracing/v2/DistributedTracingSample/README.md) + +Distributed tracing in DurableTask uses the `ActivitySource` approach, it is both OpenTelemetry and Application Insights compatible. + +## OpenTelemetry + +Add the `"DurableTask.Core"` source to the OTel trace builder. + +``` CSharp +Sdk.CreateTracerProviderBuilder() + .AddSource("DurableTask.Core") + .Build() +``` + +See [sample](../../../samples/DistributedTraceSample/OpenTelemetry) + +## Application Insights + +1. Add reference to [Microsoft.Azure.DurableTask.ApplicationInsights](https://www.nuget.org/packages/Microsoft.Azure.DurableTask.ApplicationInsights) +2. Add the `DurableTelemetryModule` to AppInsights: `services.TryAddEnumerable(ServiceDescriptor.Singleton());` + +See [sample](../../../samples/DistributedTraceSample/ApplicationInsights) diff --git a/samples/DistributedTraceSample/OpenTelemetry/images/ApplicationInsightsExporter.png b/samples/DistributedTraceSample/OpenTelemetry/images/ApplicationInsightsExporter.png index cd35fda8c..60be93c25 100644 Binary files a/samples/DistributedTraceSample/OpenTelemetry/images/ApplicationInsightsExporter.png and b/samples/DistributedTraceSample/OpenTelemetry/images/ApplicationInsightsExporter.png differ diff --git a/samples/DistributedTraceSample/OpenTelemetry/images/ZipkinExporter.png b/samples/DistributedTraceSample/OpenTelemetry/images/ZipkinExporter.png index 9f3e2049e..bc2f23ed1 100644 Binary files a/samples/DistributedTraceSample/OpenTelemetry/images/ZipkinExporter.png and b/samples/DistributedTraceSample/OpenTelemetry/images/ZipkinExporter.png differ diff --git a/samples/DurableTask.Samples/DurableTask.Samples.csproj b/samples/DurableTask.Samples/DurableTask.Samples.csproj index 36e285423..25a8f4980 100644 --- a/samples/DurableTask.Samples/DurableTask.Samples.csproj +++ b/samples/DurableTask.Samples/DurableTask.Samples.csproj @@ -11,7 +11,6 @@ - diff --git a/src/DurableTask.AzureServiceFabric/DurableTask.AzureServiceFabric.csproj b/src/DurableTask.AzureServiceFabric/DurableTask.AzureServiceFabric.csproj index 0a44c9b8a..f283ca42d 100644 --- a/src/DurableTask.AzureServiceFabric/DurableTask.AzureServiceFabric.csproj +++ b/src/DurableTask.AzureServiceFabric/DurableTask.AzureServiceFabric.csproj @@ -13,17 +13,16 @@ Orchestration message and runtime state is stored in Azure Service Fabric reliable collections. Microsoft AnyCPU;x64 + $(NoWarn);NU5104 - - diff --git a/src/DurableTask.AzureServiceFabric/Remote/RemoteOrchestrationServiceClient.cs b/src/DurableTask.AzureServiceFabric/Remote/RemoteOrchestrationServiceClient.cs index 04f85e8e4..51db1edda 100644 --- a/src/DurableTask.AzureServiceFabric/Remote/RemoteOrchestrationServiceClient.cs +++ b/src/DurableTask.AzureServiceFabric/Remote/RemoteOrchestrationServiceClient.cs @@ -378,7 +378,7 @@ private async Task ExecuteRequestWithRetriesAsync(string in return response; } } - catch (Exception ex) when (ex is SocketException || ex is WebException) + catch (Exception ex) when (ex is SocketException || ex is WebException || ex is HttpRequestException) { exception = ex; } diff --git a/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs b/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs index 951160a70..1f8b912de 100644 --- a/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs +++ b/src/DurableTask.AzureStorage/AzureStorageOrchestrationService.cs @@ -31,6 +31,7 @@ namespace DurableTask.AzureStorage using DurableTask.AzureStorage.Storage; using DurableTask.AzureStorage.Tracking; using DurableTask.Core; + using DurableTask.Core.Entities; using DurableTask.Core.Exceptions; using DurableTask.Core.History; using DurableTask.Core.Query; @@ -44,7 +45,8 @@ public sealed class AzureStorageOrchestrationService : IOrchestrationServiceClient, IDisposable, IOrchestrationServiceQueryClient, - IOrchestrationServicePurgeClient + IOrchestrationServicePurgeClient, + IEntityOrchestrationService { static readonly HistoryEvent[] EmptyHistoryEventList = new HistoryEvent[0]; @@ -280,6 +282,51 @@ public BehaviorOnContinueAsNew EventBehaviourForContinueAsNew /// public int TaskOrchestrationDispatcherCount { get; } = 1; + #region IEntityOrchestrationService + + EntityBackendProperties IEntityOrchestrationService.EntityBackendProperties + => new EntityBackendProperties() + { + EntityMessageReorderWindow = TimeSpan.FromMinutes(this.settings.EntityMessageReorderWindowInMinutes), + MaxEntityOperationBatchSize = this.settings.MaxEntityOperationBatchSize, + MaxConcurrentTaskEntityWorkItems = this.settings.MaxConcurrentTaskEntityWorkItems, + SupportsImplicitEntityDeletion = false, // not supported by this backend + MaximumSignalDelayTime = TimeSpan.FromDays(6), + UseSeparateQueueForEntityWorkItems = this.settings.UseSeparateQueueForEntityWorkItems, + }; + + EntityBackendQueries IEntityOrchestrationService.EntityBackendQueries + => new EntityTrackingStoreQueries( + this.messageManager, + this.trackingStore, + this.EnsureTaskHubAsync, + ((IEntityOrchestrationService)this).EntityBackendProperties, + this.SendTaskOrchestrationMessageAsync); + + Task IEntityOrchestrationService.LockNextOrchestrationWorkItemAsync( + TimeSpan receiveTimeout, + CancellationToken cancellationToken) + { + if (!this.settings.UseSeparateQueueForEntityWorkItems) + { + throw new InvalidOperationException("Internal configuration is inconsistent. Backend is using single queue for orchestration/entity dispatch, but frontend is pulling from individual queues."); + } + return this.LockNextTaskOrchestrationWorkItemAsync(false, cancellationToken); + } + + Task IEntityOrchestrationService.LockNextEntityWorkItemAsync( + TimeSpan receiveTimeout, + CancellationToken cancellationToken) + { + if (!this.settings.UseSeparateQueueForEntityWorkItems) + { + throw new InvalidOperationException("Internal configuration is inconsistent. Backend is using single queue for orchestration/entity dispatch, but frontend is pulling from individual queues."); + } + return this.LockNextTaskOrchestrationWorkItemAsync(entitiesOnly: true, cancellationToken); + } + + #endregion + #region Management Operations (Create/Delete/Start/Stop) /// /// Deletes and creates the neccesary Azure Storage resources for the orchestration service. @@ -628,9 +675,18 @@ static TaskHubInfo GetTaskHubInfo(string taskHub, int partitionCount) #region Orchestration Work Item Methods /// - public async Task LockNextTaskOrchestrationWorkItemAsync( + public Task LockNextTaskOrchestrationWorkItemAsync( TimeSpan receiveTimeout, CancellationToken cancellationToken) + { + if (this.settings.UseSeparateQueueForEntityWorkItems) + { + throw new InvalidOperationException("Internal configuration is inconsistent. Backend is using separate queues for orchestration/entity dispatch, but frontend is pulling from single queue."); + } + return LockNextTaskOrchestrationWorkItemAsync(entitiesOnly: false, cancellationToken); + } + + async Task LockNextTaskOrchestrationWorkItemAsync(bool entitiesOnly, CancellationToken cancellationToken) { Guid traceActivityId = StartNewLogicalTraceScope(useExisting: true); @@ -644,7 +700,7 @@ public async Task LockNextTaskOrchestrationWorkItemAs try { // This call will block until the next session is ready - session = await this.orchestrationSessionManager.GetNextSessionAsync(linkedCts.Token); + session = await this.orchestrationSessionManager.GetNextSessionAsync(entitiesOnly, linkedCts.Token); if (session == null) { return null; @@ -1634,7 +1690,7 @@ public async Task CreateTaskOrchestrationAsync(TaskMessage creationMessage, Orch // An instance in this state already exists. if (this.settings.ThrowExceptionOnInvalidDedupeStatus) { - throw new InvalidOperationException($"An Orchestration instance with the status {existingInstance.State.OrchestrationStatus} already exists."); + throw new OrchestrationAlreadyExistsException($"An Orchestration instance with the status {existingInstance.State.OrchestrationStatus} already exists."); } return; @@ -2064,6 +2120,7 @@ private static OrchestrationInstanceStatusQueryCondition ToAzureStorageCondition TaskHubNames = condition.TaskHubNames, InstanceIdPrefix = condition.InstanceIdPrefix, FetchInput = condition.FetchInputsAndOutputs, + ExcludeEntities = condition.ExcludeEntities, }; } diff --git a/src/DurableTask.AzureStorage/AzureStorageOrchestrationServiceSettings.cs b/src/DurableTask.AzureStorage/AzureStorageOrchestrationServiceSettings.cs index 941d360ae..3af54ab89 100644 --- a/src/DurableTask.AzureStorage/AzureStorageOrchestrationServiceSettings.cs +++ b/src/DurableTask.AzureStorage/AzureStorageOrchestrationServiceSettings.cs @@ -85,6 +85,12 @@ public class AzureStorageOrchestrationServiceSettings /// public int MaxConcurrentTaskOrchestrationWorkItems { get; set; } = 100; + /// + /// Gets or sets the maximum number of entity operation batches that can be processed concurrently on a single node. + /// The default value is 100. + /// + public int MaxConcurrentTaskEntityWorkItems { get; set; } = 100; + /// /// Gets or sets the maximum number of concurrent storage operations that can be executed in the context /// of a single orchestration instance. @@ -234,5 +240,31 @@ internal LogHelper Logger return this.logHelper; } } + + /// + /// Gets or sets the limit on the number of entity operations that should be processed as a single batch. + /// A null value indicates that no particular limit should be enforced. + /// + /// + /// Limiting the batch size can help to avoid timeouts in execution environments that impose time limitations on work items. + /// If set to 1, batching is disabled, and each operation executes as a separate work item. + /// + /// + /// A positive integer, or null. + /// + public int? MaxEntityOperationBatchSize { get; set; } = null; + + /// + /// Gets or sets the time window within which entity messages get deduplicated and reordered. + /// If set to zero, there is no sorting or deduplication, and all messages are just passed through. + /// + public int EntityMessageReorderWindowInMinutes { get; set; } = 30; + + /// + /// Whether to use separate work item queues for entities and orchestrators. + /// This defaults to false, to avoid issues when using this provider from code that does not support separate dispatch. + /// Consumers that require separate dispatch (such as the new out-of-proc v2 SDKs) must set this to true. + /// + public bool UseSeparateQueueForEntityWorkItems { get; set; } = false; } } diff --git a/src/DurableTask.AzureStorage/DurableTask.AzureStorage.csproj b/src/DurableTask.AzureStorage/DurableTask.AzureStorage.csproj index b99bbab9a..2c7ecf02d 100644 --- a/src/DurableTask.AzureStorage/DurableTask.AzureStorage.csproj +++ b/src/DurableTask.AzureStorage/DurableTask.AzureStorage.csproj @@ -30,7 +30,13 @@ $(VersionPrefix).$(FileVersionRevision) $(MajorVersion).$(MinorVersion).0.0 - + + + + + $(VersionPrefix) + + $(VersionPrefix)-$(VersionSuffix) @@ -45,7 +51,7 @@ - + diff --git a/src/DurableTask.AzureStorage/EntityTrackingStoreQueries.cs b/src/DurableTask.AzureStorage/EntityTrackingStoreQueries.cs new file mode 100644 index 000000000..1bf9bfadb --- /dev/null +++ b/src/DurableTask.AzureStorage/EntityTrackingStoreQueries.cs @@ -0,0 +1,278 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ----------------------------------------------------------------------------------using System; +#nullable enable +namespace DurableTask.AzureStorage +{ + using System; + using System.Collections.Generic; + using System.Diagnostics; + using System.Linq; + using System.Runtime.Serialization.Json; + using System.Text; + using System.Threading; + using System.Threading.Tasks; + using Azure; + using DurableTask.AzureStorage.Tracking; + using DurableTask.Core; + using DurableTask.Core.Entities; + + class EntityTrackingStoreQueries : EntityBackendQueries + { + readonly MessageManager messageManager; + readonly ITrackingStore trackingStore; + readonly Func ensureTaskHub; + readonly EntityBackendProperties properties; + readonly Func sendEvent; + + static TimeSpan timeLimitForCleanEntityStorageLoop = TimeSpan.FromSeconds(5); + + public EntityTrackingStoreQueries( + MessageManager messageManager, + ITrackingStore trackingStore, + Func ensureTaskHub, + EntityBackendProperties properties, + Func sendEvent) + { + this.messageManager = messageManager; + this.trackingStore = trackingStore; + this.ensureTaskHub = ensureTaskHub; + this.properties = properties; + this.sendEvent = sendEvent; + } + + public async override Task GetEntityAsync( + EntityId id, + bool includeState = false, + bool includeStateless = false, + CancellationToken cancellation = default(CancellationToken)) + { + await this.ensureTaskHub(); + OrchestrationState? state = await this.trackingStore.GetStateAsync(id.ToString(), allExecutions: false, fetchInput: includeState).FirstOrDefaultAsync(); + return await this.GetEntityMetadataAsync(state, includeStateless, includeState); + } + + public async override Task QueryEntitiesAsync(EntityQuery filter, CancellationToken cancellation) + { + var condition = new OrchestrationInstanceStatusQueryCondition() + { + InstanceId = null, + InstanceIdPrefix = string.IsNullOrEmpty(filter.InstanceIdStartsWith) ? "@" : filter.InstanceIdStartsWith, + CreatedTimeFrom = filter.LastModifiedFrom ?? default(DateTime), + CreatedTimeTo = filter.LastModifiedTo ?? default(DateTime), + FetchInput = filter.IncludeState, + FetchOutput = false, + ExcludeEntities = false, + }; + + if (condition.InstanceIdPrefix![0] != '@') + { + condition.InstanceIdPrefix = $"@{condition.InstanceIdPrefix}"; + } + + await this.ensureTaskHub(); + + List entityResult; + string? continuationToken = filter.ContinuationToken; + var stopwatch = new Stopwatch(); + stopwatch.Start(); + + do + { + Page? page = await this.trackingStore.GetStateAsync(condition, cancellation).AsPages(continuationToken, filter.PageSize ?? 100).FirstOrDefaultAsync(); + DurableStatusQueryResult result = page != null + ? new DurableStatusQueryResult { ContinuationToken = page.ContinuationToken, OrchestrationState = page.Values } + : new DurableStatusQueryResult { OrchestrationState = Array.Empty() }; + entityResult = await ConvertResultsAsync(result.OrchestrationState); + continuationToken = result.ContinuationToken; + } + while ( // continue query right away if the page is completely empty, but never in excess of 100ms + continuationToken != null + && entityResult.Count == 0 + && stopwatch.ElapsedMilliseconds <= 100); + + return new EntityQueryResult() + { + Results = entityResult, + ContinuationToken = continuationToken, + }; + + async ValueTask> ConvertResultsAsync(IEnumerable states) + { + entityResult = new List(); + foreach (OrchestrationState entry in states) + { + EntityMetadata? entityMetadata = await this.GetEntityMetadataAsync(entry, filter.IncludeTransient, filter.IncludeState); + if (entityMetadata.HasValue) + { + entityResult.Add(entityMetadata.Value); + } + } + return entityResult; + } + } + + public async override Task CleanEntityStorageAsync(CleanEntityStorageRequest request = default(CleanEntityStorageRequest), CancellationToken cancellation = default(CancellationToken)) + { + DateTime now = DateTime.UtcNow; + string? continuationToken = request.ContinuationToken; + int emptyEntitiesRemoved = 0; + int orphanedLocksReleased = 0; + var stopwatch = Stopwatch.StartNew(); + + var condition = new OrchestrationInstanceStatusQueryCondition() + { + InstanceIdPrefix = "@", + FetchInput = false, + FetchOutput = false, + ExcludeEntities = false, + }; + + await this.ensureTaskHub(); + + // list all entities (without fetching the input) and for each one that requires action, + // perform that action. Waits for all actions to finish after each page. + do + { + Page? states = await this.trackingStore.GetStateAsync(condition, cancellation).AsPages(continuationToken, 100).FirstOrDefaultAsync(); + DurableStatusQueryResult page = states != null + ? new DurableStatusQueryResult { ContinuationToken = states.ContinuationToken, OrchestrationState = states.Values } + : new DurableStatusQueryResult { OrchestrationState = Array.Empty() }; + continuationToken = page.ContinuationToken; + + var tasks = new List(); + foreach (OrchestrationState state in page.OrchestrationState) + { + EntityStatus? status = ClientEntityHelpers.GetEntityStatus(state.Status); + if (status != null) + { + if (request.ReleaseOrphanedLocks && status.LockedBy != null) + { + tasks.Add(CheckForOrphanedLockAndFixIt(state, status.LockedBy)); + } + + if (request.RemoveEmptyEntities) + { + bool isEmptyEntity = !status.EntityExists && status.LockedBy == null && status.BacklogQueueSize == 0; + bool safeToRemoveWithoutBreakingMessageSorterLogic = + (now - state.LastUpdatedTime > this.properties.EntityMessageReorderWindow); + if (isEmptyEntity && safeToRemoveWithoutBreakingMessageSorterLogic) + { + tasks.Add(DeleteIdleOrchestrationEntity(state)); + } + } + } + } + + async Task DeleteIdleOrchestrationEntity(OrchestrationState state) + { + PurgeHistoryResult result = await this.trackingStore.PurgeInstanceHistoryAsync(state.OrchestrationInstance.InstanceId); + Interlocked.Add(ref emptyEntitiesRemoved, result.InstancesDeleted); + } + + async Task CheckForOrphanedLockAndFixIt(OrchestrationState state, string lockOwner) + { + OrchestrationState? ownerState + = await this.trackingStore.GetStateAsync(lockOwner, allExecutions: false, fetchInput: false).FirstOrDefaultAsync(); + + bool OrchestrationIsRunning(OrchestrationStatus? status) + => status != null && (status == OrchestrationStatus.Running || status == OrchestrationStatus.Suspended); + + if (! OrchestrationIsRunning(ownerState?.OrchestrationStatus)) + { + // the owner is not a running orchestration. Send a lock release. + EntityMessageEvent eventToSend = ClientEntityHelpers.EmitUnlockForOrphanedLock(state.OrchestrationInstance, lockOwner); + await this.sendEvent(eventToSend.AsTaskMessage()); + Interlocked.Increment(ref orphanedLocksReleased); + } + } + + await Task.WhenAll(tasks); + } + while (continuationToken != null & stopwatch.Elapsed <= timeLimitForCleanEntityStorageLoop); + + return new CleanEntityStorageResult() + { + EmptyEntitiesRemoved = emptyEntitiesRemoved, + OrphanedLocksReleased = orphanedLocksReleased, + ContinuationToken = continuationToken, + }; + } + + async ValueTask GetEntityMetadataAsync(OrchestrationState? state, bool includeTransient, bool includeState) + { + if (state == null) + { + return null; + } + + if (!includeState) + { + if (!includeTransient) + { + // it is possible that this entity was logically deleted even though its orchestration was not purged yet. + // we can check this efficiently (i.e. without deserializing anything) by looking at just the custom status + if (!EntityStatus.TestEntityExists(state.Status)) + { + return null; + } + } + + EntityStatus? status = ClientEntityHelpers.GetEntityStatus(state.Status); + + return new EntityMetadata() + { + EntityId = EntityId.FromString(state.OrchestrationInstance.InstanceId), + LastModifiedTime = state.CreatedTime, + BacklogQueueSize = status?.BacklogQueueSize ?? 0, + LockedBy = status?.LockedBy, + SerializedState = null, // we were instructed to not include the state + }; + } + else + { + // first, retrieve the entity scheduler state (= input of the orchestration state), possibly from blob storage. + string serializedSchedulerState; + if (MessageManager.TryGetLargeMessageReference(state.Input, out Uri blobUrl)) + { + serializedSchedulerState = await this.messageManager.DownloadAndDecompressAsBytesAsync(blobUrl); + } + else + { + serializedSchedulerState = state.Input; + } + + // next, extract the entity state from the scheduler state + string? serializedEntityState = ClientEntityHelpers.GetEntityState(serializedSchedulerState); + + // return the result to the user + if (!includeTransient && serializedEntityState == null) + { + return null; + } + else + { + EntityStatus? status = ClientEntityHelpers.GetEntityStatus(state.Status); + + return new EntityMetadata() + { + EntityId = EntityId.FromString(state.OrchestrationInstance.InstanceId), + LastModifiedTime = state.CreatedTime, + BacklogQueueSize = status?.BacklogQueueSize ?? 0, + LockedBy = status?.LockedBy, + SerializedState = serializedEntityState, + }; + } + } + } + } +} diff --git a/src/DurableTask.AzureStorage/Fnv1aHashHelper.cs b/src/DurableTask.AzureStorage/Fnv1aHashHelper.cs index fbff51089..2bea574ca 100644 --- a/src/DurableTask.AzureStorage/Fnv1aHashHelper.cs +++ b/src/DurableTask.AzureStorage/Fnv1aHashHelper.cs @@ -22,32 +22,61 @@ namespace DurableTask.AzureStorage /// See https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function. /// Tested with production data and random guids. The result was good distribution. /// - static class Fnv1aHashHelper + internal static class Fnv1aHashHelper { const uint FnvPrime = unchecked(16777619); const uint FnvOffsetBasis = unchecked(2166136261); + /// + /// Compute a hash for a given string. + /// + /// The string to hash. + /// a four-byte hash public static uint ComputeHash(string value) { return ComputeHash(value, encoding: null); } + /// + /// Compute a hash for a given string and encoding. + /// + /// The string to hash. + /// The encoding. + /// a four-byte hash public static uint ComputeHash(string value, Encoding encoding) { return ComputeHash(value, encoding, hash: FnvOffsetBasis); } + /// + /// Compute a hash for a given string, encoding, and hash modifier. + /// + /// The string to hash. + /// The encoding. + /// The modifier hash. + /// a four-byte hash public static uint ComputeHash(string value, Encoding encoding, uint hash) { byte[] bytes = (encoding ?? Encoding.UTF8).GetBytes(value); return ComputeHash(bytes, hash); } + /// + /// Compute a hash for the given byte array. + /// + /// The byte array to hash. + /// a four-byte hash public static uint ComputeHash(byte[] array) { return ComputeHash(array, hash: FnvOffsetBasis); } + /// + /// Compute a hash for the given byte array. + /// + /// The byte array to hash. + /// The modifier hash. + /// a four-byte hash public static uint ComputeHash(byte[] array, uint hash) { for (var i = 0; i < array.Length; i++) diff --git a/src/DurableTask.AzureStorage/OrchestrationSessionManager.cs b/src/DurableTask.AzureStorage/OrchestrationSessionManager.cs index acd73078a..04b8b19dd 100644 --- a/src/DurableTask.AzureStorage/OrchestrationSessionManager.cs +++ b/src/DurableTask.AzureStorage/OrchestrationSessionManager.cs @@ -33,7 +33,8 @@ class OrchestrationSessionManager : IDisposable readonly Dictionary activeOrchestrationSessions = new Dictionary(StringComparer.OrdinalIgnoreCase); readonly ConcurrentDictionary ownedControlQueues = new ConcurrentDictionary(); readonly LinkedList pendingOrchestrationMessageBatches = new LinkedList(); - readonly AsyncQueue> readyForProcessingQueue = new AsyncQueue>(); + readonly AsyncQueue> orchestrationsReadyForProcessingQueue = new AsyncQueue>(); + readonly AsyncQueue> entitiesReadyForProcessingQueue = new AsyncQueue>(); readonly object messageAndSessionLock = new object(); readonly string storageAccountName; @@ -520,7 +521,15 @@ async Task ScheduleOrchestrationStatePrefetch( batch.TrackingStoreContext = history.TrackingStoreContext; } - this.readyForProcessingQueue.Enqueue(node); + if (this.settings.UseSeparateQueueForEntityWorkItems + && DurableTask.Core.Common.Entities.IsEntityInstance(batch.OrchestrationInstanceId)) + { + this.entitiesReadyForProcessingQueue.Enqueue(node); + } + else + { + this.orchestrationsReadyForProcessingQueue.Enqueue(node); + } } catch (OperationCanceledException) { @@ -544,14 +553,16 @@ async Task ScheduleOrchestrationStatePrefetch( } } - public async Task GetNextSessionAsync(CancellationToken cancellationToken) + public async Task GetNextSessionAsync(bool entitiesOnly, CancellationToken cancellationToken) { + var readyForProcessingQueue = entitiesOnly? this.entitiesReadyForProcessingQueue : this.orchestrationsReadyForProcessingQueue; + while (!cancellationToken.IsCancellationRequested) { // This call will block until: // 1) a batch of messages has been received for a particular instance and // 2) the history for that instance has been fetched - LinkedListNode node = await this.readyForProcessingQueue.DequeueAsync(cancellationToken); + LinkedListNode node = await readyForProcessingQueue.DequeueAsync(cancellationToken); lock (this.messageAndSessionLock) { @@ -597,7 +608,7 @@ async Task ScheduleOrchestrationStatePrefetch( // A message arrived for a different generation of an existing orchestration instance. // Put it back into the ready queue so that it can be processed once the current generation // is done executing. - if (this.readyForProcessingQueue.Count == 0) + if (readyForProcessingQueue.Count == 0) { // To avoid a tight dequeue loop, delay for a bit before putting this node back into the queue. // This is only necessary when the queue is empty. The main dequeue thread must not be blocked @@ -607,14 +618,14 @@ async Task ScheduleOrchestrationStatePrefetch( lock (this.messageAndSessionLock) { this.pendingOrchestrationMessageBatches.AddLast(node); - this.readyForProcessingQueue.Enqueue(node); + readyForProcessingQueue.Enqueue(node); } }); } else { this.pendingOrchestrationMessageBatches.AddLast(node); - this.readyForProcessingQueue.Enqueue(node); + readyForProcessingQueue.Enqueue(node); } } } @@ -676,7 +687,8 @@ public void GetStats( public virtual void Dispose() { this.fetchRuntimeStateQueue.Dispose(); - this.readyForProcessingQueue.Dispose(); + this.orchestrationsReadyForProcessingQueue.Dispose(); + this.entitiesReadyForProcessingQueue.Dispose(); } class PendingMessageBatch diff --git a/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs b/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs index f602cd4a8..282cd50af 100644 --- a/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs +++ b/src/DurableTask.AzureStorage/Tracking/AzureTableTrackingStore.cs @@ -390,7 +390,7 @@ public override async IAsyncEnumerable GetStateAsync(string } /// - async Task FetchInstanceStatusInternalAsync(string instanceId, bool fetchInput, CancellationToken cancellationToken) + internal async Task FetchInstanceStatusInternalAsync(string instanceId, bool fetchInput, CancellationToken cancellationToken) { if (instanceId == null) { diff --git a/src/DurableTask.AzureStorage/Tracking/OrchestrationInstanceStatusQueryCondition.cs b/src/DurableTask.AzureStorage/Tracking/OrchestrationInstanceStatusQueryCondition.cs index cc96f2be4..0f47a3979 100644 --- a/src/DurableTask.AzureStorage/Tracking/OrchestrationInstanceStatusQueryCondition.cs +++ b/src/DurableTask.AzureStorage/Tracking/OrchestrationInstanceStatusQueryCondition.cs @@ -67,6 +67,11 @@ public sealed class OrchestrationInstanceStatusQueryCondition /// public bool FetchOutput { get; set; } = true; + /// + /// Whether to exclude entities from the results. + /// + public bool ExcludeEntities { get; set; } = false; + /// /// Get the corresponding OData filter. /// @@ -78,7 +83,8 @@ internal ODataCondition ToOData() this.CreatedTimeTo == default(DateTime) && this.TaskHubNames == null && this.InstanceIdPrefix == null && - this.InstanceId == null)) + this.InstanceId == null && + !this.ExcludeEntities)) { IEnumerable? select = null; if (!this.FetchInput || !this.FetchOutput) @@ -137,6 +143,10 @@ internal ODataCondition ToOData() conditions.Add($"{nameof(OrchestrationInstanceStatus.PartitionKey)} ge '{sanitizedPrefix}'"); conditions.Add($"{nameof(OrchestrationInstanceStatus.PartitionKey)} lt '{greaterThanPrefix}'"); } + else if (this.ExcludeEntities) + { + conditions.Add($"{nameof(OrchestrationInstanceStatus.PartitionKey)} lt '@' or {nameof(OrchestrationInstanceStatus.PartitionKey)} ge 'A'"); + } if (this.InstanceId != null) { diff --git a/src/DurableTask.Core/Common/Entities.cs b/src/DurableTask.Core/Common/Entities.cs index 2484153de..dc3ba2434 100644 --- a/src/DurableTask.Core/Common/Entities.cs +++ b/src/DurableTask.Core/Common/Entities.cs @@ -10,14 +10,13 @@ // See the License for the specific language governing permissions and // limitations under the License. // ---------------------------------------------------------------------------------- - -using DurableTask.Core.History; -using System; -using System.Collections.Generic; -using System.Text; - +#nullable enable namespace DurableTask.Core.Common { + using DurableTask.Core.History; + using System; + using System.Collections.Generic; + /// /// Helpers for dealing with special naming conventions around auto-started orchestrations (entities) /// diff --git a/src/DurableTask.Core/Common/Fnv1aHashHelper.cs b/src/DurableTask.Core/Common/Fnv1aHashHelper.cs new file mode 100644 index 000000000..6184d6fdd --- /dev/null +++ b/src/DurableTask.Core/Common/Fnv1aHashHelper.cs @@ -0,0 +1,93 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +namespace DurableTask.Core.Common +{ + using System.Text; + + /// + /// Fast, non-cryptographic hash function helper. + /// + /// + /// See https://en.wikipedia.org/wiki/Fowler%E2%80%93Noll%E2%80%93Vo_hash_function. + /// Tested with production data and random guids. The result was good distribution. + /// + internal static class Fnv1aHashHelper + { + const uint FnvPrime = unchecked(16777619); + const uint FnvOffsetBasis = unchecked(2166136261); + + /// + /// Compute a hash for a given string. + /// + /// The string to hash. + /// a four-byte hash + public static uint ComputeHash(string value) + { + return ComputeHash(value, encoding: null); + } + + /// + /// Compute a hash for a given string and encoding. + /// + /// The string to hash. + /// The encoding. + /// a four-byte hash + public static uint ComputeHash(string value, Encoding encoding) + { + return ComputeHash(value, encoding, hash: FnvOffsetBasis); + } + + /// + /// Compute a hash for a given string, encoding, and hash modifier. + /// + /// The string to hash. + /// The encoding. + /// The modifier hash. + /// a four-byte hash + public static uint ComputeHash(string value, Encoding encoding, uint hash) + { + byte[] bytes = (encoding ?? Encoding.UTF8).GetBytes(value); + return ComputeHash(bytes, hash); + } + + /// + /// Compute a hash for the given byte array. + /// + /// The byte array to hash. + /// a four-byte hash + public static uint ComputeHash(byte[] array) + { + return ComputeHash(array, hash: FnvOffsetBasis); + } + + /// + /// Compute a hash for the given byte array. + /// + /// The byte array to hash. + /// The modifier hash. + /// a four-byte hash + public static uint ComputeHash(byte[] array, uint hash) + { + for (var i = 0; i < array.Length; i++) + { + unchecked + { + hash ^= array[i]; + hash *= FnvPrime; + } + } + + return hash; + } + } +} diff --git a/src/DurableTask.Core/Common/Utils.cs b/src/DurableTask.Core/Common/Utils.cs index 0d151cc15..ef3927b8e 100644 --- a/src/DurableTask.Core/Common/Utils.cs +++ b/src/DurableTask.Core/Common/Utils.cs @@ -152,7 +152,7 @@ public static string SerializeToJson(JsonSerializer serializer, object payload) /// The default value comes from the WEBSITE_SITE_NAME environment variable, which is defined /// in Azure App Service. Other environments can use DTFX_APP_NAME to set this value. /// - public static string AppName { get; set; } = + public static string AppName { get; set; } = Environment.GetEnvironmentVariable("WEBSITE_SITE_NAME") ?? Environment.GetEnvironmentVariable("DTFX_APP_NAME") ?? string.Empty; @@ -624,6 +624,40 @@ public static bool TryGetTaskScheduledId(HistoryEvent historyEvent, out int task } } + /// + /// Creates a determinstic Guid from a string using a hash function. This is a simple hash + /// meant to produce pseudo-random Guids, it is not meant to be cryptographically secure, + /// and does not follow any formatting conventions for UUIDs (such as RFC 4122). + /// + /// The string to hash. + /// A Guid constructed from the hash. + /// + internal static Guid CreateGuidFromHash(string stringToHash) + { + if (string.IsNullOrEmpty(stringToHash)) + { + throw new ArgumentException("string to hash must not be null or empty", nameof(stringToHash)); + } + + var bytes = Encoding.UTF8.GetBytes(stringToHash); + uint hash1 = Fnv1aHashHelper.ComputeHash(bytes, 0xdf0dd395); + uint hash2 = Fnv1aHashHelper.ComputeHash(bytes, 0xa19df4df); + uint hash3 = Fnv1aHashHelper.ComputeHash(bytes, 0xc88599c5); + uint hash4 = Fnv1aHashHelper.ComputeHash(bytes, 0xe24e3e64); + return new Guid( + hash1, + (ushort)(hash2 & 0xFFFF), + (ushort)((hash2 >> 16) & 0xFFFF), + (byte)(hash3 & 0xFF), + (byte)((hash3 >> 8) & 0xFF), + (byte)((hash3 >> 16) & 0xFF), + (byte)((hash3 >> 24) & 0xFF), + (byte)(hash4 & 0xFF), + (byte)((hash4 >> 8) & 0xFF), + (byte)((hash4 >> 16) & 0xFF), + (byte)((hash4 >> 24) & 0xFF)); + } + /// /// Gets the generic return type for a specific . /// diff --git a/src/DurableTask.Core/DurableTask.Core.csproj b/src/DurableTask.Core/DurableTask.Core.csproj index 802e5e13f..f413cfab4 100644 --- a/src/DurableTask.Core/DurableTask.Core.csproj +++ b/src/DurableTask.Core/DurableTask.Core.csproj @@ -17,22 +17,28 @@ 2 - 15 - 1 - + 17 + 0 $(MajorVersion).$(MinorVersion).$(PatchVersion) $(VersionPrefix).0 $(VersionPrefix).$(FileVersionRevision) $(MajorVersion).$(MinorVersion).0.0 - - $(VersionPrefix) + + + $(VersionPrefix) + + + $(VersionPrefix)-$(VersionSuffix) + + - + + diff --git a/src/DurableTask.Core/Entities/ClientEntityHelpers.cs b/src/DurableTask.Core/Entities/ClientEntityHelpers.cs new file mode 100644 index 000000000..94bc0512a --- /dev/null +++ b/src/DurableTask.Core/Entities/ClientEntityHelpers.cs @@ -0,0 +1,102 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using DurableTask.Core.Entities; + using DurableTask.Core.Entities.EventFormat; + using Newtonsoft.Json.Linq; + using Newtonsoft.Json; + using System; + + /// + /// Utility functions for clients that interact with entities, either by sending events or by accessing the entity state directly in storage + /// + public static class ClientEntityHelpers + { + /// + /// Create an event to represent an entity signal. + /// + /// The target instance. + /// A unique identifier for the request. + /// The name of the operation. + /// The serialized input for the operation. + /// The time to schedule this signal, or null if not a scheduled signal + /// The event to send. + public static EntityMessageEvent EmitOperationSignal(OrchestrationInstance targetInstance, Guid requestId, string operationName, string? input, (DateTime Original, DateTime Capped)? scheduledTimeUtc) + { + var request = new RequestMessage() + { + ParentInstanceId = null, // means this was sent by a client + ParentExecutionId = null, + Id = requestId, + IsSignal = true, + Operation = operationName, + ScheduledTime = scheduledTimeUtc?.Original, + Input = input, + }; + + var eventName = scheduledTimeUtc.HasValue + ? EntityMessageEventNames.ScheduledRequestMessageEventName(scheduledTimeUtc.Value.Capped) + : EntityMessageEventNames.RequestMessageEventName; + + return new EntityMessageEvent(eventName, request, targetInstance); + } + + /// + /// Create an event to represent an entity unlock, which is called by clients to fix orphaned locks. + /// + /// The target instance. + /// The instance id of the entity to be unlocked. + /// The event to send. + public static EntityMessageEvent EmitUnlockForOrphanedLock(OrchestrationInstance targetInstance, string lockOwnerInstanceId) + { + var message = new ReleaseMessage() + { + ParentInstanceId = lockOwnerInstanceId, + Id = "fix-orphaned-lock", // we don't know the original id but it does not matter + }; + + return new EntityMessageEvent(EntityMessageEventNames.ReleaseMessageEventName, message, targetInstance); + } + + /// + /// Extracts the user-defined entity state from the serialized scheduler state. The result is the serialized state, + /// or null if the entity has no state. + /// + public static string? GetEntityState(string? serializedSchedulerState) + { + if (serializedSchedulerState == null) + { + return null; + } + + var schedulerState = JsonConvert.DeserializeObject(serializedSchedulerState, Serializer.InternalSerializerSettings)!; + return schedulerState.EntityState; + } + + /// + /// Gets the entity status from the serialized custom status of the orchestration. + /// or null if the entity has no state. + /// + public static EntityStatus? GetEntityStatus(string? orchestrationCustomStatus) + { + if (orchestrationCustomStatus == null) + { + return null; + } + + return JsonConvert.DeserializeObject(orchestrationCustomStatus, Serializer.InternalSerializerSettings)!; + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/EntityBackendProperties.cs b/src/DurableTask.Core/Entities/EntityBackendProperties.cs new file mode 100644 index 000000000..20d8ec9e0 --- /dev/null +++ b/src/DurableTask.Core/Entities/EntityBackendProperties.cs @@ -0,0 +1,79 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using System.Threading; + + /// + /// Entity processing characteristics that are controlled by the backend provider, i.e. the orchestration service. + /// + public class EntityBackendProperties + { + /// + /// The time window within which entity messages should be deduplicated and reordered. + /// This is zero for providers that already guarantee exactly-once and ordered delivery. + /// + public TimeSpan EntityMessageReorderWindow { get; set; } + + /// + /// A limit on the number of entity operations that should be processed as a single batch, or null if there is no limit. + /// + public int? MaxEntityOperationBatchSize { get; set; } + + /// + /// The maximum number of entity operation batches that can be processed concurrently on a single node. + /// + public int MaxConcurrentTaskEntityWorkItems { get; set; } + + /// + /// Gets or sets whether the backend supports implicit deletion. Implicit deletion means that + /// the storage does not retain any data for entities that don't have any state. + /// + public bool SupportsImplicitEntityDeletion { get; set; } + + /// + /// Gets or sets the maximum durable timer delay. Used for delayed signals. + /// + public TimeSpan MaximumSignalDelayTime { get; set; } + + /// + /// Gets or sets whether the backend uses separate work item queues for entities and orchestrators. If true, + /// the frontend must use and + /// + /// to fetch entities and orchestrations. Otherwise, it must use fetch both work items using + /// . + /// + public bool UseSeparateQueueForEntityWorkItems { get; set; } + + /// + /// A utility function to compute a cap on the scheduled time of an entity signal, based on the value of + /// . + /// + /// The current time. + /// The scheduled time. + /// + public DateTime GetCappedScheduledTime(DateTime nowUtc, DateTime scheduledUtcTime) + { + if ((scheduledUtcTime - nowUtc) <= this.MaximumSignalDelayTime) + { + return scheduledUtcTime; + } + else + { + return nowUtc + this.MaximumSignalDelayTime; + } + } + } +} diff --git a/src/DurableTask.Core/Entities/EntityBackendQueries.cs b/src/DurableTask.Core/Entities/EntityBackendQueries.cs new file mode 100644 index 000000000..7f85012c6 --- /dev/null +++ b/src/DurableTask.Core/Entities/EntityBackendQueries.cs @@ -0,0 +1,199 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using System.Collections.Generic; + using System.Threading; + using System.Threading.Tasks; + + /// + /// Encapsulates support for entity queries, at the abstraction level of a storage backend. + /// + public abstract class EntityBackendQueries + { + /// + /// Tries to get the entity with ID of . + /// + /// The ID of the entity to get. + /// true to include entity state in the response, false to not. + /// whether to include metadata for entities without user-defined state. + /// The cancellation token to cancel the operation. + /// a response containing metadata describing the entity. + public abstract Task GetEntityAsync( + EntityId id, bool includeState = false, bool includeStateless = false, CancellationToken cancellation = default); + + /// + /// Queries entity instances based on the conditions specified in . + /// + /// The query filter. + /// The cancellation token to cancel the operation. + /// One page of query results. + public abstract Task QueryEntitiesAsync(EntityQuery query, CancellationToken cancellation); + + /// + /// Cleans entity storage. See for the different forms of cleaning available. + /// + /// The request which describes what to clean. + /// The cancellation token to cancel the operation. + /// A task that completes when the operation is finished. + public abstract Task CleanEntityStorageAsync( + CleanEntityStorageRequest request = default, CancellationToken cancellation = default); + + /// + /// Metadata about an entity, as returned by queries. + /// + public struct EntityMetadata + { + /// + /// Gets or sets the ID for this entity. + /// + public EntityId EntityId { get; set; } + + /// + /// Gets or sets the time the entity was last modified. + /// + public DateTime LastModifiedTime { get; set; } + + /// + /// Gets the size of the backlog queue, if there is a backlog, and if that metric is supported by the backend. + /// + public int BacklogQueueSize { get; set; } + + /// + /// Gets the instance id of the orchestration that has locked this entity, or null if the entity is not locked. + /// + public string? LockedBy { get; set; } + + /// + /// Gets or sets the serialized state for this entity. Can be null if the query + /// specified to not include the state, or to include deleted entities. + /// + public string? SerializedState { get; set; } + } + + /// + /// A description of an entity query. + /// + /// + /// The default query returns all entities (does not specify any filters). + /// + public struct EntityQuery + { + /// + /// Gets or sets the optional starts-with expression for the entity instance ID. + /// + public string? InstanceIdStartsWith { get; set; } + + /// + /// Gets or sets a value indicating to include only entity instances which were last modified after the provided time. + /// + public DateTime? LastModifiedFrom { get; set; } + + /// + /// Gets or sets a value indicating to include only entity instances which were last modified before the provided time. + /// + public DateTime? LastModifiedTo { get; set; } + + /// + /// Gets or sets a value indicating whether to include state in the query results or not. + /// + public bool IncludeState { get; set; } + + /// + /// Gets a value indicating whether to include metadata about transient entities. + /// + /// Transient entities are entities that do not have an application-defined state, but for which the storage provider is + /// tracking metadata for synchronization purposes. + /// For example, a transient entity may be observed when the entity is in the process of being created or deleted, or + /// when the entity has been locked by a critical section. By default, transient entities are not included in queries since they are + /// considered to "not exist" from the perspective of the user application. + /// + public bool IncludeTransient { get; set; } + + /// + /// Gets or sets the desired size of each page to return. + /// + /// + /// If no size is specified, the backend may choose an appropriate page size based on its implementation. + /// Note that the size of the returned page may be smaller or larger than the requested page size, and cannot + /// be used to determine whether the end of the query has been reached. + /// + public int? PageSize { get; set; } + + /// + /// Gets or sets the continuation token to resume a previous query. + /// + public string? ContinuationToken { get; set; } + } + + /// + /// A page of results. + /// + public struct EntityQueryResult + { + /// + /// Gets or sets the query results. + /// + public IEnumerable Results { get; set; } + + /// + /// Gets or sets the continuation token to continue this query, if not null. + /// + public string? ContinuationToken { get; set; } + } + + /// + /// Request struct for . + /// + public struct CleanEntityStorageRequest + { + /// + /// Gets or sets a value indicating whether to remove empty entities. + /// + public bool RemoveEmptyEntities { get; set; } + + /// + /// Gets or sets a value indicating whether to release orphaned locks or not. + /// + public bool ReleaseOrphanedLocks { get; set; } + + /// + /// Gets or sets the continuation token to resume a previous . + /// + public string? ContinuationToken { get; set; } + } + + /// + /// Result struct for . + /// + public struct CleanEntityStorageResult + { + /// + /// Gets or sets the number of empty entities removed. + /// + public int EmptyEntitiesRemoved { get; set; } + + /// + /// Gets or sets the number of orphaned locks that were removed. + /// + public int OrphanedLocksReleased { get; set; } + + /// + /// Gets or sets the continuation token to continue the , if not null. + /// + public string? ContinuationToken { get; set; } + } + } +} diff --git a/src/DurableTask.Core/Entities/EntityId.cs b/src/DurableTask.Core/Entities/EntityId.cs new file mode 100644 index 000000000..68ed4e943 --- /dev/null +++ b/src/DurableTask.Core/Entities/EntityId.cs @@ -0,0 +1,106 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using System.Runtime.Serialization; + + /// + /// A unique identifier for an entity, consisting of entity name and entity key. + /// + [DataContract] + public readonly struct EntityId : IEquatable, IComparable + { + /// + /// Create an entity id for an entity. + /// + /// The name of this class of entities. + /// The entity key. + public EntityId(string name, string key) + { + if (string.IsNullOrEmpty(name)) + { + throw new ArgumentNullException(nameof(name), "Invalid entity id: entity name must not be a null or empty string."); + } + + this.Name = name; + this.Key = key ?? throw new ArgumentNullException(nameof(key), "Invalid entity id: entity key must not be null."); + } + + /// + /// The name for this class of entities. + /// + [DataMember(Name = "name", IsRequired = true)] + public readonly string Name { get; } + + /// + /// The entity key. Uniquely identifies an entity among all entities of the same name. + /// + [DataMember(Name = "key", IsRequired = true)] + public readonly string Key { get; } + + /// + public override string ToString() + { + return $"@{this.Name}@{this.Key}"; + } + + /// + /// Returns the entity ID for a given instance ID. + /// + /// The instance ID. + /// the corresponding entity ID. + public static EntityId FromString(string instanceId) + { + if (string.IsNullOrEmpty(instanceId)) + { + throw new ArgumentException(nameof(instanceId)); + } + var pos = instanceId.IndexOf('@', 1); + if (pos <= 0 || instanceId[0] != '@') + { + throw new ArgumentException($"Instance ID '{instanceId}' is not a valid entity ID.", nameof(instanceId)); + } + var entityName = instanceId.Substring(1, pos - 1); + var entityKey = instanceId.Substring(pos + 1); + return new EntityId(entityName, entityKey); + } + + + /// + public override bool Equals(object obj) + { + return (obj is EntityId other) && this.Equals(other); + } + + /// + public bool Equals(EntityId other) + { + return (this.Name, this.Key).Equals((other.Name, other.Key)); + } + + /// + public override int GetHashCode() + { + return (this.Name, this.Key).GetHashCode(); + } + + /// + public int CompareTo(object obj) + { + var other = (EntityId)obj; + return (this.Name, this.Key).CompareTo((other.Name, other.Key)); + } + } +} diff --git a/src/DurableTask.Core/Entities/EntityMessageEvent.cs b/src/DurableTask.Core/Entities/EntityMessageEvent.cs new file mode 100644 index 000000000..4faf66512 --- /dev/null +++ b/src/DurableTask.Core/Entities/EntityMessageEvent.cs @@ -0,0 +1,113 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +using System; +using DurableTask.Core.Entities.EventFormat; +using DurableTask.Core.Serializing.Internal; +using Newtonsoft.Json; + +namespace DurableTask.Core.Entities +{ + /// + /// Encapsulates events that represent a message sent to or from an entity. + /// + public readonly struct EntityMessageEvent + { + readonly string eventName; + readonly EntityMessage message; + readonly OrchestrationInstance target; + + internal EntityMessageEvent(string eventName, EntityMessage message, OrchestrationInstance target) + { + this.eventName = eventName; + this.message = message; + this.target = target; + } + + /// + public override string ToString() + { + return this.message.ToString(); + } + + /// + /// The name of the event. + /// + public string EventName => this.eventName; + + /// + /// The target instance for the event. + /// + public OrchestrationInstance TargetInstance => this.target; + + /// + /// Returns the content of this event, as a serialized string. + /// + /// + public string AsSerializedString() + { + return JsonConvert.SerializeObject(message, Serializer.InternalSerializerSettings); + } + + /// + /// Returns this event in the form of a TaskMessage. + /// + /// + public TaskMessage AsTaskMessage() + { + return new TaskMessage + { + OrchestrationInstance = this.target, + Event = new History.EventRaisedEvent(-1, this.AsSerializedString()) + { + Name = this.eventName + } + }; + } + +#pragma warning disable CS0618 // Type or member is obsolete. Intentional internal usage. + /// + /// Returns the content as an already-serialized string. Can be used to bypass the application-defined serializer. + /// + /// + public RawInput AsRawInput() + { + return new RawInput(this.AsSerializedString()); + } +#pragma warning restore CS0618 // Type or member is obsolete + + /// + /// Utility function to compute a capped scheduled time, given a scheduled time, a timestamp representing the current time, and the maximum delay. + /// + /// a timestamp representing the current time + /// the scheduled time, or null if none. + /// The maximum delay supported by the backend. + /// the capped scheduled time, or null if none. + public static (DateTime original, DateTime capped)? GetCappedScheduledTime(DateTime nowUtc, TimeSpan maxDelay, DateTime? scheduledUtcTime) + { + if (!scheduledUtcTime.HasValue) + { + return null; + } + + if ((scheduledUtcTime - nowUtc) <= maxDelay) + { + return (scheduledUtcTime.Value, scheduledUtcTime.Value); + } + else + { + return (scheduledUtcTime.Value, nowUtc + maxDelay); + } + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/EventFormat/EntityMessage.cs b/src/DurableTask.Core/Entities/EventFormat/EntityMessage.cs new file mode 100644 index 000000000..a07f333bc --- /dev/null +++ b/src/DurableTask.Core/Entities/EventFormat/EntityMessage.cs @@ -0,0 +1,28 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.EventFormat +{ + using System.Runtime.Serialization; + + /// + /// The format of entity messages is kept json-deserialization-compatible with the original format. + /// + [DataContract] + internal abstract class EntityMessage + { + public abstract string GetShortDescription(); + + public override string ToString() => this.GetShortDescription(); + } +} diff --git a/src/DurableTask.Core/Entities/EventFormat/EntityMessageEventNames.cs b/src/DurableTask.Core/Entities/EventFormat/EntityMessageEventNames.cs new file mode 100644 index 000000000..adecea1dc --- /dev/null +++ b/src/DurableTask.Core/Entities/EventFormat/EntityMessageEventNames.cs @@ -0,0 +1,37 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.EventFormat +{ + using System; + + /// + /// Determines event names to use for messages sent to and from entities. + /// + internal static class EntityMessageEventNames + { + public static string RequestMessageEventName => "op"; + + public static string ReleaseMessageEventName => "release"; + + public static string ContinueMessageEventName => "continue"; + + public static string ScheduledRequestMessageEventName(DateTime scheduledUtc) => $"op@{scheduledUtc:o}"; + + public static string ResponseMessageEventName(Guid requestId) => requestId.ToString(); + + public static bool IsRequestMessage(string eventName) => eventName.StartsWith("op"); + + public static bool IsReleaseMessage(string eventName) => eventName == "release"; + } +} diff --git a/src/DurableTask.Core/Entities/EventFormat/ReleaseMessage.cs b/src/DurableTask.Core/Entities/EventFormat/ReleaseMessage.cs new file mode 100644 index 000000000..455c9b4fc --- /dev/null +++ b/src/DurableTask.Core/Entities/EventFormat/ReleaseMessage.cs @@ -0,0 +1,32 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.EventFormat +{ + using System.Runtime.Serialization; + + [DataContract] + internal class ReleaseMessage : EntityMessage + { + [DataMember(Name = "parent")] + public string? ParentInstanceId { get; set; } + + [DataMember(Name = "id")] + public string? Id { get; set; } + + public override string GetShortDescription() + { + return $"[Release lock {Id} by {ParentInstanceId}]"; + } + } +} diff --git a/src/DurableTask.Core/Entities/EventFormat/RequestMessage.cs b/src/DurableTask.Core/Entities/EventFormat/RequestMessage.cs new file mode 100644 index 000000000..e46d1759f --- /dev/null +++ b/src/DurableTask.Core/Entities/EventFormat/RequestMessage.cs @@ -0,0 +1,114 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.EventFormat +{ + using System; + using System.Runtime.Serialization; + + /// + /// A message sent to an entity, such as operation, signal, lock, or continue messages. + /// + [DataContract] + internal class RequestMessage : EntityMessage + { + /// + /// The name of the operation being called (if this is an operation message) or null + /// (if this is a lock request). + /// + [DataMember(Name = "op")] + public string? Operation { get; set; } + + /// + /// Whether or not this is a one-way message. + /// + [DataMember(Name = "signal", EmitDefaultValue = false)] + public bool IsSignal { get; set; } + + /// + /// The operation input. + /// + [DataMember(Name = "input", EmitDefaultValue = false)] + public string? Input { get; set; } + + /// + /// A unique identifier for this operation. + /// + [DataMember(Name = "id", IsRequired = true)] + public Guid Id { get; set; } + + /// + /// The parent instance that called this operation. + /// + [DataMember(Name = "parent", EmitDefaultValue = false)] + public string? ParentInstanceId { get; set; } + + /// + /// The parent instance that called this operation. + /// + [DataMember(Name = "parentExecution", EmitDefaultValue = false)] + public string? ParentExecutionId { get; set; } + + /// + /// Optionally, a scheduled time at which to start the operation. + /// + [DataMember(Name = "due", EmitDefaultValue = false)] + public DateTime? ScheduledTime { get; set; } + + /// + /// A timestamp for this request. + /// Used for duplicate filtering and in-order delivery. + /// + [DataMember] + public DateTime Timestamp { get; set; } + + /// + /// A timestamp for the predecessor request in the stream, or DateTime.MinValue if none. + /// Used for duplicate filtering and in-order delivery. + /// + [DataMember] + public DateTime Predecessor { get; set; } + + /// + /// For lock requests, the set of locks being acquired. Is sorted, + /// contains at least one element, and has no repetitions. + /// + [DataMember(Name = "lockset", EmitDefaultValue = false)] + public EntityId[]? LockSet { get; set; } + + /// + /// For lock requests involving multiple locks, the message number. + /// + [DataMember(Name = "pos", EmitDefaultValue = false)] + public int Position { get; set; } + + /// + /// whether this message is a lock request + /// + [DataMember] + public bool IsLockRequest => LockSet != null; + + /// + public override string GetShortDescription() + { + if (IsLockRequest) + { + return $"[Request lock {Id} by {ParentInstanceId} {ParentExecutionId}, position {Position}]"; + } + else + { + return $"[{(IsSignal ? "Signal" : "Call")} '{Operation}' operation {Id} by {ParentInstanceId} {ParentExecutionId}]"; + } + } + } +} diff --git a/src/DurableTask.Core/Entities/EventFormat/ResponseMessage.cs b/src/DurableTask.Core/Entities/EventFormat/ResponseMessage.cs new file mode 100644 index 000000000..8f78516b5 --- /dev/null +++ b/src/DurableTask.Core/Entities/EventFormat/ResponseMessage.cs @@ -0,0 +1,51 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.EventFormat +{ + using System.Runtime.Serialization; + + [DataContract] + internal class ResponseMessage : EntityMessage + { + public const string LockAcquisitionCompletion = "Lock Acquisition Completed"; + + [DataMember(Name = "result")] + public string? Result { get; set; } + + [DataMember(Name = "exceptionType", EmitDefaultValue = false)] + public string? ErrorMessage { get; set; } + + [DataMember(Name = "failureDetails", EmitDefaultValue = false)] + public FailureDetails? FailureDetails { get; set; } + + [IgnoreDataMember] + public bool IsErrorResult => this.ErrorMessage != null || this.FailureDetails != null; + + public override string GetShortDescription() + { + if (this.IsErrorResult) + { + return $"[OperationFailed {this.FailureDetails?.ErrorMessage ?? this.ErrorMessage}]"; + } + else if (this.Result == LockAcquisitionCompletion) + { + return "[LockAcquisitionComplete]"; + } + else + { + return $"[OperationSuccessful ({Result?.Length ?? 0} chars)]"; + } + } + } +} diff --git a/src/DurableTask.Core/Entities/IEntityOrchestrationService.cs b/src/DurableTask.Core/Entities/IEntityOrchestrationService.cs new file mode 100644 index 000000000..c8c9b80c8 --- /dev/null +++ b/src/DurableTask.Core/Entities/IEntityOrchestrationService.cs @@ -0,0 +1,50 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using System.Threading; + using System.Threading.Tasks; + + /// + /// Extends with methods that support processing of entities. + /// + public interface IEntityOrchestrationService : IOrchestrationService + { + /// + /// Properties of the backend implementation and configuration, as related to the new entity support in DurableTask.Core. + /// + /// An object containing properties of the entity backend, or null if the backend does not natively support DurableTask.Core entities. + EntityBackendProperties? EntityBackendProperties { get; } + + /// + /// Support for entity queries. + /// + /// An object that can be used to issue entity queries to the orchestration service, or null if the backend does not natively + /// support entity queries. + EntityBackendQueries? EntityBackendQueries { get; } + + /// + /// Specialized variant of that + /// fetches only work items for true orchestrations, not entities. + /// + Task LockNextOrchestrationWorkItemAsync(TimeSpan receiveTimeout, CancellationToken cancellationToken); + + /// + /// Specialized variant of that + /// fetches only work items for entities, not plain orchestrations. + /// + Task LockNextEntityWorkItemAsync(TimeSpan receiveTimeout, CancellationToken cancellationToken); + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/EntityBatchRequest.cs b/src/DurableTask.Core/Entities/OperationFormat/EntityBatchRequest.cs new file mode 100644 index 000000000..1a08a8ac5 --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/EntityBatchRequest.cs @@ -0,0 +1,41 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using System.Collections.Generic; + + /// + /// A request for execution of a batch of operations on an entity. + /// + public class EntityBatchRequest + { + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// The instance id for this entity. + /// + public string? InstanceId { get; set; } + + /// + /// The current state of the entity, or null if the entity does not exist. + /// + public string? EntityState { get; set; } + + /// + /// The list of operations to be performed on the entity. + /// + public List? Operations { get; set; } + } +} diff --git a/src/DurableTask.Core/Entities/OperationFormat/EntityBatchResult.cs b/src/DurableTask.Core/Entities/OperationFormat/EntityBatchResult.cs new file mode 100644 index 000000000..645fa9cf8 --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/EntityBatchResult.cs @@ -0,0 +1,49 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using System.Collections.Generic; + + /// + /// The results of executing a batch of operations on the entity out of process. + /// + public class EntityBatchResult + { + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// The results of executing the operations in the batch. If there were (non-application-level) errors, the length of this list may + /// be shorter than the number of requests. In that case, contains the reason why not all requests + /// were processed. + /// + public List? Results { get; set; } + + /// + /// The list of actions (outgoing messages) performed while executing the operations in the batch. Can be empty. + /// + public List? Actions { get; set; } + + /// + /// The state of the entity after executing the batch, + /// or null if the entity has no state (e.g. if it has been deleted). + /// + public string? EntityState { get; set; } + + /// + /// Contains the failure details, if there was a failure to process all requests (fewer results were returned than requests) + /// + public FailureDetails? FailureDetails { get; set; } + } +} diff --git a/src/DurableTask.Core/Entities/OperationFormat/OperationAction.cs b/src/DurableTask.Core/Entities/OperationFormat/OperationAction.cs new file mode 100644 index 000000000..467cecda4 --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/OperationAction.cs @@ -0,0 +1,29 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using Newtonsoft.Json; + + /// + /// Defines a set of base properties for an operator action. + /// + [JsonConverter(typeof(OperationActionConverter))] + public abstract class OperationAction + { + /// + /// The type of the orchestrator action. + /// + public abstract OperationActionType OperationActionType { get; } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/OperationActionConverter.cs b/src/DurableTask.Core/Entities/OperationFormat/OperationActionConverter.cs new file mode 100644 index 000000000..96cee6b4f --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/OperationActionConverter.cs @@ -0,0 +1,40 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +namespace DurableTask.Core.Entities.OperationFormat +{ + using System; + using Newtonsoft.Json.Linq; + using DurableTask.Core.Serializing; + + internal class OperationActionConverter : JsonCreationConverter + { + protected override OperationAction CreateObject(Type objectType, JObject jObject) + { + if (jObject.TryGetValue("OperationActionType", StringComparison.OrdinalIgnoreCase, out JToken actionType)) + { + var type = (OperationActionType)int.Parse((string)actionType); + switch (type) + { + case OperationActionType.SendSignal: + return new SendSignalOperationAction(); + case OperationActionType.StartNewOrchestration: + return new StartNewOrchestrationOperationAction(); + default: + throw new NotSupportedException("Unrecognized action type."); + } + } + + throw new NotSupportedException("Action Type not provided."); + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/OperationActionType.cs b/src/DurableTask.Core/Entities/OperationFormat/OperationActionType.cs new file mode 100644 index 000000000..8fb656e1a --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/OperationActionType.cs @@ -0,0 +1,31 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + /// + /// Enumeration of entity operation actions. + /// + public enum OperationActionType + { + /// + /// A signal was sent to an entity + /// + SendSignal, + + /// + /// A new fire-and-forget orchestration was started + /// + StartNewOrchestration, + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/OperationRequest.cs b/src/DurableTask.Core/Entities/OperationFormat/OperationRequest.cs new file mode 100644 index 000000000..ab249f88f --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/OperationRequest.cs @@ -0,0 +1,41 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using System; + + /// + /// A request message sent to an entity when calling or signaling the entity. + /// + public class OperationRequest + { + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// The name of the operation. + /// + public string? Operation { get; set; } + + /// + /// The unique GUID of the operation. + /// + public Guid Id { get; set; } + + /// + /// The input for the operation. Can be null if no input was given. + /// + public string? Input { get; set; } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/OperationResult.cs b/src/DurableTask.Core/Entities/OperationFormat/OperationResult.cs new file mode 100644 index 000000000..01cc41b11 --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/OperationResult.cs @@ -0,0 +1,50 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + /// + /// A response message sent by an entity to a caller after it executes an operation. + /// + public class OperationResult + { + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// The serialized result returned by the operation. Can be null, if the operation returned no result. + /// May contain error details, such as a serialized exception, if is true. + /// + public string? Result { get; set; } + + /// + /// Whether this operation completed successfully. + /// + public bool IsError + => this.ErrorMessage != null || this.FailureDetails != null; + + /// + /// If non-null, this string indicates that this operation did not successfully complete. + /// The content and interpretation varies depending on the SDK used. For newer SDKs, + /// we rely on the instead. + /// + public string? ErrorMessage { get; set; } + + /// + /// A structured language-independent representation of the error. Whether this field is present + /// depends on which SDK is used, and on configuration settings. For newer SDKs, we use + /// this field exclusively when collecting error information. + /// + public FailureDetails? FailureDetails { get; set; } + } +} diff --git a/src/DurableTask.Core/Entities/OperationFormat/SendSignalOperationAction.cs b/src/DurableTask.Core/Entities/OperationFormat/SendSignalOperationAction.cs new file mode 100644 index 000000000..04531ac31 --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/SendSignalOperationAction.cs @@ -0,0 +1,49 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using System; + + /// + /// Operation action for sending a signal. + /// + public class SendSignalOperationAction : OperationAction + { + /// + public override OperationActionType OperationActionType => OperationActionType.SendSignal; + + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// The destination entity for the signal. + /// + public string? InstanceId { get; set; } + + /// + /// The name of the operation being signaled. + /// + public string? Name { get; set; } + + /// + /// The input of the operation being signaled. + /// + public string? Input { get; set; } + + /// + /// Optionally, a scheduled delivery time for the signal. + /// + public DateTime? ScheduledTime { get; set; } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OperationFormat/StartNewOrchestrationOperationAction.cs b/src/DurableTask.Core/Entities/OperationFormat/StartNewOrchestrationOperationAction.cs new file mode 100644 index 000000000..4c06f80cd --- /dev/null +++ b/src/DurableTask.Core/Entities/OperationFormat/StartNewOrchestrationOperationAction.cs @@ -0,0 +1,56 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities.OperationFormat +{ + using System; + using System.Collections.Generic; + + /// + /// Entity operation action for creating sub-orchestrations. + /// + public class StartNewOrchestrationOperationAction : OperationAction + { + /// + public override OperationActionType OperationActionType => OperationActionType.StartNewOrchestration; + + // NOTE: Actions must be serializable by a variety of different serializer types to support out-of-process execution. + // To ensure maximum compatibility, all properties should be public and settable by default. + + /// + /// Gets or sets the name of the sub-orchestrator to start. + /// + public string? Name { get; set; } + + /// + /// Gets or sets the version of the sub-orchestrator to start. + /// + public string? Version { get; set; } + + /// + /// Gets or sets the instance ID of the created sub-orchestration. + /// + public string? InstanceId { get; set; } + + /// + /// Gets or sets the input of the sub-orchestration. + /// + public string? Input { get; set; } + + /// + /// Gets or sets when to start the orchestration, or null if the orchestration should be started immediately. + /// + public DateTime? ScheduledStartTime { get; set; } + + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs b/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs new file mode 100644 index 000000000..3e21f31a6 --- /dev/null +++ b/src/DurableTask.Core/Entities/OrchestrationEntityContext.cs @@ -0,0 +1,363 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using DurableTask.Core.Entities; + using DurableTask.Core.Entities.EventFormat; + using DurableTask.Core.Entities.OperationFormat; + using DurableTask.Core.Exceptions; + using Newtonsoft.Json; + using Newtonsoft.Json.Linq; + using System; + using System.Collections.Generic; + using System.Linq; + + /// + /// Tracks the entity-related state of an orchestration. + /// Tracks and validates the synchronization state. + /// + public class OrchestrationEntityContext + { + private readonly string instanceId; + private readonly string executionId; + private readonly OrchestrationContext innerContext; + private readonly MessageSorter messageSorter; + + private bool lockAcquisitionPending; + + // the following are null unless we are inside a critical section + private Guid? criticalSectionId; + private EntityId[]? criticalSectionLocks; + private HashSet? availableLocks; + + /// + /// Constructs an OrchestrationEntityContext. + /// + /// The instance id. + /// The execution id. + /// The inner context. + public OrchestrationEntityContext( + string instanceId, + string executionId, + OrchestrationContext innerContext) + { + this.instanceId = instanceId; + this.executionId = executionId; + this.innerContext = innerContext; + this.messageSorter = new MessageSorter(); + } + + /// + /// Checks whether the configured backend supports entities. + /// + public bool EntitiesAreSupported => this.innerContext.EntityParameters != null; + + /// + /// Whether this orchestration is currently inside a critical section. + /// + public bool IsInsideCriticalSection => this.criticalSectionId != null; + + /// + /// The ID of the current critical section, or null if not currently in a critical section. + /// + public Guid? CurrentCriticalSectionId => this.criticalSectionId; + + void CheckEntitySupport() + { + if (!this.EntitiesAreSupported) + { + throw new NotSupportedException("Durable entities are not supported by the current backend configuration."); + } + } + + /// + /// Enumerate all the entities that are available for calling from within a critical section. + /// This set contains all the entities that were locked prior to entering the critical section, + /// and for which there is not currently an operation call pending. + /// + /// An enumeration of all the currently available entities. + public IEnumerable GetAvailableEntities() + { + this.CheckEntitySupport(); + + if (this.IsInsideCriticalSection) + { + foreach (var e in this.availableLocks!) + { + yield return e; + } + } + } + + /// + /// Check that a suborchestration is a valid transition in the current state. + /// + /// The error message, if it is not valid, or null otherwise + /// whether the transition is valid + public bool ValidateSuborchestrationTransition(out string? errorMessage) + { + if (this.IsInsideCriticalSection) + { + errorMessage = "While holding locks, cannot call suborchestrators."; + return false; + } + + errorMessage = null; + return true; + } + + /// + /// Check that acquire is a valid transition in the current state. + /// + /// Whether this is a signal or a call. + /// The target instance id. + /// The error message, if it is not valid, or null otherwise + /// whether the transition is valid + public bool ValidateOperationTransition(string targetInstanceId, bool oneWay, out string? errorMessage) + { + if (this.IsInsideCriticalSection) + { + var lockToUse = EntityId.FromString(targetInstanceId); + if (oneWay) + { + if (this.criticalSectionLocks.Contains(lockToUse)) + { + errorMessage = "Must not signal a locked entity from a critical section."; + return false; + } + } + else + { + if (!this.availableLocks!.Remove(lockToUse)) + { + if (this.lockAcquisitionPending) + { + errorMessage = "Must await the completion of the lock request prior to calling any entity."; + return false; + } + if (this.criticalSectionLocks.Contains(lockToUse)) + { + errorMessage = "Must not call an entity from a critical section while a prior call to the same entity is still pending."; + return false; + } + else + { + errorMessage = "Must not call an entity from a critical section if it is not one of the locked entities."; + return false; + } + } + } + } + + errorMessage = null; + return true; + } + + /// + /// Check that acquire is a valid transition in the current state. + /// + /// The error message, if it is not valid, or null otherwise + /// whether the transition is valid + public bool ValidateAcquireTransition(out string? errorMessage) + { + if (this.IsInsideCriticalSection) + { + errorMessage = "Must not enter another critical section from within a critical section."; + return false; + } + + errorMessage = null; + return true; + } + + /// + /// Called after an operation call within a critical section completes. + /// + /// + public void RecoverLockAfterCall(string targetInstanceId) + { + if (this.IsInsideCriticalSection) + { + var lockToUse = EntityId.FromString(targetInstanceId); + this.availableLocks!.Add(lockToUse); + } + } + + /// + /// Get release messages for all locks in the critical section, and release them + /// + public IEnumerable EmitLockReleaseMessages() + { + if (this.IsInsideCriticalSection) + { + var message = new ReleaseMessage() + { + ParentInstanceId = instanceId, + Id = this.criticalSectionId!.Value.ToString(), + }; + + foreach (var entityId in this.criticalSectionLocks!) + { + var instance = new OrchestrationInstance() { InstanceId = entityId.ToString() }; + yield return new EntityMessageEvent(EntityMessageEventNames.ReleaseMessageEventName, message, instance); + } + + this.criticalSectionLocks = null; + this.availableLocks = null; + this.criticalSectionId = null; + } + } + + /// + /// Creates a request message to be sent to an entity. + /// + /// The target entity. + /// The name of the operation. + /// If true, this is a signal, otherwise it is a call. + /// A unique identifier for this request. + /// A time for which to schedule the delivery, or null if this is not a scheduled message + /// The operation input + /// The event to send. + public EntityMessageEvent EmitRequestMessage( + OrchestrationInstance target, + string operationName, + bool oneWay, + Guid operationId, + (DateTime Original, DateTime Capped)? scheduledTimeUtc, + string? input) + { + this.CheckEntitySupport(); + + var request = new RequestMessage() + { + ParentInstanceId = this.instanceId, + ParentExecutionId = this.executionId, + Id = operationId, + IsSignal = oneWay, + Operation = operationName, + ScheduledTime = scheduledTimeUtc?.Original, + Input = input, + }; + + this.AdjustOutgoingMessage(target.InstanceId, request, scheduledTimeUtc?.Capped, out string eventName); + + return new EntityMessageEvent(eventName, request, target); + } + + /// + /// Creates an acquire message to be sent to an entity. + /// + /// A unique request id. + /// All the entities that are to be acquired. + /// The event to send. + public EntityMessageEvent EmitAcquireMessage(Guid lockRequestId, EntityId[] entities) + { + this.CheckEntitySupport(); + + // All the entities in entity[] need to be locked, but to avoid deadlock, the locks have to be acquired + // sequentially, in order. So, we send the lock request to the first entity; when the first lock + // is granted by the first entity, the first entity will forward the lock request to the second entity, + // and so on; after the last entity grants the last lock, a response is sent back here. + + // acquire the locks in a globally fixed order to avoid deadlocks + Array.Sort(entities); + + // remove duplicates if necessary. Probably quite rare, so no need to optimize more. + for (int i = 0; i < entities.Length - 1; i++) + { + if (entities[i].Equals(entities[i + 1])) + { + entities = entities.Distinct().ToArray(); + break; + } + } + + // send lock request to first entity in the lock set + var target = new OrchestrationInstance() { InstanceId = entities[0].ToString() }; + var request = new RequestMessage() + { + Id = lockRequestId, + ParentInstanceId = this.instanceId, + ParentExecutionId = this.executionId, + LockSet = entities, + Position = 0, + }; + + this.criticalSectionId = lockRequestId; + this.criticalSectionLocks = entities; + this.lockAcquisitionPending = true; + + this.AdjustOutgoingMessage(target.InstanceId, request, null, out string eventName); + + return new EntityMessageEvent(eventName, request, target); + } + + /// + /// Called when a response to the acquire message is received from the last entity. + /// + /// The result returned. + /// The guid for the lock operation + public void CompleteAcquire(OperationResult result, Guid criticalSectionId) + { + this.availableLocks = new HashSet(this.criticalSectionLocks); + this.lockAcquisitionPending = false; + } + + internal void AdjustOutgoingMessage(string instanceId, RequestMessage requestMessage, DateTime? cappedTime, out string eventName) + { + if (cappedTime.HasValue) + { + eventName = EntityMessageEventNames.ScheduledRequestMessageEventName(cappedTime.Value); + } + else + { + this.messageSorter.LabelOutgoingMessage( + requestMessage, + instanceId, + this.innerContext.CurrentUtcDateTime, + this.innerContext.EntityParameters.EntityMessageReorderWindow); + + eventName = EntityMessageEventNames.RequestMessageEventName; + } + } + + /// + /// Extracts the operation result from an event that represents an entity response. + /// + /// The serialized event content. + /// + public OperationResult DeserializeEntityResponseEvent(string eventContent) + { + var responseMessage = new ResponseMessage(); + + // for compatibility, we deserialize in a way that is resilient to any typename presence/absence/mismatch + try + { + // restore the scheduler state from the input + JsonConvert.PopulateObject(eventContent, responseMessage, Serializer.InternalSerializerSettings); + } + catch (Exception exception) + { + throw new EntitySchedulerException("Failed to deserialize entity response.", exception); + } + + return new OperationResult() + { + Result = responseMessage.Result, + ErrorMessage = responseMessage.ErrorMessage, + FailureDetails = responseMessage.FailureDetails, + }; + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/Serializer.cs b/src/DurableTask.Core/Entities/Serializer.cs new file mode 100644 index 000000000..0a99b5f71 --- /dev/null +++ b/src/DurableTask.Core/Entities/Serializer.cs @@ -0,0 +1,30 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using Newtonsoft.Json; + + internal static class Serializer + { + /// + /// This serializer is used exclusively for internally defined data structures and cannot be customized by user. + /// This is intentional, to avoid problems caused by our unability to control the exact format. + /// For example, including typenames can cause compatibility problems if the type name is later changed. + /// + public static JsonSerializer InternalSerializer = JsonSerializer.Create(InternalSerializerSettings); + + public static JsonSerializerSettings InternalSerializerSettings + = new JsonSerializerSettings() { TypeNameHandling = TypeNameHandling.None }; + } +} diff --git a/src/DurableTask.Core/Entities/StateFormat/EntityStatus.cs b/src/DurableTask.Core/Entities/StateFormat/EntityStatus.cs new file mode 100644 index 000000000..6075f14b9 --- /dev/null +++ b/src/DurableTask.Core/Entities/StateFormat/EntityStatus.cs @@ -0,0 +1,58 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System.Runtime.Serialization; + + /// + /// Information about the current status of an entity. Excludes potentially large data + /// (such as the entity state, or the contents of the queue) so it can always be read with low latency. + /// + [DataContract] + public class EntityStatus + { + /// + /// The JSON property name for the entityExists property. + /// + const string EntityExistsProperyName = "entityExists"; + + /// + /// A fast shortcut for checking whether an entity exists, looking at the serialized json string directly. Used by queries. + /// + /// + /// + public static bool TestEntityExists(string serializedJson) + { + return serializedJson.Contains(EntityExistsProperyName); + } + + /// + /// Whether this entity currently has a user-defined state or not. + /// + [DataMember(Name = EntityExistsProperyName, EmitDefaultValue = false)] + public bool EntityExists { get; set; } + + /// + /// The size of the queue, i.e. the number of operations that are waiting for the current operation to complete. + /// + [DataMember(Name = "queueSize", EmitDefaultValue = false)] + public int BacklogQueueSize { get; set; } + + /// + /// The instance id of the orchestration that currently holds the lock of this entity. + /// + [DataMember(Name = "lockedBy", EmitDefaultValue = false)] + public string? LockedBy { get; set; } + } +} diff --git a/src/DurableTask.Core/Entities/StateFormat/MessageSorter.cs b/src/DurableTask.Core/Entities/StateFormat/MessageSorter.cs new file mode 100644 index 000000000..ced0bdb8e --- /dev/null +++ b/src/DurableTask.Core/Entities/StateFormat/MessageSorter.cs @@ -0,0 +1,284 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- + +namespace DurableTask.Core.Entities +{ + using System; + using System.Collections.Generic; + using System.Linq; + using System.Runtime.Serialization; + using DurableTask.Core.Entities.EventFormat; + + /// + /// provides message ordering and deduplication of request messages (operations or lock requests) + /// that are sent to entities, from other entities, or from orchestrations. + /// + [DataContract] + internal class MessageSorter + { + // don't update the reorder window too often since the garbage collection incurs some overhead. + private static readonly TimeSpan MinIntervalBetweenCollections = TimeSpan.FromSeconds(10); + + [DataMember(EmitDefaultValue = false)] + public Dictionary LastSentToInstance { get; set; } + + [DataMember(EmitDefaultValue = false)] + public Dictionary ReceivedFromInstance { get; set; } + + [DataMember(EmitDefaultValue = false)] + public DateTime ReceiveHorizon { get; set; } + + [DataMember(EmitDefaultValue = false)] + public DateTime SendHorizon { get; set; } + + /// + /// Used for testing purposes. + /// + [IgnoreDataMember] + internal int NumberBufferedRequests => + ReceivedFromInstance?.Select(kvp => kvp.Value.Buffered?.Count ?? 0).Sum() ?? 0; + + /// + /// Called on the sending side, to fill in timestamp and predecessor fields. + /// + public void LabelOutgoingMessage(RequestMessage message, string destination, DateTime now, TimeSpan reorderWindow) + { + if (reorderWindow.Ticks == 0) + { + return; // we are not doing any message sorting. + } + + DateTime timestamp = now; + + // whenever (SendHorizon + reorderWindow < now) it is possible to advance the send horizon to (now - reorderWindow) + // and we can then clean out all the no-longer-needed entries of LastSentToInstance. + // However, to reduce the overhead of doing this collection, we don't update the send horizon immediately when possible. + // Instead, we make sure at least MinIntervalBetweenCollections passes between collections. + if (SendHorizon + reorderWindow + MinIntervalBetweenCollections < now) + { + SendHorizon = now - reorderWindow; + + // clean out send clocks that are past the reorder window + + if (LastSentToInstance != null) + { + List expired = new List(); + + foreach (var kvp in LastSentToInstance) + { + if (kvp.Value < SendHorizon) + { + expired.Add(kvp.Key); + } + } + + foreach (var t in expired) + { + LastSentToInstance.Remove(t); + } + } + } + + if (LastSentToInstance == null) + { + LastSentToInstance = new Dictionary(StringComparer.OrdinalIgnoreCase); + } + else if (LastSentToInstance.TryGetValue(destination, out var last)) + { + message.Predecessor = last; + + // ensure timestamps are monotonic even if system clock is not + if (timestamp <= last) + { + timestamp = new DateTime(last.Ticks + 1); + } + } + + message.Timestamp = timestamp; + LastSentToInstance[destination] = timestamp; + } + + /// + /// Called on the receiving side, to reorder and deduplicate within the window. + /// + public IEnumerable ReceiveInOrder(RequestMessage message, TimeSpan reorderWindow) + { + // messages sent from clients and forwarded lock messages are not participating in the sorting. + if (reorderWindow.Ticks == 0 || message.ParentInstanceId == null || message.Position > 0) + { + // Just pass the message through. + yield return message; + yield break; + } + + // whenever (ReceiveHorizon + reorderWindow < message.Timestamp), we can advance the receive horizon to (message.Timestamp - reorderWindow) + // and then we can clean out all the no-longer-needed entries of ReceivedFromInstance. + // However, to reduce the overhead of doing this collection, we don't update the receive horizon immediately when possible. + // Instead, we make sure at least MinIntervalBetweenCollections passes between collections. + if (ReceiveHorizon + reorderWindow + MinIntervalBetweenCollections < message.Timestamp) + { + ReceiveHorizon = message.Timestamp - reorderWindow; + + // deliver any messages that were held in the receive buffers + // but are now past the reorder window + + List buffersToRemove = new List(); + + if (ReceivedFromInstance != null) + { + foreach (var kvp in ReceivedFromInstance) + { + if (kvp.Value.Last < ReceiveHorizon) + { + // we reset Last to MinValue; this means all future messages received + // are treated as if they were the first message received. + kvp.Value.Last = DateTime.MinValue; + } + + while (TryDeliverNextMessage(kvp.Value, out var next)) + { + yield return next; + } + + if (kvp.Value.Last == DateTime.MinValue + && (kvp.Value.Buffered == null || kvp.Value.Buffered.Count == 0)) + { + // we no longer need to store this buffer since it contains no relevant information anymore + // (it is back to its initial "empty" state) + buffersToRemove.Add(kvp.Key); + } + } + + foreach (var t in buffersToRemove) + { + ReceivedFromInstance.Remove(t); + } + + if (ReceivedFromInstance.Count == 0) + { + ReceivedFromInstance = null; + } + } + } + + // Messages older than the reorder window are not participating. + if (message.Timestamp < ReceiveHorizon) + { + // Just pass the message through. + yield return message; + yield break; + } + + ReceiveBuffer receiveBuffer; + + if (ReceivedFromInstance == null) + { + ReceivedFromInstance = new Dictionary(StringComparer.OrdinalIgnoreCase); + } + + if (!ReceivedFromInstance.TryGetValue(message.ParentInstanceId, out receiveBuffer)) + { + ReceivedFromInstance[message.ParentInstanceId] = receiveBuffer = new ReceiveBuffer() + { + ExecutionId = message.ParentExecutionId, + }; + } + else if (receiveBuffer.ExecutionId != message.ParentExecutionId) + { + // this message is from a new execution; release all buffered messages and start over + if (receiveBuffer.Buffered != null) + { + foreach (var kvp in receiveBuffer.Buffered) + { + yield return kvp.Value; + } + + receiveBuffer.Buffered.Clear(); + } + + receiveBuffer.Last = DateTime.MinValue; + receiveBuffer.ExecutionId = message.ParentExecutionId; + } + + if (message.Timestamp <= receiveBuffer.Last) + { + // This message was already delivered, it's a duplicate + yield break; + } + + if (message.Predecessor > receiveBuffer.Last + && message.Predecessor >= ReceiveHorizon) + { + // this message is waiting for a non-delivered predecessor in the window, buffer it + if (receiveBuffer.Buffered == null) + { + receiveBuffer.Buffered = new SortedDictionary(); + } + + receiveBuffer.Buffered[message.Timestamp] = message; + } + else + { + yield return message; + + receiveBuffer.Last = message.Timestamp >= ReceiveHorizon ? message.Timestamp : DateTime.MinValue; + + while (TryDeliverNextMessage(receiveBuffer, out var next)) + { + yield return next; + } + } + } + + private bool TryDeliverNextMessage(ReceiveBuffer buffer, out RequestMessage message) + { + if (buffer.Buffered != null) + { + using (var e = buffer.Buffered.GetEnumerator()) + { + if (e.MoveNext()) + { + var pred = e.Current.Value.Predecessor; + + if (pred <= buffer.Last || pred < ReceiveHorizon) + { + message = e.Current.Value; + + buffer.Last = message.Timestamp >= ReceiveHorizon ? message.Timestamp : DateTime.MinValue; + + buffer.Buffered.Remove(message.Timestamp); + + return true; + } + } + } + } + + message = null; + return false; + } + + [DataContract] + public class ReceiveBuffer + { + [DataMember] + public DateTime Last { get; set; }// last message delivered, or DateTime.Min if none + + [DataMember(EmitDefaultValue = false)] + public string ExecutionId { get; set; } // execution id of last message, if any + + [DataMember(EmitDefaultValue = false)] + public SortedDictionary Buffered { get; set; } + } + } +} diff --git a/src/DurableTask.Core/Entities/StateFormat/SchedulerState.cs b/src/DurableTask.Core/Entities/StateFormat/SchedulerState.cs new file mode 100644 index 000000000..aa5abe5f7 --- /dev/null +++ b/src/DurableTask.Core/Entities/StateFormat/SchedulerState.cs @@ -0,0 +1,115 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using System.Collections.Generic; + using System.Runtime.Serialization; + using DurableTask.Core.Entities.EventFormat; + + /// + /// The persisted state of an entity scheduler, as handed forward between ContinueAsNew instances. + /// + [DataContract] + internal class SchedulerState + { + [IgnoreDataMember] + public bool EntityExists => this.EntityState != null; + + /// + /// The last serialized entity state. + /// + [DataMember(Name = "state", EmitDefaultValue = false)] + public string? EntityState { get; set; } + + /// + /// The queue of waiting operations, or null if none. + /// + [DataMember(Name = "queue", EmitDefaultValue = false)] + public Queue? Queue { get; private set; } + + /// + /// The instance id of the orchestration that currently holds the lock of this entity. + /// + [DataMember(Name = "lockedBy", EmitDefaultValue = false)] + public string? LockedBy { get; set; } + + /// + /// Whether processing on this entity is currently suspended. + /// + [DataMember(Name = "suspended", EmitDefaultValue = false)] + public bool Suspended { get; set; } + + /// + /// The metadata used for reordering and deduplication of messages sent to entities. + /// + [DataMember(Name = "sorter", EmitDefaultValue = false)] + public MessageSorter MessageSorter { get; set; } = new MessageSorter(); + + [IgnoreDataMember] + public bool IsEmpty => !EntityExists && (Queue == null || Queue.Count == 0) && LockedBy == null; + + internal void Enqueue(RequestMessage operationMessage) + { + if (Queue == null) + { + Queue = new Queue(); + } + + Queue.Enqueue(operationMessage); + } + + internal void PutBack(Queue messages) + { + if (Queue != null) + { + foreach (var message in Queue) + { + messages.Enqueue(message); + } + } + + Queue = messages; + } + + internal bool MayDequeue() + { + return Queue != null + && Queue.Count > 0 + && (LockedBy == null || LockedBy == Queue.Peek().ParentInstanceId); + } + + internal RequestMessage Dequeue() + { + if (this.Queue == null) + { + throw new InvalidOperationException("Queue is empty"); + } + + var result = Queue.Dequeue(); + + if (Queue.Count == 0) + { + Queue = null; + } + + return result; + } + + public override string ToString() + { + return $"exists={EntityExists} queue.count={(Queue != null ? Queue.Count : 0)}"; + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Entities/TaskEntity.cs b/src/DurableTask.Core/Entities/TaskEntity.cs new file mode 100644 index 000000000..b3277a9ff --- /dev/null +++ b/src/DurableTask.Core/Entities/TaskEntity.cs @@ -0,0 +1,30 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- + +#nullable enable +namespace DurableTask.Core.Entities +{ + using System.Threading.Tasks; + using DurableTask.Core.Entities.OperationFormat; + + /// + /// Abstract base class for entities. + /// + public abstract class TaskEntity + { + /// + /// Execute a batch of operations on an entity. + /// + public abstract Task ExecuteOperationBatchAsync(EntityBatchRequest operations); + } +} diff --git a/src/DurableTask.Core/Entities/TaskOrchestrationEntityParameters.cs b/src/DurableTask.Core/Entities/TaskOrchestrationEntityParameters.cs new file mode 100644 index 000000000..758710ad1 --- /dev/null +++ b/src/DurableTask.Core/Entities/TaskOrchestrationEntityParameters.cs @@ -0,0 +1,48 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Entities +{ + using System; + using DurableTask.Core.Serializing; + + /// + /// Settings that determine how a task orchestrator interacts with entities. + /// + public class TaskOrchestrationEntityParameters + { + /// + /// The time window within which entity messages should be deduplicated and reordered. + /// This is zero for providers that already guarantee exactly-once and ordered delivery. + /// + public TimeSpan EntityMessageReorderWindow { get; set; } + + /// + /// Construct a based on the given backend properties. + /// + /// The backend properties. + /// The constructed object, or null if is null. + public static TaskOrchestrationEntityParameters? FromEntityBackendProperties(EntityBackendProperties? properties) + { + if (properties == null) + { + return null; + } + + return new TaskOrchestrationEntityParameters() + { + EntityMessageReorderWindow = properties.EntityMessageReorderWindow, + }; + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Exceptions/EntitySchedulerException.cs b/src/DurableTask.Core/Exceptions/EntitySchedulerException.cs new file mode 100644 index 000000000..d79541cf4 --- /dev/null +++ b/src/DurableTask.Core/Exceptions/EntitySchedulerException.cs @@ -0,0 +1,61 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +#nullable enable +namespace DurableTask.Core.Exceptions +{ + using System; + using System.Runtime.Serialization; + + /// + /// Exception used to describe various issues encountered by the entity scheduler. + /// + [Serializable] + public class EntitySchedulerException : Exception + { + /// + /// Initializes a new instance of the class. + /// + public EntitySchedulerException() + { + } + + /// + /// Initializes a new instance of the class. + /// + /// The message that describes the error. + public EntitySchedulerException(string message) + : base(message) + { + } + + /// + /// Initializes an new instance of the class. + /// + /// The message that describes the error. + /// The exception that was caught. + public EntitySchedulerException(string errorMessage, Exception innerException) + : base(errorMessage, innerException) + { + } + + /// + /// Initializes a new instance of the class with serialized data. + /// + /// The System.Runtime.Serialization.SerializationInfo that holds the serialized object data about the exception being thrown. + /// The System.Runtime.Serialization.StreamingContext that contains contextual information about the source or destination. + protected EntitySchedulerException(SerializationInfo info, StreamingContext context) + : base(info, context) + { + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/Exceptions/OrchestrationException.cs b/src/DurableTask.Core/Exceptions/OrchestrationException.cs index 07e2ff9e1..8c8a144af 100644 --- a/src/DurableTask.Core/Exceptions/OrchestrationException.cs +++ b/src/DurableTask.Core/Exceptions/OrchestrationException.cs @@ -88,8 +88,8 @@ public override void GetObjectData(SerializationInfo info, StreamingContext cont public int EventId { get; set; } /// - /// Gets additional details about the failure. May be null if the failure details collection is not enabled. + /// Gets or sets additional details about the failure. May be null if the failure details collection is not enabled. /// - public FailureDetails? FailureDetails { get; internal set; } + public FailureDetails? FailureDetails { get; set; } } } \ No newline at end of file diff --git a/src/DurableTask.Core/FailureDetails.cs b/src/DurableTask.Core/FailureDetails.cs index b99584833..5dfd02ab9 100644 --- a/src/DurableTask.Core/FailureDetails.cs +++ b/src/DurableTask.Core/FailureDetails.cs @@ -14,12 +14,18 @@ namespace DurableTask.Core { using System; + using System.Collections.Generic; + using System.Linq; + using System.Reflection; using System.Runtime.Serialization; using DurableTask.Core.Exceptions; using Newtonsoft.Json; + // NOTE: This class is very similar to https://github.com/microsoft/durabletask-dotnet/blob/main/src/Abstractions/TaskFailureDetails.cs. + // Any functional changes to this class should be mirrored in that class and vice versa. + /// - /// Details of an activity or orchestration failure. + /// Details of an activity, orchestration, or entity operation failure. /// [Serializable] public class FailureDetails : IEquatable @@ -42,6 +48,16 @@ public FailureDetails(string errorType, string errorMessage, string? stackTrace, this.IsNonRetriable = isNonRetriable; } + /// + /// Initializes a new instance of the class from an exception object. + /// + /// The exception used to generate the failure details. + /// The inner cause of the failure. + public FailureDetails(Exception e, FailureDetails innerFailure) + : this(e.GetType().FullName, GetErrorMessage(e), e.StackTrace, innerFailure, false) + { + } + /// /// Initializes a new instance of the class from an exception object. /// @@ -116,7 +132,33 @@ public override string ToString() /// Returns true if the value matches ; false otherwise. public bool IsCausedBy() where T : Exception { + // This check works for .NET exception types defined in System.Core.PrivateLib (aka mscorelib.dll) Type? exceptionType = Type.GetType(this.ErrorType, throwOnError: false); + + // For exception types defined in the same assembly as the target exception type. + exceptionType ??= typeof(T).Assembly.GetType(this.ErrorType, throwOnError: false); + + // For custom exception types defined in the app's assembly. + exceptionType ??= Assembly.GetCallingAssembly().GetType(this.ErrorType, throwOnError: false); + + if (exceptionType == null) + { + // This last check works for exception types defined in any loaded assembly (e.g. NuGet packages, etc.). + // This is a fallback that should rarely be needed except in obscure cases. + List matchingExceptionTypes = AppDomain.CurrentDomain.GetAssemblies() + .Select(a => a.GetType(this.ErrorType, throwOnError: false)) + .Where(t => t is not null) + .ToList(); + if (matchingExceptionTypes.Count == 1) + { + exceptionType = matchingExceptionTypes[0]; + } + else if (matchingExceptionTypes.Count > 1) + { + throw new AmbiguousMatchException($"Multiple exception types with the name '{this.ErrorType}' were found."); + } + } + return exceptionType != null && typeof(T).IsAssignableFrom(exceptionType); } diff --git a/src/DurableTask.Core/History/README.md b/src/DurableTask.Core/History/README.md index 17f5422e2..7f99fe5ec 100644 --- a/src/DurableTask.Core/History/README.md +++ b/src/DurableTask.Core/History/README.md @@ -11,10 +11,14 @@ The following are some common history events that make up an orchestration's sta | `TaskCompleted` | A scheduled task activity has completed successfully. The `TaskScheduledId` field will match the `EventId` field of the corresponding `TaskScheduled` event. | | `TaskFailed` | A scheduled task activity has completed with a failure. The `TaskScheduledId` field will match the `EventId` field of the corresponding `TaskScheduled` event. | | `SubOrchestrationInstanceCreated` | The orchestrator has scheduled a sub-orchestrator. This event contains the name, instance ID, input, and ordered event ID of the scheduled orchestrator, which can be used to correlate the `SubOrchestrationInstanceCreated` event with a subsequent `SubOrchestrationInstanceCompleted` or `SubOrchestrationInstanceFailed` history event. The timestamp refers to the time at which the sub-orchestrator was _scheduled_, which will be earlier than the time in which it starts executing. Note that there may be multiple `SubOrchestrationInstance***` events generated if an activity task is retried. | -| `SubOrchestrationInstanceCompleted` | A scheduled a sub-orchestrator has completed successfully. The `TaskScheduledId` field will match the `EventId` field of the corresponding `SubOrchestrationInstanceCreated` event. | -| `SubOrchestrationInstanceFailed` | A scheduled a sub-orchestrator has completed with a failure. The `TaskScheduledId` field will match the `EventId` field of the corresponding `SubOrchestrationInstanceCreated` event. | +| `SubOrchestrationInstanceCompleted` | A scheduled sub-orchestrator has completed successfully. The `TaskScheduledId` field will match the `EventId` field of the corresponding `SubOrchestrationInstanceCreated` event. | +| `SubOrchestrationInstanceFailed` | A scheduled sub-orchestrator has completed with a failure. The `TaskScheduledId` field will match the `EventId` field of the corresponding `SubOrchestrationInstanceCreated` event. | +| `TimerCreated` | The orchestrator scheduled a durable timer. The `FireAt` property contains the date at which the timer will fire. | +| `TimerFired` | A previously scheduled durable timer has fired. The `TimerId` field will match the `EventId` field of the corresponding `TimeCreated` event. | | `EventRaised` | An orchestration (or entity in the case of [Durable Entities](https://docs.microsoft.com/azure/azure-functions/durable/durable-functions-entities)) has received an external event. This record contains the name of the event, the payload, and the timestamp for when the event was _sent_ (which should be around the same time or earlier than when this history event was actually persisted). | | `EventSent` | An orchestration (or entity) has sent a one-way message to another orchestration (or entity). | | `ExecutionCompleted` | The orchestration has completed. This event includes the output of the orchestration and doesn't distinguish between success or failure. | | `ExecutionTerminated` | The orchestration was forcefully terminated by an API call. The timestamp of this event represents the time at which termination was _scheduled_ and not necessarily the time at which it actually terminated. | | `OrchestratorCompleted` | The orchestrator function has awaited and committed any side effects. You will see many of these events in your history - one for each time that an orchestrator awaits. Note that this does _NOT_ mean that the orchestrator has completed (completion is represented by either `ExecutionCompleted` or `ExecutionTerminated`). | +| `GenericEvent` | A generic history event with a `Data` field, but has no specific meaning. This history event is not commonly used. In some cases, this event is used to trigger a fresh replay of an idle orchestration, such as after an orchestration is rewound. | +| `HistoryStateEvent` | A history event that contains a snapshot of the orchestration history. This event type is not used in most modern backend types. | diff --git a/src/DurableTask.Core/Logging/EventIds.cs b/src/DurableTask.Core/Logging/EventIds.cs index 963de5f71..f7386eb99 100644 --- a/src/DurableTask.Core/Logging/EventIds.cs +++ b/src/DurableTask.Core/Logging/EventIds.cs @@ -47,6 +47,10 @@ static class EventIds public const int OrchestrationExecuted = 52; public const int OrchestrationAborted = 53; public const int DiscardingMessage = 54; + public const int EntityBatchExecuting = 55; + public const int EntityBatchExecuted = 56; + public const int EntityLockAcquired = 57; + public const int EntityLockReleased = 58; public const int TaskActivityStarting = 60; public const int TaskActivityCompleted = 61; @@ -63,5 +67,7 @@ static class EventIds public const int RenewOrchestrationWorkItemStarting = 70; public const int RenewOrchestrationWorkItemCompleted = 71; public const int RenewOrchestrationWorkItemFailed = 72; + + public const int OrchestrationDebugTrace = 73; } } diff --git a/src/DurableTask.Core/Logging/LogEvents.cs b/src/DurableTask.Core/Logging/LogEvents.cs index a914154a2..612e2664f 100644 --- a/src/DurableTask.Core/Logging/LogEvents.cs +++ b/src/DurableTask.Core/Logging/LogEvents.cs @@ -14,9 +14,11 @@ namespace DurableTask.Core.Logging { using System; + using System.Linq; using System.Text; using DurableTask.Core.Command; using DurableTask.Core.Common; + using DurableTask.Core.Entities.OperationFormat; using DurableTask.Core.History; using Microsoft.Extensions.Logging; @@ -1177,6 +1179,223 @@ void IEventSourceEvent.WriteEventSource() => Utils.PackageVersion); } + /// + /// Log event representing a task hub worker executing a batch of entity operations. + /// + internal class EntityBatchExecuting : StructuredLogEvent, IEventSourceEvent + { + public EntityBatchExecuting(EntityBatchRequest request) + { + this.InstanceId = request.InstanceId; + this.OperationCount = request.Operations.Count; + this.EntityStateLength = request.EntityState?.Length ?? 0; + } + + [StructuredLogField] + public string InstanceId { get; } + + [StructuredLogField] + public int OperationCount { get; } + + [StructuredLogField] + public int EntityStateLength { get; } + + public override EventId EventId => new EventId( + EventIds.EntityBatchExecuting, + nameof(EventIds.EntityBatchExecuting)); + + public override LogLevel Level => LogLevel.Debug; + + protected override string CreateLogMessage() => + $"{this.InstanceId}: executing batch of {this.OperationCount} operations on entity state of length {this.EntityStateLength}."; + + void IEventSourceEvent.WriteEventSource() => + StructuredEventSource.Log.EntityBatchExecuting( + this.InstanceId, + this.OperationCount, + this.EntityStateLength, + Utils.AppName, + Utils.PackageVersion); + } + + /// + /// Log event representing a task hub worker executed a batch of entity operations. + /// + internal class EntityBatchExecuted : StructuredLogEvent, IEventSourceEvent + { + public EntityBatchExecuted(EntityBatchRequest request, EntityBatchResult result) + { + this.InstanceId = request.InstanceId; + this.OperationCount = request.Operations.Count; + this.ResultCount = result.Results.Count; + this.ErrorCount = result.Results.Count(x => x.IsError); + this.ActionCount = result.Actions.Count; + this.EntityStateLength = request.EntityState?.Length ?? 0; + } + + [StructuredLogField] + public string InstanceId { get; } + + [StructuredLogField] + public int OperationCount { get; } + + [StructuredLogField] + public int ResultCount { get; } + + [StructuredLogField] + public int ErrorCount { get; } + + [StructuredLogField] + public int ActionCount { get; } + + [StructuredLogField] + public int EntityStateLength { get; } + + public override EventId EventId => new EventId( + EventIds.EntityBatchExecuted, + nameof(EventIds.EntityBatchExecuted)); + + public override LogLevel Level => LogLevel.Information; + + protected override string CreateLogMessage() => + $"{this.InstanceId}: completed {this.ResultCount} of {this.OperationCount} entity operations, resulting in {this.ErrorCount} errors, {this.ActionCount} actions, and entity state of length {this.EntityStateLength}."; + + void IEventSourceEvent.WriteEventSource() => + StructuredEventSource.Log.EntityBatchExecuted( + this.InstanceId, + this.OperationCount, + this.ResultCount, + this.ErrorCount, + this.ActionCount, + this.EntityStateLength, + Utils.AppName, + Utils.PackageVersion); + } + + /// + /// Logs that an entity processed a lock acquire message. + /// + internal class EntityLockAcquired : StructuredLogEvent, IEventSourceEvent + { + public EntityLockAcquired(string entityId, Core.Entities.EventFormat.RequestMessage message) + { + this.EntityId = entityId; + this.InstanceId = message.ParentInstanceId; + this.ExecutionId = message.ParentExecutionId; + this.CriticalSectionId = message.Id; + this.Position = message.Position; + + if (message.LockSet != null) + { + this.LockSet = string.Join(",", message.LockSet.Select(id => id.ToString())); + } + } + + /// + /// The entity that is being locked. + /// + [StructuredLogField] + public string EntityId { get; } + + /// + /// The instance ID of the orchestration that is executing the critical section. + /// + [StructuredLogField] + public string InstanceId { get; set; } + + /// + /// The execution ID of the orchestration that is executing the critical section. + /// + [StructuredLogField] + public string ExecutionId { get; set; } + + /// + /// The unique ID of the critical section that is acquiring this lock. + /// + [StructuredLogField] + public Guid CriticalSectionId { get; set; } + + /// + /// The ordered set of locks that are being acquired for this critical section. + /// + [StructuredLogField] + public string LockSet { get; set; } + + /// + /// Which of the locks in is being acquired. + /// + [StructuredLogField] + public int Position { get; set; } + + public override EventId EventId => new EventId( + EventIds.EntityLockAcquired, + nameof(EventIds.EntityLockAcquired)); + + public override LogLevel Level => LogLevel.Information; + + protected override string CreateLogMessage() => + $"{this.EntityId}: acquired lock {this.Position+1}/{this.LockSet.Length} for orchestration instanceId={this.InstanceId} executionId={this.ExecutionId} criticalSectionId={this.CriticalSectionId}"; + + void IEventSourceEvent.WriteEventSource() => + StructuredEventSource.Log.EntityLockAcquired( + this.EntityId, + this.InstanceId ?? string.Empty, + this.ExecutionId ?? string.Empty, + this.CriticalSectionId, + this.LockSet ?? string.Empty, + this.Position, + Utils.AppName, + Utils.PackageVersion); + } + + /// + /// Logs that an entity processed a lock release message. + /// + internal class EntityLockReleased : StructuredLogEvent, IEventSourceEvent + { + public EntityLockReleased(string entityId, Core.Entities.EventFormat.ReleaseMessage message) + { + this.EntityId = entityId; + this.InstanceId = message.ParentInstanceId; + this.CriticalSectionId = message.Id; + } + + /// + /// The entity that is being unlocked. + /// + [StructuredLogField] + public string EntityId { get; } + + /// + /// The instance ID of the orchestration that is executing the critical section. + /// + [StructuredLogField] + public string InstanceId { get; set; } + + /// + /// The unique ID of the critical section that is releasing the lock after completing. + /// + [StructuredLogField] + public string CriticalSectionId { get; set; } + + public override EventId EventId => new EventId( + EventIds.EntityLockReleased, + nameof(EventIds.EntityLockReleased)); + + public override LogLevel Level => LogLevel.Information; + + protected override string CreateLogMessage() => + $"{this.EntityId}: released lock for orchestration instanceId={this.InstanceId} criticalSectionId={this.CriticalSectionId}"; + + void IEventSourceEvent.WriteEventSource() => + StructuredEventSource.Log.EntityLockReleased( + this.EntityId, + this.InstanceId ?? string.Empty, + this.CriticalSectionId ?? string.Empty, + Utils.AppName, + Utils.PackageVersion); + } + /// /// Log event indicating that an activity execution is starting. /// @@ -1629,5 +1848,45 @@ void IEventSourceEvent.WriteEventSource() => Utils.AppName, Utils.PackageVersion); } + + internal class OrchestrationDebugTrace : StructuredLogEvent, IEventSourceEvent + { + public OrchestrationDebugTrace(string instanceId, string executionId, string details) + { + this.InstanceId = instanceId; + this.ExecutionId = executionId; + this.Details = details; + } + + [StructuredLogField] + public string InstanceId { get; } + + [StructuredLogField] + public string ExecutionId { get; } + + [StructuredLogField] + public string Name { get; } + + [StructuredLogField] + public string Details { get; } + + public override EventId EventId => new EventId( + EventIds.OrchestrationDebugTrace, + nameof(EventIds.OrchestrationDebugTrace)); + + public override LogLevel Level => LogLevel.Debug; + + protected override string CreateLogMessage() => + $"{this.InstanceId}: Orchestration Debug Trace: {this.Details}"; + + void IEventSourceEvent.WriteEventSource() => + StructuredEventSource.Log.OrchestrationDebugTrace( + this.InstanceId, + this.ExecutionId, + this.Details, + Utils.AppName, + Utils.PackageVersion); + } + } } diff --git a/src/DurableTask.Core/Logging/LogHelper.cs b/src/DurableTask.Core/Logging/LogHelper.cs index f4d1ffe34..109c14c42 100644 --- a/src/DurableTask.Core/Logging/LogHelper.cs +++ b/src/DurableTask.Core/Logging/LogHelper.cs @@ -17,7 +17,7 @@ namespace DurableTask.Core.Logging using System.Collections.Generic; using System.Text; using DurableTask.Core.Command; - using DurableTask.Core.Common; + using DurableTask.Core.Entities.OperationFormat; using DurableTask.Core.History; using Microsoft.Extensions.Logging; @@ -561,6 +561,58 @@ internal void RenewOrchestrationWorkItemFailed(TaskOrchestrationWorkItem workIte } } + + /// + /// Logs that an entity operation batch is about to start executing. + /// + /// The batch request. + internal void EntityBatchExecuting(EntityBatchRequest request) + { + if (this.IsStructuredLoggingEnabled) + { + this.WriteStructuredLog(new LogEvents.EntityBatchExecuting(request)); + } + } + + /// + /// Logs that an entity operation batch completed its execution. + /// + /// The batch request. + /// The batch result. + internal void EntityBatchExecuted(EntityBatchRequest request, EntityBatchResult result) + { + if (this.IsStructuredLoggingEnabled) + { + this.WriteStructuredLog(new LogEvents.EntityBatchExecuted(request, result)); + } + } + + /// + /// Logs that an entity processed a lock acquire message. + /// + /// The entity id. + /// The message. + internal void EntityLockAcquired(string entityId, Core.Entities.EventFormat.RequestMessage message) + { + if (this.IsStructuredLoggingEnabled) + { + this.WriteStructuredLog(new LogEvents.EntityLockAcquired(entityId, message)); + } + } + + /// + /// Logs that an entity processed a lock release message. + /// + /// The entity id. + /// The message. + internal void EntityLockReleased(string entityId, Core.Entities.EventFormat.ReleaseMessage message) + { + if (this.IsStructuredLoggingEnabled) + { + this.WriteStructuredLog(new LogEvents.EntityLockReleased(entityId, message)); + } + } + #endregion #region Activity dispatcher @@ -678,6 +730,14 @@ internal void RenewActivityMessageFailed(TaskActivityWorkItem workItem, Exceptio } #endregion + internal void OrchestrationDebugTrace(string instanceId, string executionId, string details) + { + if (this.IsStructuredLoggingEnabled) + { + this.WriteStructuredLog(new LogEvents.OrchestrationDebugTrace(instanceId, executionId, details)); + } + } + void WriteStructuredLog(ILogEvent logEvent, Exception? exception = null) { this.log?.LogDurableEvent(logEvent, exception); diff --git a/src/DurableTask.Core/Logging/StructuredEventSource.cs b/src/DurableTask.Core/Logging/StructuredEventSource.cs index b9edc0a46..162a226f2 100644 --- a/src/DurableTask.Core/Logging/StructuredEventSource.cs +++ b/src/DurableTask.Core/Logging/StructuredEventSource.cs @@ -624,6 +624,102 @@ internal void DiscardingMessage( } } + [Event(EventIds.EntityBatchExecuting, Level = EventLevel.Informational, Version = 1)] + internal void EntityBatchExecuting( + string InstanceId, + int OperationCount, + int EntityStateLength, + string AppName, + string ExtensionVersion) + { + if (this.IsEnabled(EventLevel.Informational)) + { + // TODO: Use WriteEventCore for better performance + this.WriteEvent( + EventIds.EntityBatchExecuting, + InstanceId, + OperationCount, + EntityStateLength, + AppName, + ExtensionVersion); + } + } + + [Event(EventIds.EntityBatchExecuted, Level = EventLevel.Informational, Version = 1)] + internal void EntityBatchExecuted( + string InstanceId, + int OperationCount, + int ResultCount, + int ErrorCount, + int ActionCount, + int EntityStateLength, + string AppName, + string ExtensionVersion) + { + if (this.IsEnabled(EventLevel.Informational)) + { + // TODO: Use WriteEventCore for better performance + this.WriteEvent( + EventIds.EntityBatchExecuted, + InstanceId, + OperationCount, + ResultCount, + ErrorCount, + ActionCount, + EntityStateLength, + AppName, + ExtensionVersion); + } + } + + [Event(EventIds.EntityLockAcquired, Level = EventLevel.Informational, Version = 1)] + internal void EntityLockAcquired( + string EntityId, + string InstanceId, + string ExecutionId, + Guid CriticalSectionId, + string LockSet, + int Position, + string AppName, + string ExtensionVersion) + { + if (this.IsEnabled(EventLevel.Informational)) + { + // TODO: Use WriteEventCore for better performance + this.WriteEvent( + EventIds.EntityLockAcquired, + EntityId, + InstanceId, + ExecutionId, + CriticalSectionId, + LockSet, + Position, + AppName, + ExtensionVersion); + } + } + + [Event(EventIds.EntityLockReleased, Level = EventLevel.Informational, Version = 1)] + internal void EntityLockReleased( + string EntityId, + string InstanceId, + string Id, + string AppName, + string ExtensionVersion) + { + if (this.IsEnabled(EventLevel.Informational)) + { + // TODO: Use WriteEventCore for better performance + this.WriteEvent( + EventIds.EntityLockReleased, + EntityId, + InstanceId, + Id, + AppName, + ExtensionVersion); + } + } + [Event(EventIds.TaskActivityStarting, Level = EventLevel.Informational, Version = 1)] internal void TaskActivityStarting( string InstanceId, @@ -865,5 +961,25 @@ internal void RenewOrchestrationWorkItemFailed( ExtensionVersion); } } + + [Event(EventIds.OrchestrationDebugTrace, Level = EventLevel.Verbose, Version = 1)] + internal void OrchestrationDebugTrace( + string InstanceId, + string ExecutionId, + string Details, + string AppName, + string ExtensionVersion) + { + if (this.IsEnabled(EventLevel.Verbose)) + { + this.WriteEvent( + EventIds.OrchestrationDebugTrace, + InstanceId, + ExecutionId, + Details, + AppName, + ExtensionVersion); + } + } } } diff --git a/src/DurableTask.Core/NameVersionHelper.cs b/src/DurableTask.Core/NameVersionHelper.cs index f45ad94c0..52e09c88d 100644 --- a/src/DurableTask.Core/NameVersionHelper.cs +++ b/src/DurableTask.Core/NameVersionHelper.cs @@ -14,7 +14,9 @@ namespace DurableTask.Core { using System; + using System.Collections.Generic; using System.Dynamic; + using System.Linq; using System.Reflection; /// @@ -99,5 +101,56 @@ internal static string GetFullyQualifiedMethodName(string declaringType, string return declaringType + "." + methodName; } + + /// + /// Gets the fully qualified method name by joining a prefix representing the declaring type and a suffix representing the parameter list. + /// For example, + /// "DurableTask.Emulator.Tests.EmulatorFunctionalTests+IInheritedTestOrchestrationTasksB`2[System.Int32,System.String].Juggle.(Int32,Boolean)" + /// would be the result for the method `Juggle(int, bool)` as member of + /// generic type interface declared like `DurableTask.Emulator.Tests.EmulatorFunctionalTests.IInheritedTestOrchestrationTasksB{int, string}`, + /// even if the method were inherited from a base interface. + /// + /// typically the result of call to Type.ToString(): Type.FullName is more verbose + /// + /// + internal static string GetFullyQualifiedMethodName(string declaringType, MethodInfo methodInfo) + { + IEnumerable paramTypeNames = methodInfo.GetParameters().Select(x => x.ParameterType.Name); + string paramTypeNamesCsv = string.Join(",", paramTypeNames); + string methodNameWithParameterList = $"{methodInfo.Name}.({paramTypeNamesCsv})"; + return GetFullyQualifiedMethodName(declaringType, methodNameWithParameterList); + } + + /// + /// Gets all methods from an interface, including those inherited from a base interface + /// + /// + /// + /// + /// + internal static IList GetAllInterfaceMethods(Type t, Func getMethodUniqueId, HashSet visited = null) + { + if (visited == null) + { + visited = new HashSet(); + } + List result = new List(); + foreach (MethodInfo m in t.GetMethods()) + { + string name = getMethodUniqueId(m); + if (!visited.Contains(name)) + { + // In some cases, such as when a generic type interface inherits an interface with the same name, Task.GetMethod includes the methods from the base interface. + // This check is to avoid dupicates from these. + result.Add(m); + visited.Add(name); + } + } + foreach (Type baseInterface in t.GetInterfaces()) + { + result.AddRange(GetAllInterfaceMethods(baseInterface, getMethodUniqueId, visited: visited)); + } + return result; + } } } \ No newline at end of file diff --git a/src/DurableTask.Core/OrchestrationContext.cs b/src/DurableTask.Core/OrchestrationContext.cs index 52238bbc2..4b2d8c7fe 100644 --- a/src/DurableTask.Core/OrchestrationContext.cs +++ b/src/DurableTask.Core/OrchestrationContext.cs @@ -18,6 +18,7 @@ namespace DurableTask.Core using System.Threading; using System.Threading.Tasks; using Castle.DynamicProxy; + using DurableTask.Core.Entities; using DurableTask.Core.Serializing; /// @@ -52,6 +53,11 @@ public abstract class OrchestrationContext /// public OrchestrationInstance OrchestrationInstance { get; internal protected set; } + /// + /// Version of the currently executing orchestration + /// + public string Version { get; internal protected set; } + /// /// Replay-safe current UTC datetime /// @@ -67,6 +73,11 @@ public abstract class OrchestrationContext /// internal ErrorPropagationMode ErrorPropagationMode { get; set; } + /// + /// Information about backend entity support, or null if the configured backend does not support entities. + /// + internal TaskOrchestrationEntityParameters EntityParameters { get; set; } + /// /// Create a proxy client class to schedule remote TaskActivities via a strongly typed interface. /// @@ -85,27 +96,25 @@ public virtual T CreateClient() where T : class /// If true, the method name translation from the interface contains /// the interface name, if false then only the method name is used /// + /// + /// This is deprecated and exists only for back-compatibility. + /// See , which adds support for C# interface features such as inheritance, generics, and method overloading. + /// /// public virtual T CreateClient(bool useFullyQualifiedMethodNames) where T : class { - if (!typeof(T).IsInterface && !typeof(T).IsClass) - { - throw new InvalidOperationException($"{nameof(T)} must be an interface or class."); - } - - IInterceptor scheduleProxy = new ScheduleProxy(this, useFullyQualifiedMethodNames); - - if (typeof(T).IsClass) - { - if (typeof(T).IsSealed) - { - throw new InvalidOperationException("Class cannot be sealed."); - } - - return ProxyGenerator.CreateClassProxy(scheduleProxy); - } + return CreateClient(() => new ScheduleProxy(this, useFullyQualifiedMethodNames)); + } - return ProxyGenerator.CreateInterfaceProxyWithoutTarget(scheduleProxy); + /// + /// Create a proxy client class to schedule remote TaskActivities via a strongly typed interface. + /// + /// + /// + /// + public virtual T CreateClientV2() where T : class + { + return CreateClient(() => new ScheduleProxyV2(this, typeof(T).ToString())); } /// @@ -404,5 +413,33 @@ public abstract Task CreateSubOrchestrationInstance(string name, string ve /// the first execution of this orchestration instance. /// public abstract void ContinueAsNew(string newVersion, object input); + + /// + /// Create a proxy client class to schedule remote TaskActivities via a strongly typed interface. + /// + /// + /// + /// + private static T CreateClient(Func createScheduleProxy) where T : class + { + if (!typeof(T).IsInterface && !typeof(T).IsClass) + { + throw new InvalidOperationException($"{nameof(T)} must be an interface or class."); + } + + IInterceptor scheduleProxy = createScheduleProxy(); + + if (typeof(T).IsClass) + { + if (typeof(T).IsSealed) + { + throw new InvalidOperationException("Class cannot be sealed."); + } + + return ProxyGenerator.CreateClassProxy(scheduleProxy); + } + + return ProxyGenerator.CreateInterfaceProxyWithoutTarget(scheduleProxy); + } } } \ No newline at end of file diff --git a/src/DurableTask.Core/OrchestrationRuntimeState.cs b/src/DurableTask.Core/OrchestrationRuntimeState.cs index e9401658b..f4e61afd4 100644 --- a/src/DurableTask.Core/OrchestrationRuntimeState.cs +++ b/src/DurableTask.Core/OrchestrationRuntimeState.cs @@ -18,6 +18,7 @@ namespace DurableTask.Core using System.Diagnostics; using DurableTask.Core.Common; using DurableTask.Core.History; + using DurableTask.Core.Logging; using DurableTask.Core.Tracing; /// @@ -184,6 +185,14 @@ public OrchestrationStatus OrchestrationStatus this.Events.Count == 1 && this.Events[0].EventType == EventType.OrchestratorStarted || this.ExecutionStartedEvent != null; + /// + /// Gets or sets a LogHelper instance that can be used to log messages. + /// + /// + /// Ideally, this would be set in the constructor but that would require a larger refactoring. + /// + internal LogHelper? LogHelper { get; set; } = null; + /// /// Adds a new history event to the Events list and NewEvents list /// @@ -260,14 +269,33 @@ void SetMarkerEvents(HistoryEvent historyEvent) } else if (historyEvent is ExecutionCompletedEvent completedEvent) { - if (ExecutionCompletedEvent != null) + if (ExecutionCompletedEvent == null) { - throw new InvalidOperationException( - "Multiple ExecutionCompletedEvent found, potential corruption in state storage"); + ExecutionCompletedEvent = completedEvent; + orchestrationStatus = completedEvent.OrchestrationStatus; + } + else + { + // It's not generally expected to receive multiple execution completed events for a given orchestrator, but it's possible under certain race conditions. + // For example: when an orchestrator is signaled to terminate at the same time as it attempts to continue-as-new. + var log = $"Received new {completedEvent.GetType().Name} event despite the orchestration being already in the {orchestrationStatus} state."; + + if (orchestrationStatus == OrchestrationStatus.ContinuedAsNew && completedEvent.OrchestrationStatus == OrchestrationStatus.Terminated) + { + // If the orchestration planned to continue-as-new but termination is requested, we transition to the terminated state. + // This is because termination should be considered to be forceful. + log += " Discarding previous 'ExecutionCompletedEvent' as termination is forceful."; + ExecutionCompletedEvent = completedEvent; + orchestrationStatus = completedEvent.OrchestrationStatus; + } + else + { + // otherwise, we ignore the new event. + log += " Discarding new 'ExecutionCompletedEvent'."; + } + + LogHelper?.OrchestrationDebugTrace(this.OrchestrationInstance?.InstanceId ?? "", this.OrchestrationInstance?.ExecutionId ?? "", log); } - - ExecutionCompletedEvent = completedEvent; - orchestrationStatus = completedEvent.OrchestrationStatus; } else if (historyEvent is ExecutionSuspendedEvent) { diff --git a/src/DurableTask.Core/Query/OrchestrationQuery.cs b/src/DurableTask.Core/Query/OrchestrationQuery.cs index 2932d11da..0fc4140f6 100644 --- a/src/DurableTask.Core/Query/OrchestrationQuery.cs +++ b/src/DurableTask.Core/Query/OrchestrationQuery.cs @@ -69,5 +69,11 @@ public OrchestrationQuery() { } /// Determines whether the query will include the input of the orchestration. /// public bool FetchInputsAndOutputs { get; set; } = true; + + /// + /// Whether to exclude entities from the query results. This defaults to false for compatibility with older SDKs, + /// but is set to true by the newer SDKs. + /// + public bool ExcludeEntities { get; set; } = false; } } \ No newline at end of file diff --git a/src/DurableTask.Core/ScheduleProxy.cs b/src/DurableTask.Core/ScheduleProxy.cs index 7f21d2037..4922d2cf1 100644 --- a/src/DurableTask.Core/ScheduleProxy.cs +++ b/src/DurableTask.Core/ScheduleProxy.cs @@ -21,6 +21,10 @@ namespace DurableTask.Core using Castle.DynamicProxy; using DurableTask.Core.Common; + /// + /// This is deprecated and exists only for back-compatibility. + /// See , which adds support for C# interface features such as inheritance, generics, and method overloading. + /// internal class ScheduleProxy : IInterceptor { private readonly OrchestrationContext context; @@ -64,7 +68,7 @@ public void Intercept(IInvocation invocation) arguments.Add(new Utils.TypeMetadata { AssemblyName = typeArg.Assembly.FullName!, FullyQualifiedTypeName = typeArg.FullName }); } - string normalizedMethodName = NameVersionHelper.GetDefaultName(invocation.Method, this.useFullyQualifiedMethodNames); + string normalizedMethodName = this.NormalizeMethodName(invocation.Method); if (returnType == typeof(Task)) { @@ -93,5 +97,10 @@ public void Intercept(IInvocation invocation) return; } + + protected virtual string NormalizeMethodName(MethodInfo method) + { + return NameVersionHelper.GetDefaultName(method, this.useFullyQualifiedMethodNames); + } } } \ No newline at end of file diff --git a/src/DurableTask.Core/ScheduleProxyV2.cs b/src/DurableTask.Core/ScheduleProxyV2.cs new file mode 100644 index 000000000..22e078dc6 --- /dev/null +++ b/src/DurableTask.Core/ScheduleProxyV2.cs @@ -0,0 +1,35 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- + +namespace DurableTask.Core +{ + using System.Reflection; + using Castle.DynamicProxy; + + internal class ScheduleProxyV2 : ScheduleProxy, IInterceptor + { + private readonly string declaringTypeFullName; + + public ScheduleProxyV2(OrchestrationContext context, string declaringTypeFullName) + : base(context) + { + this.declaringTypeFullName = declaringTypeFullName; + } + + protected override string NormalizeMethodName(MethodInfo method) + { + // uses declaring type defined externally because MethodInfo members, such as Method.DeclaringType, could return the base type that the method inherits from + return string.IsNullOrEmpty(this.declaringTypeFullName) ? method.Name : NameVersionHelper.GetFullyQualifiedMethodName(this.declaringTypeFullName, method); + } + } +} diff --git a/src/DurableTask.Core/TaskActivity.cs b/src/DurableTask.Core/TaskActivity.cs index 5cf79c09d..b05f020eb 100644 --- a/src/DurableTask.Core/TaskActivity.cs +++ b/src/DurableTask.Core/TaskActivity.cs @@ -136,7 +136,7 @@ public override async Task RunAsync(TaskContext context, string input) { string details = null; FailureDetails failureDetails = null; - if (context.ErrorPropagationMode == ErrorPropagationMode.SerializeExceptions) + if (context != null && context.ErrorPropagationMode == ErrorPropagationMode.SerializeExceptions) { details = Utils.SerializeCause(e, DataConverter); } diff --git a/src/DurableTask.Core/TaskEntityDispatcher.cs b/src/DurableTask.Core/TaskEntityDispatcher.cs new file mode 100644 index 000000000..8e6dd7d8d --- /dev/null +++ b/src/DurableTask.Core/TaskEntityDispatcher.cs @@ -0,0 +1,884 @@ +// ---------------------------------------------------------------------------------- +// Copyright Microsoft Corporation +// Licensed under the Apache License, Version 2.0 (the "License"); +// you may not use this file except in compliance with the License. +// You may obtain a copy of the License at +// http://www.apache.org/licenses/LICENSE-2.0 +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. +// ---------------------------------------------------------------------------------- +namespace DurableTask.Core +{ + using System; + using System.Collections.Generic; + using System.Diagnostics; + using System.Threading; + using System.Threading.Tasks; + using DurableTask.Core.Common; + using DurableTask.Core.Entities; + using DurableTask.Core.Entities.EventFormat; + using DurableTask.Core.Entities.OperationFormat; + using DurableTask.Core.Exceptions; + using DurableTask.Core.History; + using DurableTask.Core.Logging; + using DurableTask.Core.Middleware; + using DurableTask.Core.Tracing; + using Newtonsoft.Json; + + /// + /// Dispatcher for orchestrations and entities to handle processing and renewing, completion of orchestration events. + /// + public class TaskEntityDispatcher + { + readonly INameVersionObjectManager objectManager; + readonly IOrchestrationService orchestrationService; + readonly IEntityOrchestrationService entityOrchestrationService; + readonly WorkItemDispatcher dispatcher; + readonly DispatchMiddlewarePipeline dispatchPipeline; + readonly EntityBackendProperties entityBackendProperties; + readonly LogHelper logHelper; + readonly ErrorPropagationMode errorPropagationMode; + readonly TaskOrchestrationDispatcher.NonBlockingCountdownLock concurrentSessionLock; + + internal TaskEntityDispatcher( + IOrchestrationService orchestrationService, + INameVersionObjectManager entityObjectManager, + DispatchMiddlewarePipeline entityDispatchPipeline, + LogHelper logHelper, + ErrorPropagationMode errorPropagationMode) + { + this.objectManager = entityObjectManager ?? throw new ArgumentNullException(nameof(entityObjectManager)); + this.orchestrationService = orchestrationService ?? throw new ArgumentNullException(nameof(orchestrationService)); + this.dispatchPipeline = entityDispatchPipeline ?? throw new ArgumentNullException(nameof(entityDispatchPipeline)); + this.logHelper = logHelper ?? throw new ArgumentNullException(nameof(logHelper)); + this.errorPropagationMode = errorPropagationMode; + this.entityOrchestrationService = (orchestrationService as IEntityOrchestrationService)!; + this.entityBackendProperties = entityOrchestrationService.EntityBackendProperties; + + this.dispatcher = new WorkItemDispatcher( + "TaskEntityDispatcher", + item => item == null ? string.Empty : item.InstanceId, + this.OnFetchWorkItemAsync, + this.OnProcessWorkItemSessionAsync) + { + GetDelayInSecondsAfterOnFetchException = orchestrationService.GetDelayInSecondsAfterOnFetchException, + GetDelayInSecondsAfterOnProcessException = orchestrationService.GetDelayInSecondsAfterOnProcessException, + SafeReleaseWorkItem = orchestrationService.ReleaseTaskOrchestrationWorkItemAsync, + AbortWorkItem = orchestrationService.AbandonTaskOrchestrationWorkItemAsync, + DispatcherCount = orchestrationService.TaskOrchestrationDispatcherCount, + MaxConcurrentWorkItems = this.entityBackendProperties.MaxConcurrentTaskEntityWorkItems, + LogHelper = logHelper, + }; + + // To avoid starvation, we only allow half of all concurrently executing entities to + // leverage extended sessions. + var maxConcurrentSessions = (int)Math.Ceiling(this.dispatcher.MaxConcurrentWorkItems / 2.0); + this.concurrentSessionLock = new TaskOrchestrationDispatcher.NonBlockingCountdownLock(maxConcurrentSessions); + } + + /// + /// The entity options configured, or null if the backend does not support entities. + /// + public EntityBackendProperties EntityBackendProperties => this.entityBackendProperties; + + /// + /// Starts the dispatcher to start getting and processing entity message batches + /// + public async Task StartAsync() + { + await this.dispatcher.StartAsync(); + } + + /// + /// Stops the dispatcher to stop getting and processing entity message batches + /// + /// Flag indicating whether to stop gracefully or immediately + public async Task StopAsync(bool forced) + { + await this.dispatcher.StopAsync(forced); + } + + /// + /// Method to get the next work item to process within supplied timeout + /// + /// The max timeout to wait + /// A cancellation token used to cancel a fetch operation. + /// A new TaskOrchestrationWorkItem + protected Task OnFetchWorkItemAsync(TimeSpan receiveTimeout, CancellationToken cancellationToken) + { + return this.entityOrchestrationService.LockNextEntityWorkItemAsync(receiveTimeout, cancellationToken); + } + + async Task OnProcessWorkItemSessionAsync(TaskOrchestrationWorkItem workItem) + { + try + { + if (workItem.Session == null) + { + // Legacy behavior + await this.OnProcessWorkItemAsync(workItem); + return; + } + + var isExtendedSession = false; + + var processCount = 0; + try + { + while (true) + { + // While the work item contains messages that need to be processed, execute them. + if (workItem.NewMessages?.Count > 0) + { + bool isCompletedOrInterrupted = await this.OnProcessWorkItemAsync(workItem); + if (isCompletedOrInterrupted) + { + break; + } + + processCount++; + } + + // Fetches beyond the first require getting an extended session lock, used to prevent starvation. + if (processCount > 0 && !isExtendedSession) + { + isExtendedSession = this.concurrentSessionLock.Acquire(); + if (!isExtendedSession) + { + break; + } + } + + Stopwatch timer = Stopwatch.StartNew(); + + // Wait for new messages to arrive for the session. This call is expected to block (asynchronously) + // until either new messages are available or until a provider-specific timeout has expired. + workItem.NewMessages = await workItem.Session.FetchNewOrchestrationMessagesAsync(workItem); + if (workItem.NewMessages == null) + { + break; + } + + workItem.OrchestrationRuntimeState.NewEvents.Clear(); + } + } + finally + { + if (isExtendedSession) + { + this.concurrentSessionLock.Release(); + } + } + } + catch (SessionAbortedException e) + { + // Either the orchestration or the orchestration service explicitly abandoned the session. + OrchestrationInstance instance = workItem.OrchestrationRuntimeState?.OrchestrationInstance ?? new OrchestrationInstance { InstanceId = workItem.InstanceId }; + this.logHelper.OrchestrationAborted(instance, e.Message); + await this.orchestrationService.AbandonTaskOrchestrationWorkItemAsync(workItem); + } + } + + class WorkItemEffects + { + public List ActivityMessages; + public List TimerMessages; + public List InstanceMessages; + public int taskIdCounter; + public string InstanceId; + public OrchestrationRuntimeState RuntimeState; + } + + /// + /// Method to process a new work item + /// + /// The work item to process + protected async Task OnProcessWorkItemAsync(TaskOrchestrationWorkItem workItem) + { + OrchestrationRuntimeState originalOrchestrationRuntimeState = workItem.OrchestrationRuntimeState; + + OrchestrationRuntimeState runtimeState = workItem.OrchestrationRuntimeState; + runtimeState.AddEvent(new OrchestratorStartedEvent(-1)); + + Task renewTask = null; + using var renewCancellationTokenSource = new CancellationTokenSource(); + if (workItem.LockedUntilUtc < DateTime.MaxValue) + { + // start a task to run RenewUntil + renewTask = Task.Factory.StartNew( + () => TaskOrchestrationDispatcher.RenewUntil(workItem, this.orchestrationService, this.logHelper, nameof(TaskEntityDispatcher), renewCancellationTokenSource.Token), + renewCancellationTokenSource.Token); + } + + WorkItemEffects effects = new WorkItemEffects() + { + ActivityMessages = new List(), + TimerMessages = new List(), + InstanceMessages = new List(), + taskIdCounter = 0, + InstanceId = workItem.InstanceId, + RuntimeState = runtimeState, + }; + + try + { + // Assumes that: if the batch contains a new "ExecutionStarted" event, it is the first message in the batch. + if (!TaskOrchestrationDispatcher.ReconcileMessagesWithState(workItem, nameof(TaskEntityDispatcher), this.errorPropagationMode, this.logHelper)) + { + // TODO : mark an orchestration as faulted if there is data corruption + this.logHelper.DroppingOrchestrationWorkItem(workItem, "Received work-item for an invalid orchestration"); + } + else + { + + // we start with processing all the requests and figuring out which ones to execute now + // results can depend on whether the entity is locked, what the maximum batch size is, + // and whether the messages arrived out of order + + this.DetermineWork(workItem.OrchestrationRuntimeState, + out SchedulerState schedulerState, + out Work workToDoNow); + + if (workToDoNow.OperationCount > 0) + { + // execute the user-defined operations on this entity, via the middleware + var result = await this.ExecuteViaMiddlewareAsync(workToDoNow, runtimeState.OrchestrationInstance, schedulerState.EntityState); + var operationResults = result.Results!; + + // if we encountered an error, record it as the result of the operations + // so that callers are notified that the operation did not succeed. + if (result.FailureDetails != null) + { + OperationResult errorResult = new OperationResult() + { + // for older SDKs only + Result = result.FailureDetails.ErrorMessage, + ErrorMessage = "entity dispatch failed", + + // for newer SDKs only + FailureDetails = result.FailureDetails, + }; + + for (int i = operationResults.Count; i < workToDoNow.OperationCount; i++) + { + operationResults.Add(errorResult); + } + } + + // go through all results + // for each operation that is not a signal, send a result message back to the calling orchestrator + for (int i = 0; i < result.Results!.Count; i++) + { + var req = workToDoNow.Operations[i]; + if (!req.IsSignal) + { + this.SendResultMessage(effects, req, result.Results[i]); + } + } + + if (result.Results.Count < workToDoNow.OperationCount) + { + // some requests were not processed (e.g. due to shutdown or timeout) + // in this case we just defer the work so it can be retried + var deferred = workToDoNow.RemoveDeferredWork(result.Results.Count); + schedulerState.PutBack(deferred); + workToDoNow.ToBeContinued(schedulerState); + } + + // update the entity state based on the result + schedulerState.EntityState = result.EntityState; + + // perform the actions + foreach (var action in result.Actions!) + { + switch (action) + { + case (SendSignalOperationAction sendSignalAction): + this.SendSignalMessage(effects, schedulerState, sendSignalAction); + break; + case (StartNewOrchestrationOperationAction startAction): + this.ProcessSendStartMessage(effects, runtimeState, startAction); + break; + } + } + } + + // process the lock request, if any + if (workToDoNow.LockRequest != null) + { + this.ProcessLockRequest(effects, schedulerState, workToDoNow.LockRequest); + } + + if (workToDoNow.ToBeRescheduled != null) + { + foreach (var request in workToDoNow.ToBeRescheduled) + { + // Reschedule all signals that were received before their time + this.SendScheduledSelfMessage(effects, request); + } + } + + if (workToDoNow.SuspendAndContinue) + { + this.SendContinueSelfMessage(effects); + } + + // this batch is complete. Since this is an entity, we now + // (always) start a new execution, as in continue-as-new + + var serializedSchedulerState = this.SerializeSchedulerStateForNextExecution(schedulerState); + var nextExecutionStartedEvent = new ExecutionStartedEvent(-1, serializedSchedulerState) + { + OrchestrationInstance = new OrchestrationInstance + { + InstanceId = workItem.InstanceId, + ExecutionId = Guid.NewGuid().ToString("N") + }, + Tags = runtimeState.Tags, + ParentInstance = runtimeState.ParentInstance, + Name = runtimeState.Name, + Version = runtimeState.Version + }; + var entityStatus = new EntityStatus() + { + EntityExists = schedulerState.EntityExists, + BacklogQueueSize = schedulerState.Queue?.Count ?? 0, + LockedBy = schedulerState.LockedBy, + }; + var serializedEntityStatus = JsonConvert.SerializeObject(entityStatus, Serializer.InternalSerializerSettings); + + // create the new runtime state for the next execution + runtimeState = new OrchestrationRuntimeState(); + runtimeState.Status = serializedEntityStatus; + runtimeState.AddEvent(new OrchestratorStartedEvent(-1)); + runtimeState.AddEvent(nextExecutionStartedEvent); + runtimeState.AddEvent(new OrchestratorCompletedEvent(-1)); + } + } + finally + { + if (renewTask != null) + { + try + { + renewCancellationTokenSource.Cancel(); + await renewTask; + } + catch (ObjectDisposedException) + { + // ignore + } + catch (OperationCanceledException) + { + // ignore + } + } + } + + OrchestrationState instanceState = (runtimeState.ExecutionStartedEvent != null) ? + instanceState = Utils.BuildOrchestrationState(runtimeState) : null; + + if (workItem.RestoreOriginalRuntimeStateDuringCompletion) + { + // some backends expect the original runtime state object + workItem.OrchestrationRuntimeState = originalOrchestrationRuntimeState; + } + else + { + workItem.OrchestrationRuntimeState = runtimeState; + } + + await this.orchestrationService.CompleteTaskOrchestrationWorkItemAsync( + workItem, + runtimeState, + effects.ActivityMessages, + effects.InstanceMessages, + effects.TimerMessages, + null, + instanceState); + + if (workItem.RestoreOriginalRuntimeStateDuringCompletion) + { + workItem.OrchestrationRuntimeState = runtimeState; + } + + return true; + } + + void ProcessLockRequest(WorkItemEffects effects, SchedulerState schedulerState, RequestMessage request) + { + this.logHelper.EntityLockAcquired(effects.InstanceId, request); + + // mark the entity state as locked + schedulerState.LockedBy = request.ParentInstanceId; + + request.Position++; + + if (request.Position < request.LockSet.Length) + { + // send lock request to next entity in the lock set + var target = new OrchestrationInstance() { InstanceId = request.LockSet[request.Position].ToString() }; + this.SendLockRequestMessage(effects, schedulerState, target, request); + } + else + { + // send lock acquisition completed response back to originating orchestration instance + var target = new OrchestrationInstance() { InstanceId = request.ParentInstanceId, ExecutionId = request.ParentExecutionId }; + this.SendLockResponseMessage(effects, target, request.Id); + } + } + + string SerializeSchedulerStateForNextExecution(SchedulerState schedulerState) + { + if (this.entityBackendProperties.SupportsImplicitEntityDeletion && schedulerState.IsEmpty && !schedulerState.Suspended) + { + // this entity scheduler is idle and the entity is deleted, so the instance and history can be removed from storage + // we convey this to the durability provider by issuing a continue-as-new with null input + return null; + } + else + { + // we persist the state of the entity scheduler and entity + return JsonConvert.SerializeObject(schedulerState, typeof(SchedulerState), Serializer.InternalSerializerSettings); + } + } + + #region Preprocess to determine work + + void DetermineWork(OrchestrationRuntimeState runtimeState, out SchedulerState schedulerState, out Work batch) + { + string instanceId = runtimeState.OrchestrationInstance.InstanceId; + schedulerState = new SchedulerState(); + batch = new Work(); + + Queue lockHolderMessages = null; + + foreach (HistoryEvent e in runtimeState.Events) + { + switch (e.EventType) + { + case EventType.ExecutionStarted: + + + if (runtimeState.Input != null) + { + try + { + // restore the scheduler state from the input + JsonConvert.PopulateObject(runtimeState.Input, schedulerState, Serializer.InternalSerializerSettings); + } + catch (Exception exception) + { + throw new EntitySchedulerException("Failed to deserialize entity scheduler state - may be corrupted or wrong version.", exception); + } + } + break; + + case EventType.EventRaised: + EventRaisedEvent eventRaisedEvent = (EventRaisedEvent)e; + + if (EntityMessageEventNames.IsRequestMessage(eventRaisedEvent.Name)) + { + // we are receiving an operation request or a lock request + var requestMessage = new RequestMessage(); + + try + { + JsonConvert.PopulateObject(eventRaisedEvent.Input, requestMessage, Serializer.InternalSerializerSettings); + } + catch (Exception exception) + { + throw new EntitySchedulerException("Failed to deserialize incoming request message - may be corrupted or wrong version.", exception); + } + + IEnumerable deliverNow; + + if (requestMessage.ScheduledTime.HasValue) + { + if ((requestMessage.ScheduledTime.Value - DateTime.UtcNow) > TimeSpan.FromMilliseconds(100)) + { + // message was delivered too early. This can happen e.g. if the orchestration service has limits on the delay times for messages. + // We handle this by rescheduling the message instead of processing it. + deliverNow = Array.Empty(); + batch.AddMessageToBeRescheduled(requestMessage); + } + else + { + // the message is scheduled to be delivered immediately. + // There are no FIFO guarantees for scheduled messages, so we skip the message sorter. + deliverNow = new RequestMessage[] { requestMessage }; + } + } + else + { + // run this through the message sorter to help with reordering and duplicate filtering + deliverNow = schedulerState.MessageSorter.ReceiveInOrder(requestMessage, this.entityBackendProperties.EntityMessageReorderWindow); + } + + foreach (var message in deliverNow) + { + if (schedulerState.LockedBy != null && schedulerState.LockedBy == message.ParentInstanceId) + { + if (lockHolderMessages == null) + { + lockHolderMessages = new Queue(); + } + + lockHolderMessages.Enqueue(message); + } + else + { + schedulerState.Enqueue(message); + } + } + } + else if (EntityMessageEventNames.IsReleaseMessage(eventRaisedEvent.Name)) + { + // we are receiving a lock release + var message = new ReleaseMessage(); + try + { + // restore the scheduler state from the input + JsonConvert.PopulateObject(eventRaisedEvent.Input, message, Serializer.InternalSerializerSettings); + } + catch (Exception exception) + { + throw new EntitySchedulerException("Failed to deserialize lock release message - may be corrupted or wrong version.", exception); + } + + if (schedulerState.LockedBy == message.ParentInstanceId) + { + this.logHelper.EntityLockReleased(instanceId, message); + schedulerState.LockedBy = null; + } + } + else + { + // this is a continue message. + // Resumes processing of previously queued operations, if any. + schedulerState.Suspended = false; + } + + break; + } + } + + // lock holder messages go to the front of the queue + if (lockHolderMessages != null) + { + schedulerState.PutBack(lockHolderMessages); + } + + if (!schedulerState.Suspended) + { + // 2. We add as many requests from the queue to the batch as possible, + // stopping at lock requests or when the maximum batch size is reached + while (schedulerState.MayDequeue()) + { + if (batch.OperationCount == this.entityBackendProperties.MaxEntityOperationBatchSize) + { + // we have reached the maximum batch size already + // insert a delay after this batch to ensure write back + batch.ToBeContinued(schedulerState); + break; + } + + var request = schedulerState.Dequeue(); + + if (request.IsLockRequest) + { + batch.AddLockRequest(request); + break; + } + else + { + batch.AddOperation(request); + } + } + } + } + + class Work + { + List operationBatch; // a (possibly empty) sequence of operations to be executed on the entity + RequestMessage lockRequest = null; // zero or one lock request to be executed after all the operations + List toBeRescheduled; // a (possibly empty) list of timed messages that were delivered too early and should be rescheduled + bool suspendAndContinue; // a flag telling as to send ourselves a continue signal + + public int OperationCount => this.operationBatch?.Count ?? 0; + public IReadOnlyList Operations => this.operationBatch; + public IReadOnlyList ToBeRescheduled => this.toBeRescheduled; + public RequestMessage LockRequest => this.lockRequest; + public bool SuspendAndContinue => this.suspendAndContinue; + + public void AddOperation(RequestMessage operationMessage) + { + if (this.operationBatch == null) + { + this.operationBatch = new List(); + } + this.operationBatch.Add(operationMessage); + } + + public void AddLockRequest(RequestMessage lockRequest) + { + Debug.Assert(this.lockRequest == null); + this.lockRequest = lockRequest; + } + + public void AddMessageToBeRescheduled(RequestMessage requestMessage) + { + if (this.toBeRescheduled == null) + { + this.toBeRescheduled = new List(); + } + this.toBeRescheduled.Add(requestMessage); + } + + public void ToBeContinued(SchedulerState schedulerState) + { + if (!schedulerState.Suspended) + { + this.suspendAndContinue = true; + } + } + + public List GetOperationRequests() + { + var operations = new List(this.operationBatch.Count); + for (int i = 0; i < this.operationBatch.Count; i++) + { + var request = this.operationBatch[i]; + operations.Add(new OperationRequest() + { + Operation = request.Operation, + Id = request.Id, + Input = request.Input, + }); + } + return operations; + } + + public Queue RemoveDeferredWork(int index) + { + var deferred = new Queue(); + for (int i = index; i < this.operationBatch.Count; i++) + { + deferred.Enqueue(this.operationBatch[i]); + } + this.operationBatch.RemoveRange(index, this.operationBatch.Count - index); + if (this.lockRequest != null) + { + deferred.Enqueue(this.lockRequest); + this.lockRequest = null; + } + return deferred; + } + } + + #endregion + + #region Send Messages + + void SendResultMessage(WorkItemEffects effects, RequestMessage request, OperationResult result) + { + var destination = new OrchestrationInstance() + { + InstanceId = request.ParentInstanceId, + ExecutionId = request.ParentExecutionId, + }; + var responseMessage = new ResponseMessage() + { + Result = result.Result, + ErrorMessage = result.ErrorMessage, + FailureDetails = result.FailureDetails, + }; + this.ProcessSendEventMessage(effects, destination, EntityMessageEventNames.ResponseMessageEventName(request.Id), responseMessage); + } + + void SendSignalMessage(WorkItemEffects effects, SchedulerState schedulerState, SendSignalOperationAction action) + { + OrchestrationInstance destination = new OrchestrationInstance() + { + InstanceId = action.InstanceId + }; + RequestMessage message = new RequestMessage() + { + ParentInstanceId = effects.InstanceId, + ParentExecutionId = null, // for entities, message sorter persists across executions + Id = Guid.NewGuid(), + IsSignal = true, + Operation = action.Name, + Input = action.Input, + ScheduledTime = action.ScheduledTime, + }; + string eventName; + if (action.ScheduledTime.HasValue) + { + DateTime original = action.ScheduledTime.Value; + DateTime capped = this.entityBackendProperties.GetCappedScheduledTime(DateTime.UtcNow, original); + eventName = EntityMessageEventNames.ScheduledRequestMessageEventName(capped); + } + else + { + eventName = EntityMessageEventNames.RequestMessageEventName; + schedulerState.MessageSorter.LabelOutgoingMessage(message, action.InstanceId, DateTime.UtcNow, this.entityBackendProperties.EntityMessageReorderWindow); + } + this.ProcessSendEventMessage(effects, destination, eventName, message); + } + + void SendLockRequestMessage(WorkItemEffects effects, SchedulerState schedulerState, OrchestrationInstance target, RequestMessage message) + { + schedulerState.MessageSorter.LabelOutgoingMessage(message, target.InstanceId, DateTime.UtcNow, this.entityBackendProperties.EntityMessageReorderWindow); + this.ProcessSendEventMessage(effects, target, EntityMessageEventNames.RequestMessageEventName, message); + } + + void SendLockResponseMessage(WorkItemEffects effects, OrchestrationInstance target, Guid requestId) + { + var message = new ResponseMessage() + { + // content is ignored by receiver but helps with tracing + Result = ResponseMessage.LockAcquisitionCompletion, + }; + this.ProcessSendEventMessage(effects, target, EntityMessageEventNames.ResponseMessageEventName(requestId), message); + } + + void SendScheduledSelfMessage(WorkItemEffects effects, RequestMessage request) + { + var self = new OrchestrationInstance() + { + InstanceId = effects.InstanceId, + }; + this.ProcessSendEventMessage(effects, self, EntityMessageEventNames.ScheduledRequestMessageEventName(request.ScheduledTime.Value), request); + } + + void SendContinueSelfMessage(WorkItemEffects effects) + { + var self = new OrchestrationInstance() + { + InstanceId = effects.InstanceId, + }; + this.ProcessSendEventMessage(effects, self, EntityMessageEventNames.ContinueMessageEventName, null); + } + + void ProcessSendEventMessage(WorkItemEffects effects, OrchestrationInstance destination, string eventName, object eventContent) + { + string serializedContent = null; + if (eventContent != null) + { + serializedContent = JsonConvert.SerializeObject(eventContent, Serializer.InternalSerializerSettings); + } + + var eventSentEvent = new EventSentEvent(effects.taskIdCounter++) + { + InstanceId = destination.InstanceId, + Name = eventName, + Input = serializedContent, + }; + this.logHelper.RaisingEvent(effects.RuntimeState.OrchestrationInstance, eventSentEvent); + + effects.InstanceMessages.Add(new TaskMessage + { + OrchestrationInstance = destination, + Event = new EventRaisedEvent(-1, serializedContent) + { + Name = eventName, + Input = serializedContent, + }, + }); + } + + void ProcessSendStartMessage(WorkItemEffects effects, OrchestrationRuntimeState runtimeState, StartNewOrchestrationOperationAction action) + { + OrchestrationInstance destination = new OrchestrationInstance() + { + InstanceId = action.InstanceId, + ExecutionId = Guid.NewGuid().ToString("N"), + }; + var executionStartedEvent = new ExecutionStartedEvent(-1, action.Input) + { + Tags = OrchestrationTags.MergeTags( + runtimeState.Tags, + new Dictionary() { { OrchestrationTags.FireAndForget, "" } }), + OrchestrationInstance = destination, + ScheduledStartTime = action.ScheduledStartTime, + ParentInstance = new ParentInstance + { + OrchestrationInstance = runtimeState.OrchestrationInstance, + Name = runtimeState.Name, + Version = runtimeState.Version, + TaskScheduleId = effects.taskIdCounter++, + }, + Name = action.Name, + Version = action.Version, + }; + this.logHelper.SchedulingOrchestration(executionStartedEvent); + + effects.InstanceMessages.Add(new TaskMessage + { + OrchestrationInstance = destination, + Event = executionStartedEvent, + }); + } + + #endregion + + async Task ExecuteViaMiddlewareAsync(Work workToDoNow, OrchestrationInstance instance, string serializedEntityState) + { + // the request object that will be passed to the worker + var request = new EntityBatchRequest() + { + InstanceId = instance.InstanceId, + EntityState = serializedEntityState, + Operations = workToDoNow.GetOperationRequests(), + }; + + this.logHelper.EntityBatchExecuting(request); + + var entityId = EntityId.FromString(instance.InstanceId); + string entityName = entityId.Name; + + // Get the TaskEntity implementation. If it's not found, it either means that the developer never + // registered it (which is an error, and we'll throw for this further down) or it could be that some custom + // middleware (e.g. out-of-process execution middleware) is intended to implement the entity logic. + TaskEntity taskEntity = this.objectManager.GetObject(entityName, version: null); + + var dispatchContext = new DispatchMiddlewareContext(); + dispatchContext.SetProperty(request); + + await this.dispatchPipeline.RunAsync(dispatchContext, async _ => + { + // Check to see if the custom middleware intercepted and substituted the orchestration execution + // with its own execution behavior, providing us with the end results. If so, we can terminate + // the dispatch pipeline here. + var resultFromMiddleware = dispatchContext.GetProperty(); + if (resultFromMiddleware != null) + { + return; + } + + if (taskEntity == null) + { + throw TraceHelper.TraceExceptionInstance( + TraceEventType.Error, + "TaskOrchestrationDispatcher-EntityTypeMissing", + instance, + new TypeMissingException($"Entity not found: {entityName}")); + } + + var result = await taskEntity.ExecuteOperationBatchAsync(request); + + dispatchContext.SetProperty(result); + }); + + var result = dispatchContext.GetProperty(); + + this.logHelper.EntityBatchExecuted(request, result); + + return result; + } + } +} \ No newline at end of file diff --git a/src/DurableTask.Core/TaskHubClient.cs b/src/DurableTask.Core/TaskHubClient.cs index af03ac1a5..fb045eea0 100644 --- a/src/DurableTask.Core/TaskHubClient.cs +++ b/src/DurableTask.Core/TaskHubClient.cs @@ -33,6 +33,9 @@ public sealed class TaskHubClient readonly DataConverter defaultConverter; readonly LogHelper logHelper; + internal LogHelper LogHelper => this.logHelper; + internal DataConverter DefaultConverter => this.defaultConverter; + /// /// The orchestration service client for this task hub client /// @@ -71,7 +74,7 @@ public TaskHubClient(IOrchestrationServiceClient serviceClient, DataConverter da } /// - /// Create a new orchestration of the specified type with the specified instance id, scheduled to start at an specific time + /// Create a new orchestration of the specified type with the specified instance id, scheduled to start at the specified time /// /// Type that derives from TaskOrchestration /// Input parameter to the specified TaskOrchestration @@ -95,7 +98,7 @@ public Task CreateScheduledOrchestrationInstanceAsync( } /// - /// Create a new orchestration of the specified type with the specified instance id, scheduled to start at an specific time + /// Create a new orchestration of the specified type with the specified instance id, scheduled to start at the specified time /// /// Type that derives from TaskOrchestration /// Instance id for the orchestration to be created, must be unique across the Task Hub @@ -120,6 +123,29 @@ public Task CreateScheduledOrchestrationInstanceAsync( startAt: startAt); } + /// + /// Create a new orchestration of the specified name and version with the specified instance id, scheduled to start at the specified time. + /// + /// Name of the orchestration as specified by the ObjectCreator + /// Name of the orchestration as specified by the ObjectCreator + /// Instance id for the orchestration to be created, must be unique across the Task Hub + /// Input parameter to the specified TaskOrchestration + /// Orchestration start time + /// OrchestrationInstance that represents the orchestration that was created + public Task CreateScheduledOrchestrationInstanceAsync(string name, string version, string instanceId, object input, DateTime startAt) + { + return InternalCreateOrchestrationInstanceWithRaisedEventAsync( + name, + version, + instanceId, + input, + null, + null, + null, + null, + startAt: startAt); + } + /// /// Create a new orchestration of the specified type with an automatically generated instance id /// diff --git a/src/DurableTask.Core/TaskHubWorker.cs b/src/DurableTask.Core/TaskHubWorker.cs index 1b93bd62e..24271dc78 100644 --- a/src/DurableTask.Core/TaskHubWorker.cs +++ b/src/DurableTask.Core/TaskHubWorker.cs @@ -21,6 +21,7 @@ namespace DurableTask.Core using System.Runtime.CompilerServices; using System.Threading; using System.Threading.Tasks; + using DurableTask.Core.Entities; using DurableTask.Core.Exceptions; using DurableTask.Core.Logging; using DurableTask.Core.Middleware; @@ -34,10 +35,13 @@ public sealed class TaskHubWorker : IDisposable { readonly INameVersionObjectManager activityManager; readonly INameVersionObjectManager orchestrationManager; + readonly INameVersionObjectManager entityManager; readonly DispatchMiddlewarePipeline orchestrationDispatchPipeline = new DispatchMiddlewarePipeline(); + readonly DispatchMiddlewarePipeline entityDispatchPipeline = new DispatchMiddlewarePipeline(); readonly DispatchMiddlewarePipeline activityDispatchPipeline = new DispatchMiddlewarePipeline(); + readonly bool dispatchEntitiesSeparately; readonly SemaphoreSlim slimLock = new SemaphoreSlim(1, 1); readonly LogHelper logHelper; @@ -51,6 +55,7 @@ public sealed class TaskHubWorker : IDisposable TaskActivityDispatcher activityDispatcher; TaskOrchestrationDispatcher orchestrationDispatcher; + TaskEntityDispatcher entityDispatcher; /// /// Create a new TaskHubWorker with given OrchestrationService @@ -60,7 +65,8 @@ public TaskHubWorker(IOrchestrationService orchestrationService) : this( orchestrationService, new NameVersionObjectManager(), - new NameVersionObjectManager()) + new NameVersionObjectManager(), + new NameVersionObjectManager()) { } @@ -75,6 +81,7 @@ public TaskHubWorker(IOrchestrationService orchestrationService, ILoggerFactory orchestrationService, new NameVersionObjectManager(), new NameVersionObjectManager(), + new NameVersionObjectManager(), loggerFactory) { } @@ -93,11 +100,11 @@ public TaskHubWorker( orchestrationService, orchestrationObjectManager, activityObjectManager, + new NameVersionObjectManager(), loggerFactory: null) { } - /// /// Create a new with given and name version managers /// @@ -110,11 +117,36 @@ public TaskHubWorker( INameVersionObjectManager orchestrationObjectManager, INameVersionObjectManager activityObjectManager, ILoggerFactory loggerFactory = null) + : this( + orchestrationService, + orchestrationObjectManager, + activityObjectManager, + new NameVersionObjectManager(), + loggerFactory) + { + } + + /// + /// Create a new TaskHubWorker with given OrchestrationService and name version managers + /// + /// Reference the orchestration service implementation + /// NameVersionObjectManager for Orchestrations + /// NameVersionObjectManager for Activities + /// The NameVersionObjectManager for entities. The version is the entity key. + /// The to use for logging + public TaskHubWorker( + IOrchestrationService orchestrationService, + INameVersionObjectManager orchestrationObjectManager, + INameVersionObjectManager activityObjectManager, + INameVersionObjectManager entityObjectManager, + ILoggerFactory loggerFactory = null) { this.orchestrationManager = orchestrationObjectManager ?? throw new ArgumentException("orchestrationObjectManager"); this.activityManager = activityObjectManager ?? throw new ArgumentException("activityObjectManager"); + this.entityManager = entityObjectManager ?? throw new ArgumentException("entityObjectManager"); this.orchestrationService = orchestrationService ?? throw new ArgumentException("orchestrationService"); this.logHelper = new LogHelper(loggerFactory?.CreateLogger("DurableTask.Core")); + this.dispatchEntitiesSeparately = (orchestrationService as IEntityOrchestrationService)?.EntityBackendProperties?.UseSeparateQueueForEntityWorkItems ?? false; } /// @@ -153,6 +185,15 @@ public void AddOrchestrationDispatcherMiddleware(Func + /// Adds a middleware delegate to the entity dispatch pipeline. + /// + /// Delegate to invoke whenever a message is dispatched to an entity. + public void AddEntityDispatcherMiddleware(Func, Task> middleware) + { + this.entityDispatchPipeline.Add(middleware ?? throw new ArgumentNullException(nameof(middleware))); + } + /// /// Adds a middleware delegate to the activity dispatch pipeline. /// @@ -192,10 +233,25 @@ public async Task StartAsync() this.logHelper, this.ErrorPropagationMode); + if (this.dispatchEntitiesSeparately) + { + this.entityDispatcher = new TaskEntityDispatcher( + this.orchestrationService, + this.entityManager, + this.entityDispatchPipeline, + this.logHelper, + this.ErrorPropagationMode); + } + await this.orchestrationService.StartAsync(); await this.orchestrationDispatcher.StartAsync(); await this.activityDispatcher.StartAsync(); + if (this.dispatchEntitiesSeparately) + { + await this.entityDispatcher.StartAsync(); + } + this.logHelper.TaskHubWorkerStarted(sw.Elapsed); this.isStarted = true; } @@ -233,6 +289,7 @@ public async Task StopAsync(bool isForced) { this.orchestrationDispatcher.StopAsync(isForced), this.activityDispatcher.StopAsync(isForced), + this.dispatchEntitiesSeparately ? this.entityDispatcher.StopAsync(isForced) : Task.CompletedTask, }; await Task.WhenAll(dispatcherShutdowns); @@ -282,6 +339,53 @@ public TaskHubWorker AddTaskOrchestrations(params ObjectCreator + /// Loads user defined TaskEntity classes in the TaskHubWorker + /// + /// Types deriving from TaskEntity class + /// + public TaskHubWorker AddTaskEntities(params Type[] taskEntityTypes) + { + if (!this.dispatchEntitiesSeparately) + { + throw new NotSupportedException("The configured backend does not support separate entity dispatch."); + } + + foreach (Type type in taskEntityTypes) + { + ObjectCreator creator = new NameValueObjectCreator( + type.Name, + string.Empty, + type); + + this.entityManager.Add(creator); + } + + return this; + } + + /// + /// Loads user defined TaskEntity classes in the TaskHubWorker + /// + /// + /// User specified ObjectCreators that will + /// create classes deriving TaskEntities with specific names and versions + /// + public TaskHubWorker AddTaskEntities(params ObjectCreator[] taskEntityCreators) + { + if (!this.dispatchEntitiesSeparately) + { + throw new NotSupportedException("The configured backend does not support separate entity dispatch."); + } + + foreach (ObjectCreator creator in taskEntityCreators) + { + this.entityManager.Add(creator); + } + + return this; + } + /// /// Loads user defined TaskActivity objects in the TaskHubWorker /// @@ -335,6 +439,10 @@ public TaskHubWorker AddTaskActivities(params ObjectCreator[] task /// and version set to an empty string. Methods can then be invoked from task orchestrations /// by calling ScheduleTask(name, version) with name as the method name and string.Empty as the version. /// + /// + /// This is deprecated and exists only for back-compatibility. + /// See , which adds support for C# interface features such as inheritance, generics, and method overloading. + /// /// Interface /// Object that implements this interface public TaskHubWorker AddTaskActivitiesFromInterface(T activities) @@ -348,6 +456,10 @@ public TaskHubWorker AddTaskActivitiesFromInterface(T activities) /// and version set to an empty string. Methods can then be invoked from task orchestrations /// by calling ScheduleTask(name, version) with name as the method name and string.Empty as the version. /// + /// + /// This is deprecated and exists only for back-compatibility. + /// See , which adds support for C# interface features such as inheritance, generics, and method overloading. + /// /// Interface /// Object that implements this interface /// @@ -365,6 +477,23 @@ public TaskHubWorker AddTaskActivitiesFromInterface(T activities, bool useFul /// and version set to an empty string. Methods can then be invoked from task orchestrations /// by calling ScheduleTask(name, version) with name as the method name and string.Empty as the version. /// + /// Interface + /// Object that implements this interface + public TaskHubWorker AddTaskActivitiesFromInterfaceV2(object activities) + { + return this.AddTaskActivitiesFromInterfaceV2(typeof(T), activities); + } + + /// + /// Infers and adds every method in the specified interface T on the + /// passed in object as a different TaskActivity with Name set to the method name + /// and version set to an empty string. Methods can then be invoked from task orchestrations + /// by calling ScheduleTask(name, version) with name as the method name and string.Empty as the version. + /// + /// + /// This is deprecated and exists only for back-compatibility. + /// See , which adds support for C# interface features such as inheritance, generics, and method overloading. + /// /// Interface type. /// Object that implements the interface /// @@ -373,16 +502,7 @@ public TaskHubWorker AddTaskActivitiesFromInterface(T activities, bool useFul /// public TaskHubWorker AddTaskActivitiesFromInterface(Type @interface, object activities, bool useFullyQualifiedMethodNames = false) { - if (!@interface.IsInterface) - { - throw new Exception("Contract can only be an interface."); - } - - if (!@interface.IsAssignableFrom(activities.GetType())) - { - throw new ArgumentException($"{activities.GetType().FullName} does not implement {@interface.FullName}", nameof(activities)); - } - + this.ValidateActivitiesInterfaceType(@interface, activities); foreach (MethodInfo methodInfo in @interface.GetMethods()) { TaskActivity taskActivity = new ReflectionBasedTaskActivity(activities, methodInfo); @@ -396,6 +516,29 @@ public TaskHubWorker AddTaskActivitiesFromInterface(Type @interface, object acti return this; } + /// + /// Infers and adds every method in the specified interface T on the + /// passed in object as a different TaskActivity with Name set to the method name + /// and version set to an empty string. Methods can then be invoked from task orchestrations + /// by calling ScheduleTask(name, version) with name as the method name and string.Empty as the version. + /// + /// Interface type. + /// Object that implements the interface + public TaskHubWorker AddTaskActivitiesFromInterfaceV2(Type @interface, object activities) + { + this.ValidateActivitiesInterfaceType(@interface, activities); + var methods = NameVersionHelper.GetAllInterfaceMethods(@interface, (MethodInfo m) => NameVersionHelper.GetFullyQualifiedMethodName(@interface.ToString(), m)); + foreach (MethodInfo methodInfo in methods) + { + TaskActivity taskActivity = new ReflectionBasedTaskActivity(activities, methodInfo); + string name = NameVersionHelper.GetFullyQualifiedMethodName(@interface.ToString(), methodInfo); + ObjectCreator creator = new NameValueObjectCreator(name, NameVersionHelper.GetDefaultVersion(methodInfo), taskActivity); + this.AddTaskActivities(creator); + } + + return this; + } + /// /// Infers and adds every method in the specified interface or class T on the /// passed in object as a different TaskActivity with Name set to the method name @@ -486,5 +629,18 @@ public void Dispose() { ((IDisposable)this.slimLock).Dispose(); } + + private void ValidateActivitiesInterfaceType(Type @interface, object activities) + { + if (!@interface.IsInterface) + { + throw new Exception("Contract can only be an interface."); + } + + if (!@interface.IsAssignableFrom(activities.GetType())) + { + throw new ArgumentException($"type {activities.GetType().FullName} does not implement {@interface.FullName}"); + } + } } } \ No newline at end of file diff --git a/src/DurableTask.Core/TaskOrchestrationContext.cs b/src/DurableTask.Core/TaskOrchestrationContext.cs index d56b18de6..3b2d5a797 100644 --- a/src/DurableTask.Core/TaskOrchestrationContext.cs +++ b/src/DurableTask.Core/TaskOrchestrationContext.cs @@ -17,10 +17,12 @@ namespace DurableTask.Core using System.Collections.Generic; using System.Diagnostics; using System.Globalization; + using System.Linq; using System.Threading; using System.Threading.Tasks; using DurableTask.Core.Command; using DurableTask.Core.Common; + using DurableTask.Core.Entities; using DurableTask.Core.Exceptions; using DurableTask.Core.History; using DurableTask.Core.Serializing; @@ -34,6 +36,7 @@ internal class TaskOrchestrationContext : OrchestrationContext private bool executionCompletedOrTerminated; private int idCounter; private readonly Queue eventsWhileSuspended; + private readonly IDictionary suspendedActionsMap; public bool IsSuspended { get; private set; } @@ -47,6 +50,7 @@ public void AddEventToNextIteration(HistoryEvent he) public TaskOrchestrationContext( OrchestrationInstance orchestrationInstance, TaskScheduler taskScheduler, + TaskOrchestrationEntityParameters entityParameters = null, ErrorPropagationMode errorPropagationMode = ErrorPropagationMode.SerializeExceptions) { Utils.UnusedParameter(taskScheduler); @@ -58,8 +62,10 @@ public TaskOrchestrationContext( this.ErrorDataConverter = JsonDataConverter.Default; OrchestrationInstance = orchestrationInstance; IsReplaying = false; + this.EntityParameters = entityParameters; ErrorPropagationMode = errorPropagationMode; this.eventsWhileSuspended = new Queue(); + this.suspendedActionsMap = new SortedDictionary(); } public IEnumerable OrchestratorActions => this.orchestratorActionsMap.Values; @@ -196,7 +202,8 @@ public override void SendEvent(OrchestrationInstance orchestrationInstance, stri } int id = this.idCounter++; - string serializedEventData = this.MessageDataConverter.SerializeInternal(eventData); + + string serializedEventData = this.MessageDataConverter.SerializeInternal(eventData); var action = new SendEventOrchestratorAction { @@ -416,7 +423,6 @@ public void HandleEventRaisedEvent(EventRaisedEvent eventRaisedEvent, bool skipC } } - public void HandleTaskCompletedEvent(TaskCompletedEvent completedEvent) { int taskId = completedEvent.TaskScheduledId; @@ -497,8 +503,8 @@ public void HandleSubOrchestrationInstanceFailedEvent(SubOrchestrationInstanceFa // When using ErrorPropagationMode.UseFailureDetails we instead use FailureDetails to convey // error information, which doesn't involve any serialization at all. Exception cause = this.ErrorPropagationMode == ErrorPropagationMode.SerializeExceptions ? - Utils.RetrieveCause(failedEvent.Details, this.ErrorDataConverter) : - null; + Utils.RetrieveCause(failedEvent.Details, this.ErrorDataConverter) + : null; var failedException = new SubOrchestrationFailedException(failedEvent.EventId, taskId, info.Name, info.Version, @@ -565,11 +571,35 @@ public void HandleEventWhileSuspended(HistoryEvent historyEvent) public void HandleExecutionSuspendedEvent(ExecutionSuspendedEvent suspendedEvent) { this.IsSuspended = true; + + // When the orchestrator is suspended, a task could potentially be added to the orchestratorActionsMap. + // This could lead to the task being executed repeatedly without completion until the orchestrator is resumed. + // To prevent this scenario, check if orchestratorActionsMap is empty before proceeding. + if (this.orchestratorActionsMap.Any()) + { + // If not, store its contents to a temporary dictionary to allow processing of the task later when orchestrator resumes. + foreach (var pair in this.orchestratorActionsMap) + { + this.suspendedActionsMap.Add(pair.Key, pair.Value); + } + this.orchestratorActionsMap.Clear(); + } } public void HandleExecutionResumedEvent(ExecutionResumedEvent resumedEvent, Action eventProcessor) { this.IsSuspended = false; + + // Add the actions stored in the suspendedActionsMap before back to orchestratorActionsMap to ensure proper sequencing. + if (this.suspendedActionsMap.Any()) + { + foreach(var pair in this.suspendedActionsMap) + { + this.orchestratorActionsMap.Add(pair.Key, pair.Value); + } + this.suspendedActionsMap.Clear(); + } + while (eventsWhileSuspended.Count > 0) { eventProcessor(eventsWhileSuspended.Dequeue()); @@ -608,7 +638,13 @@ public void FailOrchestration(Exception failure) details = orchestrationFailureException.Details; } } - else + else if (failure is TaskFailedException taskFailedException && + this.ErrorPropagationMode == ErrorPropagationMode.UseFailureDetails) + { + // Propagate the original FailureDetails + failureDetails = taskFailedException.FailureDetails; + } + else { if (this.ErrorPropagationMode == ErrorPropagationMode.UseFailureDetails) { diff --git a/src/DurableTask.Core/TaskOrchestrationDispatcher.cs b/src/DurableTask.Core/TaskOrchestrationDispatcher.cs index 6ebeaf89b..fc051994f 100644 --- a/src/DurableTask.Core/TaskOrchestrationDispatcher.cs +++ b/src/DurableTask.Core/TaskOrchestrationDispatcher.cs @@ -15,13 +15,13 @@ namespace DurableTask.Core { using System; using System.Collections.Generic; - using System.Collections.ObjectModel; using System.Diagnostics; using System.Linq; using System.Threading; using System.Threading.Tasks; using DurableTask.Core.Command; using DurableTask.Core.Common; + using DurableTask.Core.Entities; using DurableTask.Core.Exceptions; using DurableTask.Core.History; using DurableTask.Core.Logging; @@ -43,6 +43,9 @@ public class TaskOrchestrationDispatcher readonly LogHelper logHelper; ErrorPropagationMode errorPropagationMode; readonly NonBlockingCountdownLock concurrentSessionLock; + readonly IEntityOrchestrationService? entityOrchestrationService; + readonly EntityBackendProperties? entityBackendProperties; + readonly TaskOrchestrationEntityParameters? entityParameters; internal TaskOrchestrationDispatcher( IOrchestrationService orchestrationService, @@ -56,6 +59,9 @@ internal TaskOrchestrationDispatcher( this.dispatchPipeline = dispatchPipeline ?? throw new ArgumentNullException(nameof(dispatchPipeline)); this.logHelper = logHelper ?? throw new ArgumentNullException(nameof(logHelper)); this.errorPropagationMode = errorPropagationMode; + this.entityOrchestrationService = orchestrationService as IEntityOrchestrationService; + this.entityBackendProperties = this.entityOrchestrationService?.EntityBackendProperties; + this.entityParameters = TaskOrchestrationEntityParameters.FromEntityBackendProperties(this.entityBackendProperties); this.dispatcher = new WorkItemDispatcher( "TaskOrchestrationDispatcher", @@ -113,7 +119,18 @@ public async Task StopAsync(bool forced) /// A new TaskOrchestrationWorkItem protected Task OnFetchWorkItemAsync(TimeSpan receiveTimeout, CancellationToken cancellationToken) { - return this.orchestrationService.LockNextTaskOrchestrationWorkItemAsync(receiveTimeout, cancellationToken); + if (this.entityBackendProperties?.UseSeparateQueueForEntityWorkItems == true) + { + // only orchestrations should be served by this dispatcher, so we call + // the method which returns work items for orchestrations only. + return this.entityOrchestrationService!.LockNextOrchestrationWorkItemAsync(receiveTimeout, cancellationToken); + } + else + { + // both entities and orchestrations are served by this dispatcher, + // so we call the method that may return work items for either. + return this.orchestrationService.LockNextTaskOrchestrationWorkItemAsync(receiveTimeout, cancellationToken); + } } @@ -296,6 +313,7 @@ protected async Task OnProcessWorkItemAsync(TaskOrchestrationWorkItem work IList? carryOverEvents = null; string? carryOverStatus = null; + workItem.OrchestrationRuntimeState.LogHelper = this.logHelper; OrchestrationRuntimeState runtimeState = workItem.OrchestrationRuntimeState; runtimeState.AddEvent(new OrchestratorStartedEvent(-1)); @@ -318,14 +336,14 @@ protected async Task OnProcessWorkItemAsync(TaskOrchestrationWorkItem work { // start a task to run RenewUntil renewTask = Task.Factory.StartNew( - () => this.RenewUntil(workItem, renewCancellationTokenSource.Token), + () => RenewUntil(workItem, this.orchestrationService, this.logHelper, nameof(TaskOrchestrationDispatcher), renewCancellationTokenSource.Token), renewCancellationTokenSource.Token); } try { // Assumes that: if the batch contains a new "ExecutionStarted" event, it is the first message in the batch. - if (!this.ReconcileMessagesWithState(workItem)) + if (!ReconcileMessagesWithState(workItem, nameof(TaskOrchestrationDispatcher), this.errorPropagationMode, logHelper)) { // TODO : mark an orchestration as faulted if there is data corruption this.logHelper.DroppingOrchestrationWorkItem(workItem, "Received work-item for an invalid orchestration"); @@ -613,10 +631,10 @@ static OrchestrationExecutionContext GetOrchestrationExecutionContext(Orchestrat return new OrchestrationExecutionContext { OrchestrationTags = runtimeState.Tags ?? new Dictionary(capacity: 0) }; } - TimeSpan MinRenewalInterval = TimeSpan.FromSeconds(5); // prevents excessive retries if clocks are off - TimeSpan MaxRenewalInterval = TimeSpan.FromSeconds(30); + static TimeSpan MinRenewalInterval = TimeSpan.FromSeconds(5); // prevents excessive retries if clocks are off + static TimeSpan MaxRenewalInterval = TimeSpan.FromSeconds(30); - async Task RenewUntil(TaskOrchestrationWorkItem workItem, CancellationToken cancellationToken) + internal static async Task RenewUntil(TaskOrchestrationWorkItem workItem, IOrchestrationService orchestrationService, LogHelper logHelper, string dispatcher, CancellationToken cancellationToken) { while (!cancellationToken.IsCancellationRequested) { @@ -639,16 +657,16 @@ async Task RenewUntil(TaskOrchestrationWorkItem workItem, CancellationToken canc try { - this.logHelper.RenewOrchestrationWorkItemStarting(workItem); - TraceHelper.Trace(TraceEventType.Information, "TaskOrchestrationDispatcher-RenewWorkItemStarting", "Renewing work item for instance {0}", workItem.InstanceId); - await this.orchestrationService.RenewTaskOrchestrationWorkItemLockAsync(workItem); - this.logHelper.RenewOrchestrationWorkItemCompleted(workItem); - TraceHelper.Trace(TraceEventType.Information, "TaskOrchestrationDispatcher-RenewWorkItemCompleted", "Successfully renewed work item for instance {0}", workItem.InstanceId); + logHelper.RenewOrchestrationWorkItemStarting(workItem); + TraceHelper.Trace(TraceEventType.Information, $"{dispatcher}-RenewWorkItemStarting", "Renewing work item for instance {0}", workItem.InstanceId); + await orchestrationService.RenewTaskOrchestrationWorkItemLockAsync(workItem); + logHelper.RenewOrchestrationWorkItemCompleted(workItem); + TraceHelper.Trace(TraceEventType.Information, $"{dispatcher}-RenewWorkItemCompleted", "Successfully renewed work item for instance {0}", workItem.InstanceId); } catch (Exception exception) when (!Utils.IsFatal(exception)) { - this.logHelper.RenewOrchestrationWorkItemFailed(workItem, exception); - TraceHelper.TraceException(TraceEventType.Warning, "TaskOrchestrationDispatcher-RenewWorkItemFailed", exception, "Failed to renew work item for instance {0}", workItem.InstanceId); + logHelper.RenewOrchestrationWorkItemFailed(workItem, exception); + TraceHelper.TraceException(TraceEventType.Warning, $"{dispatcher}-RenewWorkItemFailed", exception, "Failed to renew work item for instance {0}", workItem.InstanceId); } } } @@ -666,6 +684,7 @@ async Task ExecuteOrchestrationAsync(Orchestration dispatchContext.SetProperty(runtimeState); dispatchContext.SetProperty(workItem); dispatchContext.SetProperty(GetOrchestrationExecutionContext(runtimeState)); + dispatchContext.SetProperty(this.entityParameters); TaskOrchestrationExecutor? executor = null; @@ -693,7 +712,9 @@ await this.dispatchPipeline.RunAsync(dispatchContext, _ => runtimeState, taskOrchestration, this.orchestrationService.EventBehaviourForContinueAsNew, + this.entityParameters, this.errorPropagationMode); + OrchestratorExecutionResult resultFromOrchestrator = executor.Execute(); dispatchContext.SetProperty(resultFromOrchestrator); return CompletedTask; @@ -735,8 +756,11 @@ await this.dispatchPipeline.RunAsync(dispatchContext, _ => /// Assumes that: if the batch contains a new "ExecutionStarted" event, it is the first message in the batch. /// /// A batch of work item messages. + /// The name of the dispatcher, used for tracing. + /// The error propagation mode. + /// The log helper. /// True if workItem should be processed further. False otherwise. - bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) + internal static bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem, string dispatcher, ErrorPropagationMode errorPropagationMode, LogHelper logHelper) { foreach (TaskMessage message in workItem.NewMessages) { @@ -745,7 +769,7 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) { throw TraceHelper.TraceException( TraceEventType.Error, - "TaskOrchestrationDispatcher-OrchestrationInstanceMissing", + $"{dispatcher}-OrchestrationInstanceMissing", new InvalidOperationException("Message does not contain any OrchestrationInstance information")); } @@ -763,10 +787,10 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) return false; } - this.logHelper.ProcessingOrchestrationMessage(workItem, message); + logHelper.ProcessingOrchestrationMessage(workItem, message); TraceHelper.TraceInstance( TraceEventType.Information, - "TaskOrchestrationDispatcher-ProcessEvent", + $"{dispatcher}-ProcessEvent", orchestrationInstance!, "Processing new event with Id {0} and type {1}", message.Event.EventId, @@ -777,10 +801,10 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) if (workItem.OrchestrationRuntimeState.ExecutionStartedEvent != null) { // this was caused due to a dupe execution started event, swallow this one - this.logHelper.DroppingOrchestrationMessage(workItem, message, "Duplicate start event"); + logHelper.DroppingOrchestrationMessage(workItem, message, "Duplicate start event"); TraceHelper.TraceInstance( TraceEventType.Warning, - "TaskOrchestrationDispatcher-DuplicateStartEvent", + $"{dispatcher}-DuplicateStartEvent", orchestrationInstance!, "Duplicate start event. Ignoring event with Id {0} and type {1} ", message.Event.EventId, @@ -794,13 +818,13 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) workItem.OrchestrationRuntimeState.OrchestrationInstance?.ExecutionId)) { // eat up any events for previous executions - this.logHelper.DroppingOrchestrationMessage( + logHelper.DroppingOrchestrationMessage( workItem, message, $"ExecutionId of event ({orchestrationInstance.ExecutionId}) does not match current executionId"); TraceHelper.TraceInstance( TraceEventType.Warning, - "TaskOrchestrationDispatcher-ExecutionIdMismatch", + $"{dispatcher}-ExecutionIdMismatch", orchestrationInstance, "ExecutionId of event does not match current executionId. Ignoring event with Id {0} and type {1} ", message.Event.EventId, @@ -818,29 +842,29 @@ bool ReconcileMessagesWithState(TaskOrchestrationWorkItem workItem) } else if (historyEvent is SubOrchestrationInstanceCompletedEvent subOrchestrationInstanceCompletedEvent) { - SubOrchestrationInstanceCreatedEvent subOrchestrationCreatedEvent = (SubOrchestrationInstanceCreatedEvent)workItem.OrchestrationRuntimeState.Events.FirstOrDefault(x => x.EventId == subOrchestrationInstanceCompletedEvent.TaskScheduledId); + SubOrchestrationInstanceCreatedEvent subOrchestrationCreatedEvent = workItem.OrchestrationRuntimeState.Events.OfType().FirstOrDefault(x => x.EventId == subOrchestrationInstanceCompletedEvent.TaskScheduledId); // We immediately publish the activity span for this sub-orchestration by creating the activity and immediately calling Dispose() on it. TraceHelper.EmitTraceActivityForSubOrchestrationCompleted(workItem.OrchestrationRuntimeState.OrchestrationInstance, subOrchestrationCreatedEvent); } else if (historyEvent is SubOrchestrationInstanceFailedEvent subOrchestrationInstanceFailedEvent) { - SubOrchestrationInstanceCreatedEvent subOrchestrationCreatedEvent = (SubOrchestrationInstanceCreatedEvent)workItem.OrchestrationRuntimeState.Events.FirstOrDefault(x => x.EventId == subOrchestrationInstanceFailedEvent.TaskScheduledId); + SubOrchestrationInstanceCreatedEvent subOrchestrationCreatedEvent = workItem.OrchestrationRuntimeState.Events.OfType().FirstOrDefault(x => x.EventId == subOrchestrationInstanceFailedEvent.TaskScheduledId); // We immediately publish the activity span for this sub-orchestration by creating the activity and immediately calling Dispose() on it. - TraceHelper.EmitTraceActivityForSubOrchestrationFailed(workItem.OrchestrationRuntimeState.OrchestrationInstance, subOrchestrationCreatedEvent, subOrchestrationInstanceFailedEvent, this.errorPropagationMode); + TraceHelper.EmitTraceActivityForSubOrchestrationFailed(workItem.OrchestrationRuntimeState.OrchestrationInstance, subOrchestrationCreatedEvent, subOrchestrationInstanceFailedEvent, errorPropagationMode); } } if (message.Event is TaskCompletedEvent taskCompletedEvent) { - TaskScheduledEvent taskScheduledEvent = (TaskScheduledEvent)workItem.OrchestrationRuntimeState.Events.LastOrDefault(x => x.EventId == taskCompletedEvent.TaskScheduledId); + TaskScheduledEvent taskScheduledEvent = workItem.OrchestrationRuntimeState.Events.OfType().LastOrDefault(x => x.EventId == taskCompletedEvent.TaskScheduledId); TraceHelper.EmitTraceActivityForTaskCompleted(workItem.OrchestrationRuntimeState.OrchestrationInstance, taskScheduledEvent); } else if (message.Event is TaskFailedEvent taskFailedEvent) { - TaskScheduledEvent taskScheduledEvent = (TaskScheduledEvent)workItem.OrchestrationRuntimeState.Events.LastOrDefault(x => x.EventId == taskFailedEvent.TaskScheduledId); - TraceHelper.EmitTraceActivityForTaskFailed(workItem.OrchestrationRuntimeState.OrchestrationInstance, taskScheduledEvent, taskFailedEvent, this.errorPropagationMode); + TaskScheduledEvent taskScheduledEvent = workItem.OrchestrationRuntimeState.Events.OfType().LastOrDefault(x => x.EventId == taskFailedEvent.TaskScheduledId); + TraceHelper.EmitTraceActivityForTaskFailed(workItem.OrchestrationRuntimeState.OrchestrationInstance, taskScheduledEvent, taskFailedEvent, errorPropagationMode); } workItem.OrchestrationRuntimeState.AddEvent(message.Event); @@ -1145,7 +1169,7 @@ TaskMessage ProcessSendEventDecision( }; } - class NonBlockingCountdownLock + internal class NonBlockingCountdownLock { int available; diff --git a/src/DurableTask.Core/TaskOrchestrationExecutor.cs b/src/DurableTask.Core/TaskOrchestrationExecutor.cs index b0ca99976..81ea37778 100644 --- a/src/DurableTask.Core/TaskOrchestrationExecutor.cs +++ b/src/DurableTask.Core/TaskOrchestrationExecutor.cs @@ -14,7 +14,6 @@ namespace DurableTask.Core { using System; - using System.Collections; using System.Collections.Generic; using System.Diagnostics; using System.Linq; @@ -22,6 +21,7 @@ namespace DurableTask.Core using System.Threading; using System.Threading.Tasks; using DurableTask.Core.Common; + using DurableTask.Core.Entities; using DurableTask.Core.Exceptions; using DurableTask.Core.History; @@ -43,23 +43,43 @@ public class TaskOrchestrationExecutor /// /// /// + /// /// public TaskOrchestrationExecutor( OrchestrationRuntimeState orchestrationRuntimeState, TaskOrchestration taskOrchestration, BehaviorOnContinueAsNew eventBehaviourForContinueAsNew, + TaskOrchestrationEntityParameters? entityParameters, ErrorPropagationMode errorPropagationMode = ErrorPropagationMode.SerializeExceptions) { this.decisionScheduler = new SynchronousTaskScheduler(); this.context = new TaskOrchestrationContext( orchestrationRuntimeState.OrchestrationInstance, this.decisionScheduler, + entityParameters, errorPropagationMode); this.orchestrationRuntimeState = orchestrationRuntimeState; this.taskOrchestration = taskOrchestration; this.skipCarryOverEvents = eventBehaviourForContinueAsNew == BehaviorOnContinueAsNew.Ignore; } + /// + /// Initializes a new instance of the class. + /// This overload is needed only to avoid breaking changes because this is a public constructor. + /// + /// + /// + /// + /// + public TaskOrchestrationExecutor( + OrchestrationRuntimeState orchestrationRuntimeState, + TaskOrchestration taskOrchestration, + BehaviorOnContinueAsNew eventBehaviourForContinueAsNew, + ErrorPropagationMode errorPropagationMode = ErrorPropagationMode.SerializeExceptions) + : this(orchestrationRuntimeState, taskOrchestration, eventBehaviourForContinueAsNew, entityParameters: null, errorPropagationMode) + { + } + internal bool IsCompleted => this.result != null && (this.result.IsCompleted || this.result.IsFaulted); /// @@ -188,6 +208,7 @@ void ProcessEvent(HistoryEvent historyEvent) { case EventType.ExecutionStarted: var executionStartedEvent = (ExecutionStartedEvent)historyEvent; + this.context.Version = executionStartedEvent.Version; this.result = this.taskOrchestration.Execute(this.context, executionStartedEvent.Input); break; case EventType.ExecutionTerminated: diff --git a/src/DurableTask.Core/Tracing/TraceHelper.cs b/src/DurableTask.Core/Tracing/TraceHelper.cs index 768d6bda6..143ba8af6 100644 --- a/src/DurableTask.Core/Tracing/TraceHelper.cs +++ b/src/DurableTask.Core/Tracing/TraceHelper.cs @@ -20,7 +20,6 @@ namespace DurableTask.Core.Tracing using System.Runtime.ExceptionServices; using DurableTask.Core.Common; using DurableTask.Core.History; - using DurableTask.Core.Serializing; /// /// Helper class for logging/tracing diff --git a/src/DurableTask.Emulator/DurableTask.Emulator.csproj b/src/DurableTask.Emulator/DurableTask.Emulator.csproj index e165bbe31..b89e40bcc 100644 --- a/src/DurableTask.Emulator/DurableTask.Emulator.csproj +++ b/src/DurableTask.Emulator/DurableTask.Emulator.csproj @@ -8,14 +8,6 @@ NU5125;NU5048 - - - - - - - - diff --git a/src/DurableTask.ServiceBus/Common/Abstraction/ServiceBusAbstraction.cs b/src/DurableTask.ServiceBus/Common/Abstraction/ServiceBusAbstraction.cs index 0b18a4910..779f95c9d 100644 --- a/src/DurableTask.ServiceBus/Common/Abstraction/ServiceBusAbstraction.cs +++ b/src/DurableTask.ServiceBus/Common/Abstraction/ServiceBusAbstraction.cs @@ -8,6 +8,7 @@ namespace DurableTask.ServiceBus.Common.Abstraction using System.Runtime.Serialization; using System.Threading.Tasks; using System.Xml; + using DurableTask.ServiceBus.Tracking; #if !NETSTANDARD2_0 using Microsoft.ServiceBus.Messaging; @@ -119,50 +120,60 @@ public override void WriteStartObject(XmlDictionaryWriter writer, object graph) /// public class IMessageSession { - Microsoft.Azure.ServiceBus.IMessageSession session; + Azure.Messaging.ServiceBus.ServiceBusSessionReceiver sessionReceiver; - public IMessageSession(Microsoft.Azure.ServiceBus.IMessageSession session) + public IMessageSession(Azure.Messaging.ServiceBus.ServiceBusSessionReceiver sessionReceiver) { - this.session = session; + this.sessionReceiver = sessionReceiver; } - public string SessionId => this.session.SessionId; + public string SessionId => this.sessionReceiver.SessionId; - public DateTime LockedUntilUtc => this.session.LockedUntilUtc; + public DateTime LockedUntilUtc => this.sessionReceiver.SessionLockedUntil.UtcDateTime; public async Task GetStateAsync() { - return await this.session.GetStateAsync(); + return (await this.sessionReceiver.GetSessionStateAsync())?.ToArray(); } public async Task SetStateAsync(byte[] sessionState) { - await this.session.SetStateAsync(sessionState); + if (sessionState == null) + { + await this.sessionReceiver.SetSessionStateAsync(new BinaryData(new byte[] { })); + } + else + { + await this.sessionReceiver.SetSessionStateAsync(new BinaryData(sessionState)); + } } public async Task RenewSessionLockAsync() { - await this.session.RenewSessionLockAsync(); + await this.sessionReceiver.RenewSessionLockAsync(); } - public async Task AbandonAsync(string lockToken) + public async Task AbandonAsync(Message message) { - await this.session.AbandonAsync(lockToken); + await this.sessionReceiver.AbandonMessageAsync(message); } public async Task> ReceiveAsync(int maxMessageCount) { - return (await this.session.ReceiveAsync(maxMessageCount)).Select(x => (Message)x).ToList(); + return (await this.sessionReceiver.ReceiveMessagesAsync(maxMessageCount)).Select(x => (Message)x).ToList(); } - public async Task CompleteAsync(IEnumerable lockTokens) + public async Task CompleteAsync(IEnumerable messages) { - await this.session.CompleteAsync(lockTokens); + foreach (var message in messages) + { + await this.sessionReceiver.CompleteMessageAsync(message); + } } public async Task CloseAsync() { - await this.session.CloseAsync(); + await this.sessionReceiver.CloseAsync(); } #else public class IMessageSession @@ -213,9 +224,9 @@ public async Task RenewSessionLockAsync() await this.session.RenewLockAsync(); } - public async Task AbandonAsync(string lockToken) + public async Task AbandonAsync(Message message) { - await this.session.AbandonAsync(Guid.Parse(lockToken)); + await this.session.AbandonAsync(message.SystemProperties.LockToken); } public async Task> ReceiveAsync(int maxMessageCount) @@ -223,9 +234,9 @@ public async Task> ReceiveAsync(int maxMessageCount) return (await this.session.ReceiveBatchAsync(maxMessageCount)).Select(x => (Message)x).ToList(); } - public async Task CompleteAsync(IEnumerable lockTokens) + public async Task CompleteAsync(IEnumerable messages) { - await this.session.CompleteBatchAsync(lockTokens.Select(Guid.Parse)); + await this.session.CompleteBatchAsync(messages.Select(m => m.SystemProperties.LockToken).ToList()); } public async Task CloseAsync() @@ -237,59 +248,175 @@ public async Task CloseAsync() } #if NETSTANDARD2_0 + public abstract class Union + { + public abstract T Match(Func f, Func g); + + public abstract void Switch(Action f, Action g); + + private Union() { } + + public sealed class Case1 : Union + { + public readonly A Item; + public Case1(A item) : base() { this.Item = item; } + public override T Match(Func f, Func g) + { + return f(Item); + } + public override void Switch(Action f, Action g) + { + f(Item); + } + } + + public sealed class Case2 : Union + { + public readonly B Item; + public Case2(B item) { this.Item = item; } + public override T Match(Func f, Func g) + { + return g(Item); + } + public override void Switch(Action f, Action g) + { + g(Item); + } + } + } + /// public class Message { - Microsoft.Azure.ServiceBus.Message msg; + private readonly Union message; + + public Message(Azure.Messaging.ServiceBus.ServiceBusMessage msg) + { + this.message = new Union.Case1(msg); + this.SystemProperties = new SystemPropertiesCollection(message); + } - public Message(Microsoft.Azure.ServiceBus.Message msg) + public Message(Azure.Messaging.ServiceBus.ServiceBusReceivedMessage msg) { - this.msg = msg; + this.message = new Union.Case2(msg); + this.SystemProperties = new SystemPropertiesCollection(message); } - public static implicit operator Message(Microsoft.Azure.ServiceBus.Message m) + public static implicit operator Message(Azure.Messaging.ServiceBus.ServiceBusMessage m) { return m == null ? null : new Message(m); } - public static implicit operator Microsoft.Azure.ServiceBus.Message(Message m) + public static implicit operator Azure.Messaging.ServiceBus.ServiceBusMessage(Message m) { - return m.msg; + return m.message.Match( + m => m, + rm => new Azure.Messaging.ServiceBus.ServiceBusMessage(rm)); + } + + public static implicit operator Message(Azure.Messaging.ServiceBus.ServiceBusReceivedMessage m) + { + return m == null ? null : new Message(m); + } + + public static implicit operator Azure.Messaging.ServiceBus.ServiceBusReceivedMessage(Message m) + { + return m.message.Match( + m => throw NotReceivedMessagePropertyGetException(), + rm => rm); } public Message() { - this.msg = new Microsoft.Azure.ServiceBus.Message(); + this.message = new Union.Case1(new Azure.Messaging.ServiceBus.ServiceBusMessage()); + this.SystemProperties = new SystemPropertiesCollection(message); } public Message(byte[] serializableObject) { - this.msg = new Microsoft.Azure.ServiceBus.Message(serializableObject); + this.message = new Union.Case1(new Azure.Messaging.ServiceBus.ServiceBusMessage(new BinaryData(serializableObject))); + this.SystemProperties = new SystemPropertiesCollection(message); } public string MessageId { - get => this.msg?.MessageId; - set => this.msg.MessageId = value; + get => this.message?.Match( + m => m.MessageId, + rm => rm.MessageId); + set => this.message.Switch( + m => { m.MessageId = value; }, + rm => throw ReceivedMessagePropertySetException()); } public DateTime ScheduledEnqueueTimeUtc { - get => this.msg.ScheduledEnqueueTimeUtc; - set => this.msg.ScheduledEnqueueTimeUtc = value; + get => this.message.Match( + m => m.ScheduledEnqueueTime.UtcDateTime, + rm => rm.ScheduledEnqueueTime.UtcDateTime); + set => this.message.Switch( + m => { m.ScheduledEnqueueTime = new DateTimeOffset(value); }, + rm => throw ReceivedMessagePropertySetException()); } - public Microsoft.Azure.ServiceBus.Message.SystemPropertiesCollection SystemProperties => this.msg.SystemProperties; + public SystemPropertiesCollection SystemProperties { get; private set; } - public IDictionary UserProperties => this.msg.UserProperties; + public IDictionary UserProperties => this.message.Match( + m => m.ApplicationProperties, + rm => (IDictionary)rm.ApplicationProperties); - public byte[] Body => this.msg.Body; + public byte[] Body => this.message.Match( + m => m.Body.ToArray(), + rm => rm.Body.ToArray()); public string SessionId { - get => this.msg?.SessionId; - set => this.msg.SessionId = value; + get => this.message?.Match( + m => m.SessionId, + rm => rm.SessionId); + set => this.message.Switch( + m => { m.SessionId = value; }, + rm => throw ReceivedMessagePropertySetException()); + } + + private static Exception NotReceivedMessagePropertyGetException() + { + return new InvalidOperationException("The property cannot be accessed because the message is not received."); + } + + private static Exception ReceivedMessagePropertySetException() + { + return new InvalidOperationException("The property cannot be set because the message is received and the value is readonly."); + } + + public class SystemPropertiesCollection + { + private readonly Union msg; + + public SystemPropertiesCollection(Union msg) + { + this.msg = msg; + } + + public Guid LockToken => this.msg.Match( + _ => throw NotReceivedMessagePropertyGetException(), + rm => Guid.Parse(rm.LockToken)); + + public int DeliveryCount => this.msg.Match( + _ => throw NotReceivedMessagePropertyGetException(), + rm => rm.DeliveryCount); + + public DateTime LockedUntilUtc => this.msg.Match( + _ => throw NotReceivedMessagePropertyGetException(), + rm => rm.LockedUntil.UtcDateTime); + + public long SequenceNumber => this.msg.Match( + _ => throw NotReceivedMessagePropertyGetException(), + rm => rm.SequenceNumber); + + public DateTime EnqueuedTimeUtc => this.msg.Match( + _ => throw NotReceivedMessagePropertyGetException(), + rm => rm.EnqueuedTime.UtcDateTime); } #else @@ -426,7 +553,7 @@ public SystemPropertiesCollection(BrokeredMessage brokered) #if NETSTANDARD2_0 /// - public abstract class RetryPolicy : Microsoft.Azure.ServiceBus.RetryPolicy + public abstract class RetryPolicy : Azure.Messaging.ServiceBus.ServiceBusRetryPolicy { #else @@ -456,32 +583,48 @@ public static implicit operator Microsoft.ServiceBus.RetryPolicy(RetryPolicy rp) #if NETSTANDARD2_0 /// - public class ServiceBusConnection : Microsoft.Azure.ServiceBus.ServiceBusConnection + public class ServiceBusConnection { + private readonly Dictionary sendViaEntityScopedServiceBusClients; + private readonly Func serviceBusClientFactory; + public ServiceBusConnection(ServiceBusConnectionStringBuilder connectionStringBuilder) - : base(connectionStringBuilder) { - } + var options = new Azure.Messaging.ServiceBus.ServiceBusClientOptions + { + EnableCrossEntityTransactions = true, + }; - public ServiceBusConnection(string namespaceConnectionString) - : base(namespaceConnectionString) - { + sendViaEntityScopedServiceBusClients = new Dictionary(); + serviceBusClientFactory = () => new Azure.Messaging.ServiceBus.ServiceBusClient( + connectionStringBuilder.ConnectionString, + options); } - public ServiceBusConnection(string namespaceConnectionString, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(namespaceConnectionString, retryPolicy) + public ServiceBusConnection(string endpoint, Azure.Messaging.ServiceBus.ServiceBusTransportType transportType, Azure.Core.TokenCredential tokenCredential, RetryPolicy retryPolicy = null) { - } + var options = new Azure.Messaging.ServiceBus.ServiceBusClientOptions + { + EnableCrossEntityTransactions = true, + }; - [Obsolete] - public ServiceBusConnection(string namespaceConnectionString, TimeSpan operationTimeout, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(namespaceConnectionString, operationTimeout, retryPolicy) - { + sendViaEntityScopedServiceBusClients = new Dictionary(); + serviceBusClientFactory = () => new Azure.Messaging.ServiceBus.ServiceBusClient( + endpoint, + tokenCredential, + options); } - public ServiceBusConnection(string endpoint, Microsoft.Azure.ServiceBus.TransportType transportType, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(endpoint, transportType, retryPolicy) + public Azure.Messaging.ServiceBus.ServiceBusClient GetSendViaEntityScopedClient(string sendViaEntity) { + if (sendViaEntityScopedServiceBusClients.TryGetValue(sendViaEntity, out var client)) + { + return client; + } + + var newClient = serviceBusClientFactory(); + sendViaEntityScopedServiceBusClients.Add(sendViaEntity, newClient); + return newClient; } #else public class ServiceBusConnection @@ -500,12 +643,21 @@ public ServiceBusConnection(ServiceBusConnectionStringBuilder connectionStringBu #if NETSTANDARD2_0 /// - public class ServiceBusConnectionStringBuilder : Microsoft.Azure.ServiceBus.ServiceBusConnectionStringBuilder + public class ServiceBusConnectionStringBuilder { + public string ConnectionString { get; set; } + + public Azure.Messaging.ServiceBus.ServiceBusConnectionStringProperties ConnectionStringProperties { get; set; } + public ServiceBusConnectionStringBuilder(string connectionString) - : base(connectionString) { + ConnectionString = connectionString; + ConnectionStringProperties = Azure.Messaging.ServiceBus.ServiceBusConnectionStringProperties.Parse(ConnectionString); } + + public string SasKeyName => ConnectionStringProperties.SharedAccessKeyName; + + public string SasKey => ConnectionStringProperties.SharedAccessKey; #else public class ServiceBusConnectionStringBuilder : Microsoft.ServiceBus.ServiceBusConnectionStringBuilder { @@ -522,30 +674,6 @@ public ServiceBusConnectionStringBuilder(string connectionString) } #if NETSTANDARD2_0 - /// - public class TokenProvider : Microsoft.Azure.ServiceBus.Primitives.ITokenProvider - { - private readonly Microsoft.Azure.ServiceBus.Primitives.TokenProvider tokenProvider; - - public TokenProvider(Microsoft.Azure.ServiceBus.Primitives.TokenProvider t) - { - this.tokenProvider = t; - } - - public static implicit operator TokenProvider(Microsoft.Azure.ServiceBus.Primitives.TokenProvider t) - { - return new TokenProvider(t); - } - - public async Task GetTokenAsync(string appliesTo, TimeSpan timeout) - { - return await this.tokenProvider.GetTokenAsync(appliesTo, timeout); - } - - public static TokenProvider CreateSharedAccessSignatureTokenProvider(string keyName, string sharedAccessKey, TimeSpan tokenTimeToLive) - { - return Microsoft.Azure.ServiceBus.Primitives.TokenProvider.CreateSharedAccessSignatureTokenProvider(keyName, sharedAccessKey, tokenTimeToLive); - } #else public class TokenProvider { @@ -570,46 +698,44 @@ public static TokenProvider CreateSharedAccessSignatureTokenProvider(string keyN { return Microsoft.ServiceBus.TokenProvider.CreateSharedAccessSignatureTokenProvider(keyName, sharedAccessKey, tokenTimeToLive); } -#endif } +#endif #if NETSTANDARD2_0 /// - public class MessageSender : Microsoft.Azure.ServiceBus.Core.MessageSender + public class MessageSender { - public MessageSender(ServiceBusConnectionStringBuilder connectionStringBuilder, RetryPolicy retryPolicy = null) - : base(connectionStringBuilder, retryPolicy) - { - } - - public MessageSender(string connectionString, string entityPath, RetryPolicy retryPolicy = null) - : base(connectionString, entityPath, retryPolicy) - { - } - - public MessageSender(string endpoint, string entityPath, Microsoft.Azure.ServiceBus.Primitives.ITokenProvider tokenProvider, Microsoft.Azure.ServiceBus.TransportType transportType = Microsoft.Azure.ServiceBus.TransportType.Amqp, RetryPolicy retryPolicy = null) - : base(endpoint, entityPath, tokenProvider, transportType, retryPolicy) - { - } + private readonly Azure.Messaging.ServiceBus.ServiceBusSender sender; public MessageSender(ServiceBusConnection serviceBusConnection, string entityPath, RetryPolicy retryPolicy = null) - : base(serviceBusConnection, entityPath, retryPolicy) { + sender = serviceBusConnection.GetSendViaEntityScopedClient(entityPath).CreateSender(entityPath); } public MessageSender(ServiceBusConnection serviceBusConnection, string entityPath, string viaEntityPath, RetryPolicy retryPolicy = null) - : base(serviceBusConnection, entityPath, viaEntityPath, retryPolicy) { + sender = serviceBusConnection.GetSendViaEntityScopedClient(viaEntityPath).CreateSender(entityPath); } public async Task SendAsync(Message message) { - await base.SendAsync(message); + await sender.SendMessageAsync(message); } public async Task SendAsync(IEnumerable messageList) { - await base.SendAsync(messageList.Select(x => (Microsoft.Azure.ServiceBus.Message)x).ToList()); + var batch = await sender.CreateMessageBatchAsync(); + foreach (var message in messageList) + { + batch.TryAddMessage(message); + } + + await sender.SendMessagesAsync(batch); + } + + public async Task CloseAsync() + { + await sender.CloseAsync(); } @@ -676,26 +802,43 @@ public async Task CloseAsync() #if NETSTANDARD2_0 /// - public class MessageReceiver : Microsoft.Azure.ServiceBus.Core.MessageReceiver + public class MessageReceiver { - public MessageReceiver(Microsoft.Azure.ServiceBus.ServiceBusConnectionStringBuilder connectionStringBuilder, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null, int prefetchCount = 0) - : base(connectionStringBuilder, receiveMode, retryPolicy, prefetchCount) + private readonly Azure.Messaging.ServiceBus.ServiceBusReceiver receiver; + + public MessageReceiver(ServiceBusConnection serviceBusConnection, string entityPath, Azure.Messaging.ServiceBus.ServiceBusReceiveMode receiveMode = Azure.Messaging.ServiceBus.ServiceBusReceiveMode.PeekLock, RetryPolicy retryPolicy = null, int prefetchCount = 0) { + var options = new Azure.Messaging.ServiceBus.ServiceBusReceiverOptions + { + ReceiveMode = receiveMode, + PrefetchCount = prefetchCount + }; + this.receiver = serviceBusConnection.GetSendViaEntityScopedClient(entityPath).CreateReceiver(entityPath, options); } - public MessageReceiver(string connectionString, string entityPath, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null, int prefetchCount = 0) - : base(connectionString, entityPath, receiveMode, retryPolicy, prefetchCount) + public async Task ReceiveAsync(TimeSpan serverWaitTime) { + return await this.receiver.ReceiveMessageAsync(serverWaitTime); } - public MessageReceiver(string endpoint, string entityPath, Microsoft.Azure.ServiceBus.Primitives.ITokenProvider tokenProvider, Microsoft.Azure.ServiceBus.TransportType transportType = Microsoft.Azure.ServiceBus.TransportType.Amqp, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null, int prefetchCount = 0) - : base(endpoint, entityPath, tokenProvider, transportType, receiveMode, retryPolicy, prefetchCount) + public async Task RenewLockAsync(Message message) { + await this.receiver.RenewMessageLockAsync(message); } - public MessageReceiver(Microsoft.Azure.ServiceBus.ServiceBusConnection serviceBusConnection, string entityPath, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null, int prefetchCount = 0) - : base(serviceBusConnection, entityPath, receiveMode, retryPolicy, prefetchCount) + public async Task CompleteAsync(Message message) + { + await this.receiver.CompleteMessageAsync(message); + } + + public async Task CloseAsync() { + await this.receiver.CloseAsync(); + } + + public async Task AbandonAsync(Message message) + { + await this.receiver.AbandonMessageAsync(message); } #else @@ -736,9 +879,9 @@ public static implicit operator MessageReceiver(Microsoft.ServiceBus.Messaging.M return new MessageReceiver(mr); } - public async Task AbandonAsync(Guid lockToken) + public async Task AbandonAsync(Message message) { - await this.msgReceiver.AbandonAsync(lockToken); + await this.msgReceiver.AbandonAsync(message.SystemProperties.LockToken); } public async Task CloseAsync() @@ -746,9 +889,9 @@ public async Task CloseAsync() await this.msgReceiver.CloseAsync(); } - public async Task CompleteAsync(Guid lockToken) + public async Task CompleteAsync(Message message) { - await this.msgReceiver.CompleteAsync(lockToken); + await this.msgReceiver.CompleteAsync(message.SystemProperties.LockToken); } public async Task ReceiveAsync(TimeSpan serverWaitTime) @@ -766,31 +909,23 @@ public async Task RenewLockAsync(Message message) #if NETSTANDARD2_0 /// - public class QueueClient : Microsoft.Azure.ServiceBus.QueueClient + public class QueueClient { - public QueueClient(Microsoft.Azure.ServiceBus.ServiceBusConnectionStringBuilder connectionStringBuilder, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(connectionStringBuilder, receiveMode, retryPolicy) - { - } + private readonly Azure.Messaging.ServiceBus.ServiceBusSender sender; - public QueueClient(string connectionString, string entityPath, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(connectionString, entityPath, receiveMode, retryPolicy) + public QueueClient(ServiceBusConnection serviceBusConnection, string entityPath) { + sender = serviceBusConnection.GetSendViaEntityScopedClient(entityPath).CreateSender(entityPath); } - public QueueClient(string endpoint, string entityPath, Microsoft.Azure.ServiceBus.Primitives.ITokenProvider tokenProvider, Microsoft.Azure.ServiceBus.TransportType transportType = Microsoft.Azure.ServiceBus.TransportType.Amqp, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode = Microsoft.Azure.ServiceBus.ReceiveMode.PeekLock, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy = null) - : base(endpoint, entityPath, tokenProvider, transportType, receiveMode, retryPolicy) - { - } - - public QueueClient(Microsoft.Azure.ServiceBus.ServiceBusConnection serviceBusConnection, string entityPath, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode, Microsoft.Azure.ServiceBus.RetryPolicy retryPolicy) - : base(serviceBusConnection, entityPath, receiveMode, retryPolicy) + public async Task SendAsync(List messageList) { + await sender.SendMessagesAsync(messageList.Select(m => (Azure.Messaging.ServiceBus.ServiceBusMessage)m).ToList()); } - public async Task SendAsync(List messageList) + public async Task SendAsync(Message message) { - await SendAsync(messageList.Select(x => (Microsoft.Azure.ServiceBus.Message)x).ToList()); + await sender.SendMessageAsync((Azure.Messaging.ServiceBus.ServiceBusMessage)message); } #else @@ -857,22 +992,75 @@ public async Task AcceptMessageSessionAsync(TimeSpan operationT } #if NETSTANDARD2_0 + public class QueueDescription + { + private readonly Union propertiesUnion; + + public QueueDescription(Azure.Messaging.ServiceBus.Administration.QueueProperties queueProperties) + { + this.propertiesUnion = new Union.Case1(queueProperties); + } + + public QueueDescription(Azure.Messaging.ServiceBus.Administration.QueueRuntimeProperties queueRuntimeProperties) + { + this.propertiesUnion = new Union.Case2(queueRuntimeProperties); + } + + public long MessageCount => this.propertiesUnion.Match( + _ => throw NotRuntimePropertiesGetException(), + runtimeProps => runtimeProps.TotalMessageCount); + + public string Path => this.propertiesUnion.Match( + props => props.Name, + runtimeProps => runtimeProps.Name); + + public int MaxDeliveryCount => this.propertiesUnion.Match( + props => props.MaxDeliveryCount, + _ => throw RuntimePropertiesGetException()); + + public long SizeInBytes => this.propertiesUnion.Match( + _ => throw NotRuntimePropertiesGetException(), + runtimeProps => runtimeProps.SizeInBytes); + + private static Exception RuntimePropertiesGetException() + { + return new InvalidOperationException($"The property cannot be accessed because the underlying object is {nameof(Azure.Messaging.ServiceBus.Administration.QueueRuntimeProperties)}"); + } + + private static Exception NotRuntimePropertiesGetException() + { + return new InvalidOperationException($"The property cannot be accessed because the underlying object is {nameof(Azure.Messaging.ServiceBus.Administration.QueueProperties)}"); + } + } + /// - public class ManagementClient : Microsoft.Azure.ServiceBus.Management.ManagementClient + public class ManagementClient : Azure.Messaging.ServiceBus.Administration.ServiceBusAdministrationClient { public ManagementClient(string connectionString) : base(connectionString) { } - public ManagementClient(string endpoint, Microsoft.Azure.ServiceBus.Primitives.ITokenProvider tokenProvider) - : base(endpoint, tokenProvider) + public ManagementClient(string endpoint, Azure.Core.TokenCredential tokenCredential) + : base(endpoint, tokenCredential) { } - public ManagementClient(Microsoft.Azure.ServiceBus.ServiceBusConnectionStringBuilder connectionStringBuilder, Microsoft.Azure.ServiceBus.Primitives.ITokenProvider tokenProvider = null) - : base(connectionStringBuilder, tokenProvider) + public async Task GetQueueRuntimeInfoAsync(string name) { + var response = await this.GetQueueRuntimePropertiesAsync(name); + return new QueueDescription(response.Value); + } + + public async Task> GetQueuesAsync() + { + List queueDescriptions = new List(); + await foreach (var queueProperties in base.GetQueuesAsync()) + { + queueDescriptions.Add(new QueueDescription(queueProperties)); + } + + return queueDescriptions; } #else public class ManagementClient @@ -919,40 +1107,38 @@ public async Task> GetQueuesAsync() #if NETSTANDARD2_0 public class SessionClient { - Microsoft.Azure.ServiceBus.SessionClient sessionClient; + private readonly Azure.Messaging.ServiceBus.ServiceBusClient serviceBusClient; + private readonly string entityPath; + private readonly Azure.Messaging.ServiceBus.ServiceBusReceiveMode receiveMode; - public SessionClient(ServiceBusConnection serviceBusConnection, string entityPath, Microsoft.Azure.ServiceBus.ReceiveMode receiveMode) + public SessionClient(ServiceBusConnection serviceBusConnection, string entityPath, Azure.Messaging.ServiceBus.ServiceBusReceiveMode receiveMode) { - this.sessionClient = new Microsoft.Azure.ServiceBus.SessionClient(serviceBusConnection, entityPath, receiveMode); - } - - public SessionClient(Microsoft.Azure.ServiceBus.SessionClient sessionClient) - { - this.sessionClient = sessionClient; - } - - public static implicit operator SessionClient(Microsoft.Azure.ServiceBus.SessionClient sc) - { - return new SessionClient(sc); - } - - public async Task CloseAsync() - { - await this.sessionClient.CloseAsync(); + this.entityPath = entityPath; + this.receiveMode = receiveMode; + this.serviceBusClient = serviceBusConnection.GetSendViaEntityScopedClient(entityPath); } public async Task AcceptMessageSessionAsync(TimeSpan operationTimeout) { try { - return new IMessageSession(await this.sessionClient.AcceptMessageSessionAsync(operationTimeout)); + var options = new Azure.Messaging.ServiceBus.ServiceBusSessionReceiverOptions + { + ReceiveMode = receiveMode, + }; + return new IMessageSession(await this.serviceBusClient.AcceptNextSessionAsync(entityPath, options)); } - catch (Microsoft.Azure.ServiceBus.ServiceBusTimeoutException) + catch (Azure.Messaging.ServiceBus.ServiceBusException e) when (e.Reason.Equals(Azure.Messaging.ServiceBus.ServiceBusFailureReason.ServiceTimeout)) { return null; } } + public Task CloseAsync() + { + return Task.CompletedTask; + } + #else public class SessionClient : QueueClient { diff --git a/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs b/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs index aa0b540e8..cc9472a1a 100644 --- a/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs +++ b/src/DurableTask.ServiceBus/Common/ServiceBusUtils.cs @@ -28,9 +28,6 @@ namespace DurableTask.ServiceBus.Common.Abstraction using DurableTask.Core.Tracking; using DurableTask.ServiceBus.Settings; using Newtonsoft.Json; -#if NETSTANDARD2_0 - using Microsoft.Azure.ServiceBus.InteropExtensions; -#endif internal static class ServiceBusUtils { @@ -60,7 +57,7 @@ public static async Task GetBrokeredMessageFromObjectAsync( #if NETSTANDARD2_0 using (var ms = new MemoryStream()) { - var serialiser = (XmlObjectSerializer)typeof(DataContractBinarySerializer<>) + var serialiser = (XmlObjectSerializer)typeof(DataContractSerializer) .MakeGenericType(serializableObject.GetType()) .GetField("Instance") ?.GetValue(null); @@ -216,8 +213,9 @@ public static async Task GetObjectFromBrokeredMessageAsync(Message message { // no compression, legacy style #if NETSTANDARD2_0 + var dataContractSerializer = new DataContractSerializer(typeof(T)); using (var ms = new MemoryStream(message.Body)) - deserializedObject = (T)DataContractBinarySerializer.Instance.ReadObject(ms); + deserializedObject = (T)dataContractSerializer.ReadObject(ms); #else deserializedObject = message.GetBody(); #endif diff --git a/src/DurableTask.ServiceBus/DurableTask.ServiceBus.csproj b/src/DurableTask.ServiceBus/DurableTask.ServiceBus.csproj index 397981d95..93096f69f 100644 --- a/src/DurableTask.ServiceBus/DurableTask.ServiceBus.csproj +++ b/src/DurableTask.ServiceBus/DurableTask.ServiceBus.csproj @@ -8,8 +8,8 @@ - 2 - 7 + 3 + 0 0 $(MajorVersion).$(MinorVersion).$(PatchVersion) @@ -29,7 +29,6 @@ - @@ -37,7 +36,7 @@ - + diff --git a/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs b/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs index 907054c60..3d49bb749 100644 --- a/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs +++ b/src/DurableTask.ServiceBus/ServiceBusOrchestrationService.cs @@ -35,21 +35,19 @@ namespace DurableTask.ServiceBus using DurableTask.ServiceBus.Tracking; using Message = DurableTask.ServiceBus.Common.Abstraction.Message; using IMessageSession = DurableTask.ServiceBus.Common.Abstraction.IMessageSession; - using RetryPolicy = DurableTask.ServiceBus.Common.Abstraction.RetryPolicy; using MessageSender = DurableTask.ServiceBus.Common.Abstraction.MessageSender; using MessageReceiver = DurableTask.ServiceBus.Common.Abstraction.MessageReceiver; using QueueClient = DurableTask.ServiceBus.Common.Abstraction.QueueClient; using SessionClient = DurableTask.ServiceBus.Common.Abstraction.SessionClient; using ServiceBusConnection = DurableTask.ServiceBus.Common.Abstraction.ServiceBusConnection; - using TokenProvider = DurableTask.ServiceBus.Common.Abstraction.TokenProvider; using ManagementClient = DurableTask.ServiceBus.Common.Abstraction.ManagementClient; using ServiceBusConnectionStringBuilder = DurableTask.ServiceBus.Common.Abstraction.ServiceBusConnectionStringBuilder; #if NETSTANDARD2_0 - using Microsoft.Azure.ServiceBus; - using Microsoft.Azure.ServiceBus.Management; - using Microsoft.Azure.ServiceBus.Primitives; + using Azure.Core; + using ReceiveMode = Azure.Messaging.ServiceBus.ServiceBusReceiveMode; #else using Microsoft.ServiceBus.Messaging; + using RetryPolicy = DurableTask.ServiceBus.Common.Abstraction.RetryPolicy; #endif /// @@ -92,7 +90,7 @@ public class ServiceBusOrchestrationService : IOrchestrationService, IOrchestrat readonly string hubName; MessageSender orchestratorSender; - readonly MessageSender orchestrationBatchMessageSender; + readonly MessageSender orchestratorBatchMessageSender; QueueClient orchestratorQueueClient; MessageSender workerSender; MessageSender trackingSender; @@ -117,20 +115,20 @@ public class ServiceBusOrchestrationService : IOrchestrationService, IOrchestrat /// Create a new ServiceBusOrchestrationService to the given service bus namespace and hub name /// /// Service Bus namespace host name - /// Service Bus authentication token provider + /// Service Bus authentication token credential /// Hub name to use with the Service Bus namespace /// Instance store Provider, where state and history messages will be stored /// Blob store Provider, where oversized messages and sessions will be stored /// Settings object for service and client public ServiceBusOrchestrationService( string namespaceHostName, - ITokenProvider tokenProvider, + TokenCredential tokenCredential, string hubName, IOrchestrationServiceInstanceStore instanceStore, IOrchestrationServiceBlobStore blobStore, ServiceBusOrchestrationServiceSettings settings) : this( - ServiceBusConnectionSettings.Create(namespaceHostName, tokenProvider), + ServiceBusConnectionSettings.Create(namespaceHostName, tokenCredential), hubName, instanceStore, blobStore, @@ -189,19 +187,21 @@ public ServiceBusOrchestrationService( if (!string.IsNullOrEmpty(connectionSettings.ConnectionString)) { var sbConnectionStringBuilder = new ServiceBusConnectionStringBuilder(connectionSettings.ConnectionString); +#if NETSTANDARD2_0 + this.serviceBusConnection = new ServiceBusConnection(sbConnectionStringBuilder); +#else this.serviceBusConnection = new ServiceBusConnection(sbConnectionStringBuilder) { TokenProvider = TokenProvider.CreateSharedAccessSignatureTokenProvider(sbConnectionStringBuilder.SasKeyName, - sbConnectionStringBuilder.SasKey, ServiceBusUtils.TokenTimeToLive) + sbConnectionStringBuilder.SasKey, ServiceBusUtils.TokenTimeToLive), }; +#endif } #if NETSTANDARD2_0 - else if (connectionSettings.Endpoint != null && connectionSettings.TokenProvider != null) + else if (connectionSettings.Endpoint != null && connectionSettings.TokenCredential != null) { - this.serviceBusConnection = new ServiceBusConnection(connectionSettings.Endpoint.ToString(), connectionSettings.TransportType) - { - TokenProvider = connectionSettings.TokenProvider - }; + + this.serviceBusConnection = new ServiceBusConnection(connectionSettings.Endpoint.Host, connectionSettings.TransportType, connectionSettings.TokenCredential); } #endif else @@ -210,7 +210,7 @@ public ServiceBusOrchestrationService( } this.Settings = settings ?? new ServiceBusOrchestrationServiceSettings(); - this.orchestrationBatchMessageSender = new MessageSender(this.serviceBusConnection, this.orchestratorEntityName); + this.orchestratorBatchMessageSender = new MessageSender(this.serviceBusConnection, this.orchestratorEntityName); this.BlobStore = blobStore; @@ -248,7 +248,11 @@ public async Task StartAsync() this.orchestratorSender = new MessageSender(this.serviceBusConnection, this.orchestratorEntityName, this.workerEntityName); this.workerSender = new MessageSender(this.serviceBusConnection, this.workerEntityName, this.orchestratorEntityName); this.trackingSender = new MessageSender(this.serviceBusConnection, this.trackingEntityName, this.orchestratorEntityName); +#if !NETSTANDARD2_0 this.orchestratorQueueClient = new QueueClient(this.serviceBusConnection, this.orchestratorEntityName, ReceiveMode.PeekLock, RetryPolicy.Default); +#else + this.orchestratorQueueClient = new QueueClient(this.serviceBusConnection, this.orchestratorEntityName); +#endif this.workerReceiver = new MessageReceiver(serviceBusConnection, this.workerEntityName); this.orchestratorSessionClient = new SessionClient(serviceBusConnection, this.orchestratorEntityName, ReceiveMode.PeekLock); this.trackingClient = new SessionClient(serviceBusConnection, this.trackingEntityName, ReceiveMode.PeekLock); @@ -288,7 +292,7 @@ public async Task StopAsync(bool isForced) await Task.WhenAll( this.workerSender.CloseAsync(), this.orchestratorSender.CloseAsync(), - this.orchestrationBatchMessageSender?.CloseAsync(), + this.orchestratorBatchMessageSender?.CloseAsync(), this.trackingSender.CloseAsync(), this.orchestratorSessionClient.CloseAsync(), this.trackingClient.CloseAsync(), @@ -396,7 +400,7 @@ public async Task HubExistsAsync() { ManagementClient managementClient = this.CreateManagementClient(); - IEnumerable queueDescriptions = (await managementClient.GetQueuesAsync()).Where(x => x.Path.StartsWith(this.hubName)).ToList(); + var queueDescriptions = (await managementClient.GetQueuesAsync()).Where(x => x.Path.StartsWith(this.hubName)).ToList(); return queueDescriptions.Any(q => string.Equals(q.Path, this.orchestratorEntityName)) && queueDescriptions.Any(q => string.Equals(q.Path, this.workerEntityName)) @@ -448,7 +452,7 @@ internal async Task> GetHubQueueMaxDeliveryCountsAsync() var result = new Dictionary(3); - IEnumerable queues = + var queues = (await managementClient.GetQueuesAsync()).Where(x => x.Path.StartsWith(this.hubName)).ToList(); result.Add("TaskOrchestration", queues.Single(q => string.Equals(q.Path, this.orchestratorEntityName))?.MaxDeliveryCount ?? -1); @@ -851,7 +855,7 @@ public async Task CompleteTaskOrchestrationWorkItemAsync( return $"Completing orchestration messages sequence and lock tokens: {allIds}"; }); - await session.CompleteAsync(sessionState.LockTokens.Keys); + await session.CompleteAsync(sessionState.LockTokens.Values); this.ServiceStats.OrchestrationDispatcherStats.SessionBatchesCompleted.Increment(); ts.Complete(); } @@ -891,9 +895,9 @@ public async Task AbandonTaskOrchestrationWorkItemAsync(TaskOrchestrationWorkIte } TraceHelper.TraceSession(TraceEventType.Error, "ServiceBusOrchestrationService-AbandonTaskOrchestrationWorkItem", workItem.InstanceId, "Abandoning {0} messages due to work item abort", sessionState.LockTokens.Keys.Count()); - foreach (string lockToken in sessionState.LockTokens.Keys) + foreach (var message in sessionState.LockTokens.Values) { - await sessionState.Session.AbandonAsync(lockToken); + await sessionState.Session.AbandonAsync(message); } try @@ -1039,7 +1043,7 @@ public async Task CompleteTaskActivityWorkItemAsync(TaskActivityWorkItem workIte Transaction.Current.TransactionInformation.LocalIdentifier } - message sequence and lock token: [SEQ: {originalMessage.SystemProperties.SequenceNumber} LT: {originalMessage.SystemProperties.LockToken}]"); - await this.workerReceiver.CompleteAsync(originalMessage.SystemProperties.LockToken); + await this.workerReceiver.CompleteAsync(originalMessage); await this.orchestratorSender.SendAsync(brokeredResponseMessage); ts.Complete(); this.ServiceStats.ActivityDispatcherStats.SessionBatchesCompleted.Increment(); @@ -1059,7 +1063,7 @@ public Task AbandonTaskActivityWorkItemAsync(TaskActivityWorkItem workItem) return message == null ? Task.FromResult(null) - : this.workerReceiver.AbandonAsync(message.SystemProperties.LockToken); + : this.workerReceiver.AbandonAsync(message); } /// @@ -1171,7 +1175,7 @@ public async Task SendTaskOrchestrationMessageBatchAsync(params TaskMessage[] me } Message[] brokeredMessages = await Task.WhenAll(tasks); - await this.orchestrationBatchMessageSender.SendAsync(brokeredMessages).ConfigureAwait(false); + await this.orchestratorBatchMessageSender.SendAsync(brokeredMessages).ConfigureAwait(false); } async Task GetBrokeredMessageAsync(TaskMessage message) @@ -1330,7 +1334,7 @@ static bool IsTransientException(Exception exception) { // TODO : Once we change the exception model, check for inner exception #if NETSTANDARD2_0 - return (exception as ServiceBusException)?.IsTransient ?? false; + return (exception as Azure.Messaging.ServiceBus.ServiceBusException)?.IsTransient ?? false; #else return (exception as MessagingException)?.IsTransient ?? false; #endif @@ -1511,7 +1515,7 @@ async Task ProcessTrackingWorkItemAsync(TrackingWorkItem workItem) } // Cleanup our session - await sessionState.Session.CompleteAsync(sessionState.LockTokens.Keys); + await sessionState.Session.CompleteAsync(sessionState.LockTokens.Values); await sessionState.Session.CloseAsync(); } @@ -1681,11 +1685,19 @@ await Utils.ExecuteWithRetries(async () => { await managementClient.DeleteQueueAsync(path); } +#if !NETSTANDARD2_0 catch (MessagingEntityAlreadyExistsException) +#else + catch (Azure.Messaging.ServiceBus.ServiceBusException e) when (e.Reason.Equals(Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityAlreadyExists)) +#endif { await Task.FromResult(0); } +#if !NETSTANDARD2_0 catch (MessagingEntityNotFoundException) +#else + catch (Azure.Messaging.ServiceBus.ServiceBusException e) when (e.Reason.Equals(Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityNotFound)) +#endif { await Task.FromResult(0); } @@ -1706,7 +1718,11 @@ await Utils.ExecuteWithRetries(async () => { await CreateQueueAsync(managementClient, path, requiresSessions, requiresDuplicateDetection, maxDeliveryCount, maxSizeInMegabytes); } +#if !NETSTANDARD2_0 catch (MessagingEntityAlreadyExistsException) +#else + catch (Azure.Messaging.ServiceBus.ServiceBusException e) when (e.Reason.Equals(Azure.Messaging.ServiceBus.ServiceBusFailureReason.MessagingEntityAlreadyExists)) +#endif { await Task.FromResult(0); } @@ -1740,17 +1756,17 @@ async Task CreateQueueAsync( throw new ArgumentException($"The specified value {maxSizeInMegabytes} is invalid for the maximum queue size in megabytes.\r\nIt must be one of the following values:\r\n{string.Join(";", ValidQueueSizes)}", nameof(maxSizeInMegabytes)); } +#if NETSTANDARD2_0 + var description = new Azure.Messaging.ServiceBus.Administration.CreateQueueOptions(path) +#else var description = new QueueDescription(path) +#endif { RequiresSession = requiresSessions, MaxDeliveryCount = maxDeliveryCount, RequiresDuplicateDetection = requiresDuplicateDetection, DuplicateDetectionHistoryTimeWindow = TimeSpan.FromHours(DuplicateDetectionWindowInHours), -#if NETSTANDARD2_0 - MaxSizeInMB = maxSizeInMegabytes -#else MaxSizeInMegabytes = maxSizeInMegabytes -#endif }; await managementClient.CreateQueueAsync(description); @@ -1790,9 +1806,9 @@ ManagementClient CreateManagementClient() return new ManagementClient(this.connectionSettings.ConnectionString); } #if NETSTANDARD2_0 - else if (connectionSettings.Endpoint != null && connectionSettings.TokenProvider != null) + else if (connectionSettings.Endpoint != null && connectionSettings.TokenCredential != null) { - return new ManagementClient(connectionSettings.Endpoint.ToString(), connectionSettings.TokenProvider); + return new ManagementClient(connectionSettings.Endpoint.Host, connectionSettings.TokenCredential); } #endif else diff --git a/src/DurableTask.ServiceBus/Settings/ServiceBusConnectionSettings.cs b/src/DurableTask.ServiceBus/Settings/ServiceBusConnectionSettings.cs index c87007656..075207532 100644 --- a/src/DurableTask.ServiceBus/Settings/ServiceBusConnectionSettings.cs +++ b/src/DurableTask.ServiceBus/Settings/ServiceBusConnectionSettings.cs @@ -14,8 +14,7 @@ namespace DurableTask.ServiceBus.Settings { #if NETSTANDARD2_0 - using Microsoft.Azure.ServiceBus; - using Microsoft.Azure.ServiceBus.Primitives; + using Azure.Core; #endif using System; @@ -43,15 +42,15 @@ public static ServiceBusConnectionSettings Create(string connectionString) /// Creates an instance of /// /// Service Bus namespace host name - /// Service Bus authentication token provider + /// Service Bus authentication token credential /// Service Bus messaging protocol /// - public static ServiceBusConnectionSettings Create(string namespaceHostName, ITokenProvider tokenProvider, TransportType transportType = TransportType.Amqp) + public static ServiceBusConnectionSettings Create(string namespaceHostName, TokenCredential tokenCredential, Azure.Messaging.ServiceBus.ServiceBusTransportType transportType = Azure.Messaging.ServiceBus.ServiceBusTransportType.AmqpTcp) { return new ServiceBusConnectionSettings { Endpoint = new Uri($"sb://{namespaceHostName}/"), - TokenProvider = tokenProvider, + TokenCredential = tokenCredential, TransportType = transportType }; } @@ -60,15 +59,15 @@ public static ServiceBusConnectionSettings Create(string namespaceHostName, ITok /// Creates an instance of /// /// Service Bus endpoint - /// Service Bus authentication token provider + /// Service Bus authentication token credential /// Service Bus messaging protocol /// - public static ServiceBusConnectionSettings Create(Uri serviceBusEndpoint, ITokenProvider tokenProvider, TransportType transportType = TransportType.Amqp) + public static ServiceBusConnectionSettings Create(Uri serviceBusEndpoint, TokenCredential tokenCredential, Azure.Messaging.ServiceBus.ServiceBusTransportType transportType = Azure.Messaging.ServiceBus.ServiceBusTransportType.AmqpTcp) { return new ServiceBusConnectionSettings { Endpoint = serviceBusEndpoint, - TokenProvider = tokenProvider, + TokenCredential = tokenCredential, TransportType = transportType }; } @@ -92,14 +91,14 @@ private ServiceBusConnectionSettings() public Uri Endpoint { get; private set; } /// - /// Service Bus authentication token provider + /// Service Bus messaging protocol /// - public ITokenProvider TokenProvider { get; private set; } + public Azure.Messaging.ServiceBus.ServiceBusTransportType TransportType { get; private set; } /// - /// Service Bus messaging protocol + /// Azure.Identity TokenCredential used to authenticate with the service bus /// - public TransportType TransportType { get; private set; } + public TokenCredential TokenCredential { get; private set; } #endif } diff --git a/test/DurableTask.AzureServiceFabric.Integration.Tests/DurableTask.AzureServiceFabric.Integration.Tests.csproj b/test/DurableTask.AzureServiceFabric.Integration.Tests/DurableTask.AzureServiceFabric.Integration.Tests.csproj index a8198945e..f23978e45 100644 --- a/test/DurableTask.AzureServiceFabric.Integration.Tests/DurableTask.AzureServiceFabric.Integration.Tests.csproj +++ b/test/DurableTask.AzureServiceFabric.Integration.Tests/DurableTask.AzureServiceFabric.Integration.Tests.csproj @@ -8,10 +8,9 @@ - - - - + + + diff --git a/test/DurableTask.AzureServiceFabric.Tests/DurableTask.AzureServiceFabric.Tests.csproj b/test/DurableTask.AzureServiceFabric.Tests/DurableTask.AzureServiceFabric.Tests.csproj index ee2a5ceb0..276f1d70a 100644 --- a/test/DurableTask.AzureServiceFabric.Tests/DurableTask.AzureServiceFabric.Tests.csproj +++ b/test/DurableTask.AzureServiceFabric.Tests/DurableTask.AzureServiceFabric.Tests.csproj @@ -7,10 +7,9 @@ - - - - + + + diff --git a/test/DurableTask.AzureStorage.Tests/DurableTask.AzureStorage.Tests.csproj b/test/DurableTask.AzureStorage.Tests/DurableTask.AzureStorage.Tests.csproj index 8bcf2d2a4..0fd66ca7f 100644 --- a/test/DurableTask.AzureStorage.Tests/DurableTask.AzureStorage.Tests.csproj +++ b/test/DurableTask.AzureStorage.Tests/DurableTask.AzureStorage.Tests.csproj @@ -9,7 +9,6 @@ - @@ -24,9 +23,9 @@ - - - + + + diff --git a/test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj b/test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj index 9d2e73bc1..8796c5fe8 100644 --- a/test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj +++ b/test/DurableTask.Core.Tests/DurableTask.Core.Tests.csproj @@ -10,15 +10,14 @@ - - - - - + + + + diff --git a/test/DurableTask.Core.Tests/ExceptionHandlingIntegrationTests.cs b/test/DurableTask.Core.Tests/ExceptionHandlingIntegrationTests.cs index 9e8042254..ed41ea931 100644 --- a/test/DurableTask.Core.Tests/ExceptionHandlingIntegrationTests.cs +++ b/test/DurableTask.Core.Tests/ExceptionHandlingIntegrationTests.cs @@ -15,6 +15,7 @@ namespace DurableTask.Core.Tests { using System; using System.Diagnostics; + using System.Runtime.Serialization; using System.Threading.Tasks; using DurableTask.Core.Exceptions; using DurableTask.Emulator; @@ -148,6 +149,22 @@ await this.worker } } + [TestMethod] + public void TaskFailureOnNullContextTaskActivity() + { + TaskActivity activity = new ThrowInvalidOperationExceptionAsync(); + string input = JsonConvert.SerializeObject(new string[] { "test" }); + + // Pass a null context to check that it doesn't affect error handling. + Task task = activity.RunAsync(null, input); + + Assert.IsTrue(task.IsFaulted); + Assert.IsNotNull(task.Exception); + Assert.IsNotNull(task.Exception?.InnerException); + Assert.IsInstanceOfType(task.Exception?.InnerException, typeof(TaskFailureException)); + Assert.AreEqual("This is a test exception", task.Exception?.InnerException?.Message); + } + class ExceptionHandlingOrchestration : TaskOrchestration { public override async Task RunTask(OrchestrationContext context, string input) @@ -187,7 +204,10 @@ await context.ScheduleWithRetry( tfe.FailureDetails.ErrorMessage == "This is a test exception" && tfe.FailureDetails.StackTrace!.Contains(typeof(ThrowInvalidOperationException).Name) && tfe.FailureDetails.IsCausedBy() && - tfe.FailureDetails.IsCausedBy()) // check that base types work too + tfe.FailureDetails.IsCausedBy() && + tfe.FailureDetails.InnerFailure != null && + tfe.FailureDetails.InnerFailure.IsCausedBy() && + tfe.FailureDetails.InnerFailure.ErrorMessage == "And this is its custom inner exception") { // Stop retrying return false; @@ -219,7 +239,31 @@ class ThrowInvalidOperationException : TaskActivity { protected override string Execute(TaskContext context, string input) { - throw new InvalidOperationException("This is a test exception"); + throw new InvalidOperationException("This is a test exception", + new CustomException("And this is its custom inner exception")); + } + } + + class ThrowInvalidOperationExceptionAsync : AsyncTaskActivity + { + protected override Task ExecuteAsync(TaskContext context, string input) + { + throw new InvalidOperationException("This is a test exception", + new CustomException("And this is its custom inner exception")); + } + } + + [Serializable] + class CustomException : Exception + { + public CustomException(string message) + : base(message) + { + } + + protected CustomException(SerializationInfo info, StreamingContext context) + : base(info, context) + { } } } diff --git a/test/DurableTask.Core.Tests/MessageSorterTests.cs b/test/DurableTask.Core.Tests/MessageSorterTests.cs new file mode 100644 index 000000000..13bb90170 --- /dev/null +++ b/test/DurableTask.Core.Tests/MessageSorterTests.cs @@ -0,0 +1,340 @@ +// --------------------------------------------------------------- +// Copyright (c) Microsoft Corporation. All rights reserved. +// --------------------------------------------------------------- + +namespace DurableTask.Core.Tests +{ + using System; + using System.Collections.Generic; + using System.IO; + using System.Linq; + using System.Threading.Tasks; + using DurableTask.Core.Entities; + using DurableTask.Core.Entities.EventFormat; + using Microsoft.VisualStudio.TestTools.UnitTesting; + + [TestClass] + public class MessageSorterTests + { + private static readonly TimeSpan ReorderWindow = TimeSpan.FromMinutes(30); + + [TestMethod] + public void SimpleInOrder() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + + var message1 = Send(senderId, receiverId, "1", senderSorter, DateTime.UtcNow); + var message2 = Send(senderId, receiverId, "2", senderSorter, DateTime.UtcNow); + var message3 = Send(senderId, receiverId, "3", senderSorter, DateTime.UtcNow); + + List batch; + MessageSorter receiverSorter = new MessageSorter(); + + // delivering the sequence in order produces 1 message each time + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("1"); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("2"); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("3"); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + [TestMethod] + public void WackySystemClock() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + + // simulate system clock that goes backwards - mechanism should still guarantee monotonicitty + var message1 = Send(senderId, receiverId, "1", senderSorter, DateTime.UtcNow); + var message2 = Send(senderId, receiverId, "2", senderSorter, DateTime.UtcNow - TimeSpan.FromSeconds(1)); + var message3 = Send(senderId, receiverId, "3", senderSorter, DateTime.UtcNow - TimeSpan.FromSeconds(2)); + + List batch; + MessageSorter receiverSorter = new MessageSorter(); + + // delivering the sequence in order produces 1 message each time + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("1"); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("2"); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("3"); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + [TestMethod] + public void DelayedElement() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + + var message1 = Send(senderId, receiverId, "1", senderSorter, DateTime.UtcNow); + var message2 = Send(senderId, receiverId, "2", senderSorter, DateTime.UtcNow); + var message3 = Send(senderId, receiverId, "3", senderSorter, DateTime.UtcNow); + + List batch; + MessageSorter receiverSorter; + + // delivering first message last delays all messages until getting the first one + receiverSorter = new MessageSorter(); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Collection( + batch, + first => first.Input.Equals("1"), + second => second.Input.Equals("2"), + third => third.Input.Equals("3")); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + [TestMethod] + public void NoFilteringOrSortingPastReorderWindow() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + var now = DateTime.UtcNow; + + // last message is sent after an interval exceeding the reorder window + var message1 = Send(senderId, receiverId, "1", senderSorter, now); + var message2 = Send(senderId, receiverId, "2", senderSorter, now + TimeSpan.FromTicks(1)); + var message3 = Send(senderId, receiverId, "3", senderSorter, now + TimeSpan.FromTicks(2) + ReorderWindow); + + List batch; + MessageSorter receiverSorter = new MessageSorter(); + + // delivering the sequence in order produces 1 message each time + receiverSorter = new MessageSorter(); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("1"); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("2"); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("3"); + + // duplicates are not filtered or sorted, but simply passed through, because we are past the reorder window + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("2"); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("1"); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + [TestMethod] + public void DuplicatedElements() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + + var message1 = Send(senderId, receiverId, "1", senderSorter, DateTime.UtcNow); + var message2 = Send(senderId, receiverId, "2", senderSorter, DateTime.UtcNow); + var message3 = Send(senderId, receiverId, "3", senderSorter, DateTime.UtcNow); + + List batch; + MessageSorter receiverSorter; + + // delivering first message last delays all messages until getting the first one + receiverSorter = new MessageSorter(); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Collection( + batch, + first => first.Input.Equals("1"), + second => second.Input.Equals("2")); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Single(batch).Input.Equals("3"); + batch = receiverSorter.ReceiveInOrder(message3, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message2, ReorderWindow).ToList(); + Assert.That.Empty(batch); + batch = receiverSorter.ReceiveInOrder(message1, ReorderWindow).ToList(); + Assert.That.Empty(batch); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + [TestMethod] + public void RandomShuffleAndDuplication() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + var receiverSorter = new MessageSorter(); + + var messageCount = 100; + var duplicateCount = 100; + + // create a ordered sequence of messages + var messages = new List(); + for (int i = 0; i < messageCount; i++) + { + messages.Add(Send(senderId, receiverId, i.ToString(), senderSorter, DateTime.UtcNow)); + } + + // add some random duplicates + var random = new Random(0); + for (int i = 0; i < duplicateCount; i++) + { + messages.Add(messages[random.Next(messageCount)]); + } + + // shuffle the messages + Shuffle(messages, random); + + // deliver all the messages + var deliveredMessages = new List(); + + foreach (var msg in messages) + { + foreach (var deliveredMessage in receiverSorter.ReceiveInOrder(msg, ReorderWindow)) + { + deliveredMessages.Add(deliveredMessage); + } + } + + // check that the delivered messages are the original sequence + Assert.AreEqual(messageCount, deliveredMessages.Count()); + for (int i = 0; i < messageCount; i++) + { + Assert.AreEqual(i.ToString(), deliveredMessages[i].Input); + } + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + /// + /// Tests that if messages get reordered beyond the supported reorder window, + /// we still deliver them all but they may now be out of order. + /// + [TestMethod] + public void RandomCollection() + { + var senderId = "A"; + var receiverId = "B"; + + var senderSorter = new MessageSorter(); + var receiverSorter = new MessageSorter(); + + var messageCount = 100; + + var random = new Random(0); + var now = DateTime.UtcNow; + + // create a ordered sequence of messages + var messages = new List(); + for (int i = 0; i < messageCount; i++) + { + messages.Add(Send(senderId, receiverId, i.ToString(), senderSorter, now + TimeSpan.FromSeconds(random.Next(5)), TimeSpan.FromSeconds(10))); + } + + // shuffle the messages + Shuffle(messages, random); + + // add a final message + messages.Add(Send(senderId, receiverId, (messageCount + 1).ToString(), senderSorter, now + TimeSpan.FromSeconds(1000), TimeSpan.FromSeconds(10))); + + // deliver all the messages + var deliveredMessages = new List(); + + for (int i = 0; i < messageCount; i++) + { + foreach (var deliveredMessage in receiverSorter.ReceiveInOrder(messages[i], TimeSpan.FromSeconds(10))) + { + deliveredMessages.Add(deliveredMessage); + } + + Assert.AreEqual(i + 1, deliveredMessages.Count + receiverSorter.NumberBufferedRequests); + } + + // receive the final messages + foreach (var deliveredMessage in receiverSorter.ReceiveInOrder(messages[messageCount], TimeSpan.FromSeconds(10))) + { + deliveredMessages.Add(deliveredMessage); + } + + // check that all messages were delivered + Assert.AreEqual(messageCount + 1, deliveredMessages.Count()); + + Assert.AreEqual(0, receiverSorter.NumberBufferedRequests); + } + + private static RequestMessage Send(string senderId, string receiverId, string input, MessageSorter sorter, DateTime now, TimeSpan? reorderWindow = null) + { + var msg = new RequestMessage() + { + Id = Guid.NewGuid(), + ParentInstanceId = senderId, + Input = input, + }; + sorter.LabelOutgoingMessage(msg, receiverId, now, reorderWindow.HasValue ? reorderWindow.Value : ReorderWindow); + return msg; + } + + private static void Shuffle(IList list, Random random) + { + int n = list.Count; + while (n > 1) + { + n--; + int k = random.Next(n + 1); + T value = list[k]; + list[k] = list[n]; + list[n] = value; + } + } + } + + internal static class AssertExtensions + { + public static void Empty(this Assert assert, IEnumerable collection) + { + Assert.AreEqual(0, collection.Count()); + } + + public static T Single(this Assert assert, IEnumerable collection) + { + var e = collection.GetEnumerator(); + Assert.IsTrue(e.MoveNext()); + T element = e.Current; + Assert.IsFalse(e.MoveNext()); + return element; + } + + public static void Collection(this Assert assert, IEnumerable collection, params Action[] elementInspectors) + { + var list = collection.ToList(); + Assert.AreEqual(elementInspectors.Length, list.Count); + for(int i = 0; i < elementInspectors.Length; i++) + { + elementInspectors[i](list[i]); + } + } + } +} \ No newline at end of file diff --git a/test/DurableTask.Emulator.Tests/EmulatorFunctionalTests.cs b/test/DurableTask.Emulator.Tests/EmulatorFunctionalTests.cs index 564de7a3a..0ba937bbd 100644 --- a/test/DurableTask.Emulator.Tests/EmulatorFunctionalTests.cs +++ b/test/DurableTask.Emulator.Tests/EmulatorFunctionalTests.cs @@ -314,11 +314,163 @@ await worker.AddTaskOrchestrations(orchestrationType) await worker.StopAsync(true); } + [TestMethod] + public async Task RegisterOrchestrationTasksFromInterface_InterfaceUsingInheritanceGenericsMethodOverloading_OrchestrationSuccess() + { + var orchestrationService = new LocalOrchestrationService(); + + var worker = new TaskHubWorker(orchestrationService); + await worker.AddTaskOrchestrations(typeof(TestInheritedTasksOrchestration)) + .AddTaskActivitiesFromInterfaceV2(new InheritedTestOrchestrationTasksA()) + .AddTaskActivitiesFromInterfaceV2(typeof(IInheritedTestOrchestrationTasksB), new InheritedTestOrchestrationTasksB()) + .StartAsync(); + + var client = new TaskHubClient(orchestrationService); + OrchestrationInstance id = await client.CreateOrchestrationInstanceAsync(typeof(TestInheritedTasksOrchestration), + null); + + var state = await client.WaitForOrchestrationAsync(id, TimeSpan.FromSeconds(30), CancellationToken.None); + Assert.AreEqual(OrchestrationStatus.Completed, state.OrchestrationStatus); + + Assert.AreEqual(InheritedTestOrchestrationTasksA.BumbleResult, TestInheritedTasksOrchestration.BumbleResultA); + Assert.AreEqual(InheritedTestOrchestrationTasksA.WobbleResult, TestInheritedTasksOrchestration.WobbleResultA); + Assert.AreEqual(InheritedTestOrchestrationTasksA.DerivedTaskResult, TestInheritedTasksOrchestration.DerivedTaskResultA); + Assert.AreEqual(InheritedTestOrchestrationTasksA.JuggleResult, TestInheritedTasksOrchestration.JuggleResultA); + + Assert.AreEqual(InheritedTestOrchestrationTasksB.BumbleResult, TestInheritedTasksOrchestration.BumbleResultB); + Assert.AreEqual(InheritedTestOrchestrationTasksB.WobbleResult, TestInheritedTasksOrchestration.WobbleResultB); + Assert.AreEqual(InheritedTestOrchestrationTasksB.OverloadedWobbleResult1, TestInheritedTasksOrchestration.OverloadedWobbleResult1B); + Assert.AreEqual(InheritedTestOrchestrationTasksB.OverloadedWobbleResult2, TestInheritedTasksOrchestration.OverloadedWobbleResult2B); + Assert.AreEqual(InheritedTestOrchestrationTasksB.JuggleResult, TestInheritedTasksOrchestration.JuggleResultB); + } + private static void AssertTagsEqual(IDictionary expectedTags, IDictionary actualTags) { Assert.IsNotNull(actualTags); Assert.AreEqual(expectedTags.Count, actualTags.Count); Assert.IsTrue(expectedTags.All(tag => actualTags.TryGetValue(tag.Key, out var value) && value == tag.Value)); } + + // base interface without generic type parameters + public interface IBaseTestOrchestrationTasks + { + Task Juggle(int toss, bool withFlair); + } + + // generic type base interface inheriting non-generic type with same name + public interface IBaseTestOrchestrationTasks : IBaseTestOrchestrationTasks + { + Task Bumble(TIn fumble, bool likeAKlutz); + Task Wobble(TIn jiggle, bool withGusto); + } + + // interface with derived task + public interface IInheritedTestOrchestrationTasksA : IBaseTestOrchestrationTasks + { + Task DerivedTask(int i); + } + + // interface with overloaded method + public interface IInheritedTestOrchestrationTasksB : IBaseTestOrchestrationTasks + { + // this method overloads methods from both inherited interface and this interface + Task Wobble(TIn name); + Task Wobble(string id, TIn subId); + } + + public class InheritedTestOrchestrationTasksA : IInheritedTestOrchestrationTasksA + { + public const string BumbleResult = nameof(Bumble) + "-A"; + public const string WobbleResult = nameof(Wobble) + "-A"; + public const string DerivedTaskResult = nameof(DerivedTask) + "-A"; + public const int JuggleResult = 419; + + public Task Bumble(string fumble, bool likeAKlutz) + { + return Task.FromResult(BumbleResult); + } + + public Task Wobble(string jiggle, bool withGusto) + { + return Task.FromResult(WobbleResult); + } + + public Task DerivedTask(int i) + { + return Task.FromResult(DerivedTaskResult); + } + + public Task Juggle(int toss, bool withFlair) + { + return Task.FromResult(JuggleResult); + } + } + + public class InheritedTestOrchestrationTasksB : IInheritedTestOrchestrationTasksB + { + public const string BumbleResult = nameof(Bumble) + "-B"; + public const string WobbleResult = nameof(Wobble) + "-B"; + public const string OverloadedWobbleResult1 = nameof(Wobble) + "-B-overloaded-1"; + public const string OverloadedWobbleResult2 = nameof(Wobble) + "-B-overloaded-2"; + public const int JuggleResult = 420; + + public Task Bumble(int fumble, bool likeAKlutz) + { + return Task.FromResult(BumbleResult); + } + + public Task Wobble(int jiggle, bool withGusto) + { + return Task.FromResult(WobbleResult); + } + + public Task Wobble(string id, int subId) + { + return Task.FromResult(OverloadedWobbleResult1); + } + + public Task Wobble(int id) + { + return Task.FromResult(OverloadedWobbleResult2); + } + + public Task Juggle(int toss, bool withFlair) + { + return Task.FromResult(JuggleResult); + } + } + + public class TestInheritedTasksOrchestration : TaskOrchestration + { + // HACK: This is just a hack to communicate result of orchestration back to test + public static string BumbleResultA; + public static string WobbleResultA; + public static string DerivedTaskResultA; + public static int JuggleResultA; + public static string BumbleResultB; + public static string WobbleResultB; + public static string OverloadedWobbleResult1B; + public static string OverloadedWobbleResult2B; + public static int JuggleResultB; + + public override async Task RunTask(OrchestrationContext context, string input) + { + var tasksA = context.CreateClientV2(); + var tasksB = context.CreateClientV2>(); + + BumbleResultA = await tasksA.Bumble(string.Empty, false); + WobbleResultA = await tasksA.Wobble(string.Empty, false); + DerivedTaskResultA = await tasksA.DerivedTask(0); + JuggleResultA = await tasksA.Juggle(1, true); + + BumbleResultB = await tasksB.Bumble(0, false); + WobbleResultB = await tasksB.Wobble(-1, false); + OverloadedWobbleResult1B = await tasksB.Wobble("a", 2); + OverloadedWobbleResult2B = await tasksB.Wobble(1); + JuggleResultB = await tasksB.Juggle(1, true); + + return string.Empty; + } + } } } \ No newline at end of file diff --git a/test/DurableTask.ServiceBus.Tests/DispatcherTests.cs b/test/DurableTask.ServiceBus.Tests/DispatcherTests.cs index 2bd870dea..fe9ad6e36 100644 --- a/test/DurableTask.ServiceBus.Tests/DispatcherTests.cs +++ b/test/DurableTask.ServiceBus.Tests/DispatcherTests.cs @@ -95,6 +95,9 @@ await this.taskHub.AddTaskOrchestrations(typeof (CompressionCompatTest)) Assert.AreEqual(OrchestrationStatus.Completed, state.OrchestrationStatus, TestHelpers.GetInstanceNotCompletedMessage(this.client, id, 60)); } +#if NETCOREAPP + [TestCategory("DisabledInCI")] +#endif [TestMethod] public async Task MessageCompressionToNoCompressionTest() { @@ -844,6 +847,9 @@ protected override string Execute(TaskContext context, string input) #region Compression Tests +#if NETCOREAPP + [TestCategory("DisabledInCI")] +#endif [TestMethod] public async Task CompressionToNoCompressionCompatTest() { diff --git a/test/DurableTask.ServiceBus.Tests/DurableTask.ServiceBus.Tests.csproj b/test/DurableTask.ServiceBus.Tests/DurableTask.ServiceBus.Tests.csproj index d85027fe2..a834f1308 100644 --- a/test/DurableTask.ServiceBus.Tests/DurableTask.ServiceBus.Tests.csproj +++ b/test/DurableTask.ServiceBus.Tests/DurableTask.ServiceBus.Tests.csproj @@ -22,7 +22,6 @@ - @@ -30,10 +29,11 @@ - - - + + + + diff --git a/test/DurableTask.ServiceBus.Tests/ErrorHandlingTests.cs b/test/DurableTask.ServiceBus.Tests/ErrorHandlingTests.cs index 76385ca5d..c74beeeae 100644 --- a/test/DurableTask.ServiceBus.Tests/ErrorHandlingTests.cs +++ b/test/DurableTask.ServiceBus.Tests/ErrorHandlingTests.cs @@ -426,6 +426,9 @@ await this.taskHub.AddTaskOrchestrations(new TestObjectCreator - - - - + + + + diff --git a/test/DurableTask.Test.Orchestrations/DurableTask.Test.Orchestrations.csproj b/test/DurableTask.Test.Orchestrations/DurableTask.Test.Orchestrations.csproj index e2e94505d..c390fe115 100644 --- a/test/DurableTask.Test.Orchestrations/DurableTask.Test.Orchestrations.csproj +++ b/test/DurableTask.Test.Orchestrations/DurableTask.Test.Orchestrations.csproj @@ -4,14 +4,6 @@ netstandard2.0;net462 - - - - - - - -