.NET: Workflow as Agent - unable to get any output from agent run.

[antero-lukkonen]

Hi,

Unable to get output from a Node when running a workflow as an Agent.

System under test:
A dual node Workflow: ChatForwardingExecutor -> CustomExecutor.

CustomExecutor is handling List and is returning List. The expected output is a hardcoded string.

Observations:
Both executors run when debugging with breakpoints.

Expected Behavior:
AgentRunResponse.Text is filled with the output of CustomExecutor.

Actual Behavior:
AgentRunResponse.Text is empty string.
Also no AgentRunUpdateEvent is raised when i run in streaming mode.

Started trying the Workflow as Agent example in git repo. In this case the english and french agents produced messages and they were visinle in AgentRunResponse.Text. The final AggeragatingExecutor output was not.

After simplifying this example i arrived at a minimal repro:

Packages:

  <ItemGroup>
    <PackageReference Include="Azure.AI.OpenAI" Version="2.8.0-beta.1" />
    <PackageReference Include="Azure.Identity" Version="1.18.0-beta.2" />
    <PackageReference Include="Microsoft.Agents.AI.OpenAI" Version="1.0.0-preview.251204.1" />
    <PackageReference Include="Microsoft.Agents.AI.Workflows" Version="1.0.0-preview.251112.1" />
    <PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="10.1.1-preview.1.25612.2" />
  </ItemGroup>


namespace WorkflowAsAgentTests
{
    using Microsoft.Agents.AI.Workflows;
    using WorkflowAsAnAgentSample;

    public class UnitTest1
    {
        [Fact]
        public async Task WorkflowAsAgentOutput()
        {
            var result = await WorkflowFactory
                .BuildWorkflow()
                .AsAgent()
                .RunAsync("Some Input");

            Assert.NotNull(result);
            Assert.Equal("Expected Output", result.Text);
        }
    }
}

namespace WorkflowAsAnAgentSample;

using Microsoft.Agents.AI.Workflows;
using Microsoft.Extensions.AI;

internal static class WorkflowFactory
{
    internal static Workflow BuildWorkflow()
    {
        var startExecutor = new ChatForwardingExecutor("Start");
        var customExecutor = new CustomExecutor();
        return new WorkflowBuilder(startExecutor)
            .AddEdge(startExecutor, customExecutor)
            .WithOutputFrom(customExecutor)
            .Build();
    }

    private sealed class CustomExecutor() :
        Executor<List<ChatMessage>>("CustomExecutor")
    {
        public override async ValueTask HandleAsync(List<ChatMessage> message, IWorkflowContext context, CancellationToken cancellationToken = default)
        {
// Tried different things, like SendMessageAsync, also returning, raising an event, nothing helped.
            await context.YieldOutputAsync(new ChatMessage(ChatRole.Assistant, "Expected Output"));
            await context.SendMessageAsync(new TurnToken());
        }
    }
}

[antero-lukkonen]

I got my executor output if i raised an agent update event just like the agent executor does.

If this is the expected usage, may i suggest that the docs would be explicit about it.

And maybe there is also an api improvement hiding.

I am sorry if i missed it and it is somewhere.

I was under the impression that withoutputfrom and yieldoutput should take care of all plumbing necessary.

This took quite a while to figure out.

[Dazfl]

@antero-lukkonen ... I have been having this exact same problem. I have some custom executors in a sequential workflow, but I cannot get any output. I know the workflow runs, but the final step doesn't produce any output.

Are you able to point to the code that shows how to raise an agent update event?

UPDATE: When I run the workflow as an agent, I still don't have any output, plus I don't have any token usage output

[Dazfl]

This may or may not be related, but after a bit more testing I have found that if my executors run agents as non-streaming (RunAsync()), I don't get an output. But if I run the agents as streaming (RunStreamingAsync()) I get an output and tokens. For example,

This doesn't work:

public sealed class EnglishTranslatorExecutor(IChatClient chatClient) : Executor<List<ChatMessage>>("EnglishTranslator"), IResettableExecutor
{
    public override async ValueTask HandleAsync(List<ChatMessage> messages, IWorkflowContext context, CancellationToken cancellationToken = default)
    {
        var englishAgent = chatClient
            .AsBuilder()
            .UseOpenTelemetry(configure: c => c.EnableSensitiveData = true)
            .BuildAIAgent(name: "EnglishAgent", description: "English Translator", instructions: "Translate the message to English.");

        var response = await englishAgent.RunAsync(messages.Last(), cancellationToken: cancellationToken).ConfigureAwait(false);

        var runResponse = response;
        messages.AddRange(runResponse.Messages);
        await context.AddEventAsync(new AgentRunResponseEvent("EnglishTranslator", runResponse), cancellationToken);
        await context.SendMessageAsync(messages, cancellationToken);
        await context.SendMessageAsync(new TurnToken(emitEvents: true), cancellationToken);
    }

    ValueTask IResettableExecutor.ResetAsync() => default;
}

This works:

public sealed class EnglishTranslatorExecutor(IChatClient chatClient) : Executor<List<ChatMessage>>("EnglishTranslator"), IResettableExecutor
{
    public override async ValueTask HandleAsync(List<ChatMessage> messages, IWorkflowContext context, CancellationToken cancellationToken = default)
    {
        var englishAgent = chatClient
            .AsBuilder()
            .UseOpenTelemetry(configure: c => c.EnableSensitiveData = true)
            .BuildAIAgent(name: "EnglishAgent", description: "English Translator", instructions: "Translate the message to English.");

        List<AgentRunResponseUpdate> updates = [];
        await foreach (var response in englishAgent.RunStreamingAsync(messages.Last(), cancellationToken: cancellationToken))
        {
            updates.Add(response);
            await context.AddEventAsync(new AgentRunUpdateEvent("EnglishTranslator", response), cancellationToken);
        }

        var runResponse = updates.ToAgentRunResponse();
        messages.AddRange(runResponse.Messages);
        await context.AddEventAsync(new AgentRunResponseEvent("EnglishTranslator", runResponse), cancellationToken);
        await context.SendMessageAsync(messages, cancellationToken);
        await context.SendMessageAsync(new TurnToken(emitEvents: true), cancellationToken);
    }

    ValueTask IResettableExecutor.ResetAsync() => default;
}

As best as I can tell, it has something to do with emitting a AgentRunUpdateEvent. This needs to be emitted for output to work.

[antero-lukkonen]

await context.AddEventAsync(new AgentRunUpdateEvent("EnglishTranslator", response), cancellationToken);

Made output work for me too.
Meaning, i have the .Text property of result filled with the string i emitted with the event.

Going forward as a workaround with an executor, similar to ChatForwardingExecutor that has a single purpose to take care of the output part of the agent protocol. Forwards messages and emits the event.

[antero-lukkonen]

Bringing out the behavior is very similar to #2957 .
I am also not able have a "silent" chatclient agent in the workflow. As soon as any executor rises an agenruntupdateevent it is surfaced.
Withoutputfrom seems to have no effect ( or i misunderstand the purpose of it )

[antero-lukkonen]

context.YieldOutputAsync and builder.WithOutputFrom(executorWithYield) together make sure that the yielded output is allowed to be emitted as a WorkflowOutputEvent by the workflow runner.

WorkflowThread.InvokeStageAsync converts workflow events to Agent events.

Currently it converts mostly all events into a AgentRunResponseUpdate events. Including the WorkflowOutputEvent.

AgentRunResponseUpdate events are not considered when the final aggregated Text is created, since these designate various kinds of events generally designated for Observability, Logging etc.

Adding a conversion for the WorkflowOutputEvent into AgentRunResponseUpdate has the effect of yielded output appearing in AgentRunResponse.Text.

So this is a hacky fix.

WorkflowThread.InvokeStageAsync:

default:
    if (evt is WorkflowOutputEvent wfoe)
    {
        yield return new AgentRunResponseUpdate(ChatRole.Assistant, [])
        {
            CreatedAt = DateTimeOffset.UtcNow,
            MessageId = Guid.NewGuid().ToString("N"),
            Role = ChatRole.Assistant,
            ResponseId = this.LastResponseId,
            RawRepresentation = wfoe,
            Contents = wfoe.Data is not null && wfoe.Is<string>()
                ? new List<AIContent> { new TextContent(wfoe.As<string>()) }
                : [],
        };
    }
    else
    {
        // Emit all other workflow events for observability (DevUI, logging, etc.)
        yield return new AgentRunResponseUpdate(ChatRole.Assistant, [])
        {
            CreatedAt = DateTimeOffset.UtcNow,
            MessageId = Guid.NewGuid().ToString("N"),
            Role = ChatRole.Assistant,
            ResponseId = this.LastResponseId,
            RawRepresentation = evt,
        };
    }
    break;

[antero-lukkonen]

I would be willing to do the work to fix this one.
But for that i would need to consult with someone authoritative, about what the real fix would be.
This feels more about agreeing on principles and philosophy behind treating workflows as agents, than a technical fix.
This area is also not covered with unit-tests as far as i saw.

My current conclusion is that the library works as designed. The python counterpart looks to behave the same way.
In python case, docs and examples explain the behavior and usage in more detail and cover this part.

The fix for the current state in .net would have to be in documentation and samples. Covering the case where a user would like to get workflow output from an executor that is not an agent.

This can be achieved basically in 2 ways that i found:

1. Inside an executor, raise the AgentRunResponseUpdate event. Works for streaming and non streaming automatically.
2. While processing events, check the RawRepresentation field and in case it is a WorkflowOutputEvent handle the Contents field as needed.

for example:

            var result = await eventStream 
                .Select(e => 
                    e.RawRepresentation is WorkflowOutputEvent 
                        ? (e.RawRepresentation as WorkflowOutputEvent)!.As<string>()
                        : e.Text)
                .Where(text => !string.IsNullOrEmpty(text))
                .AggregateAsync(string.Empty, (acc, text) => acc + text);