Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions python/observability-with-azure-monitor/.env.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# Azure Monitor / Application Insights
APPLICATIONINSIGHTS_CONNECTION_STRING=InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://...

# OpenAI / Azure OpenAI (the OpenAI Agents SDK reads these)
# For OpenAI:
OPENAI_API_KEY=<<YOUR_OPENAI_API_KEY>>
# For Azure OpenAI (set these instead of OPENAI_API_KEY):
# AZURE_OPENAI_API_KEY=
# AZURE_OPENAI_ENDPOINT=
# OPENAI_API_VERSION=2024-08-01-preview

# Service identification (optional; used as service.name attribute)
AGENT_SERVICE_NAME=sample-agent-azure-monitor

# REQUIRED: enable Agent 365 span emission. The SDK gates scope creation
# behind one of these flags; without either set to a truthy value, the
# scopes produce zero spans (silent failure mode).
ENABLE_OBSERVABILITY=true
6 changes: 6 additions & 0 deletions python/observability-with-azure-monitor/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.env
.venv/
__pycache__/
*.pyc
dist/
*.egg-info/
78 changes: 78 additions & 0 deletions python/observability-with-azure-monitor/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Observability — Agent 365 SDK alongside Azure Monitor

This sample shows how to add the [Microsoft Agent 365 Python SDK](https://github.com/microsoft/Agent365-python) to an app that **already** uses [Azure Monitor / Application Insights OpenTelemetry](https://learn.microsoft.com/azure/azure-monitor/app/opentelemetry-overview). After running this, both Azure Monitor and the Agent 365 backend receive your agent's spans.

> This is **not** a from-scratch tracing setup. For a full agent host with Microsoft 365 Agents SDK, see the [`python/openai/sample-agent`](../openai/sample-agent) sample.

Comment thread
juliomenendez marked this conversation as resolved.
## Demonstrates

- The recommended init order: existing OTel → Agent 365 `configure()` → OpenAI Agents SDK instrumentor.
- Auto-instrumentation via `microsoft-agents-a365-observability-extensions-openai` — no manual span code in the agent body.
- Both Azure Monitor and the Agent 365 backend receive every span produced by the agent.

## Prerequisites

- Python 3.11+
- An OpenAI or Azure OpenAI key
- An Application Insights resource (connection string)

## Setup

1. Copy the env template and fill in your values:

```bash
cp .env.template .env
```

The template includes `ENABLE_OBSERVABILITY=true` — leave this as-is. Without it, the SDK silently emits zero spans.

2. Create a virtualenv and install:

```bash
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e .
```

## Run

```bash
python main.py
```

Expected stdout: a one-line weather answer for Seattle.

## What to look for

In Azure Portal → your Application Insights resource → **Transaction search**, look for spans with the custom-dimension attribute `gen_ai.operation.name`. You should see three operation types across one turn:

- `invoke_agent` — the agent invocation (root span; display name `invoke_agent WeatherAgent`)
- `chat` — one or more LLM call spans (display name e.g. `chat gpt-4.1`; the auto-instrumentation extension uses lowercase `chat` per the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/))
- `execute_tool` — the `get_weather` tool span (display name `execute_tool get_weather`)

If you see those three operation types, the integration is working. The Agent 365 backend receives the same spans (configured via the stub token resolver — replace with a real one for production).

## Where the integration happens

`main.py` is organized into the following sections (Step 2b is a sub-step that must run after Step 2):

1. **Step 1 — Azure Monitor.** `configure_azure_monitor(...)` installs an OTel TracerProvider and the Azure Monitor exporter. This is the part of the file you'd already have in your real app.
2. **Step 2 — Agent 365 `configure()`.** Detects the TracerProvider set by Step 1 and adds its processors to it. Both backends now receive spans. Replace `_stub_token_resolver` with your production token resolver.
3. **Step 2b — `OpenAIAgentsTraceInstrumentor`.** Must run after `configure()`; the instrumentor raises `RuntimeError` otherwise. After this call, OpenAI Agents SDK spans flow through Agent 365's scope classes automatically.
4. **Step 3 — Build the agent.** Standard OpenAI Agents SDK code; no observability code needed (the instrumentor handles it).
5. **Step 4 — Run + flush.** `force_flush()` is critical — without it, batched spans may not export before the process exits.

To diff against your own app: copy Steps 1, 2, and 2b into the file where your app currently initializes Azure Monitor.

## Going further

- Integration patterns and pitfalls: [Integrating with existing OpenTelemetry](https://github.com/microsoft/Agent365-python/blob/main/docs/integrating-with-existing-opentelemetry.md) (in the SDK repo)
- Manual instrumentation example (no agent framework): [`python/observability-with-otlp`](../observability-with-otlp)

## Troubleshooting

- **Sample runs without errors but no spans appear** — most commonly `ENABLE_OBSERVABILITY` is not set to a truthy value. The SDK gates span creation behind this env var and produces zero spans silently when it's missing. The sample's `.env.template` includes it; if you assembled `.env` manually, add `ENABLE_OBSERVABILITY=true`.
- **`SystemExit: APPLICATIONINSIGHTS_CONNECTION_STRING is not set`** — set the env var via `.env`. The connection string is on your App Insights resource → **Overview** → **Connection String**.
- **No spans visible in App Insights** — wait 1–2 minutes for ingestion; confirm the connection string targets the right resource. If the agent ran successfully but spans never appear, temporarily add a `ConsoleSpanExporter` (see [the integration guide's verify recipe](https://github.com/microsoft/Agent365-python/blob/main/docs/integrating-with-existing-opentelemetry.md#verifying-the-integration)) to prove the SDK is producing them.
- **`SystemExit: Agent 365 observability configuration failed`** — check logs for the failing step (most often a missing or unreachable token resolver in production; the sample uses a stub).
- **OpenAI auth errors** — verify `OPENAI_API_KEY` (or `AZURE_OPENAI_*` variables) in `.env`. The OpenAI Agents SDK reads these directly.
109 changes: 109 additions & 0 deletions python/observability-with-azure-monitor/main.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,109 @@
# Copyright (c) Microsoft Corporation.
# Licensed under the MIT License.

"""Sample: Agent 365 SDK alongside an existing Azure Monitor OpenTelemetry setup.

Demonstrates the recommended initialization order:

1. Initialize your existing OTel stack first (Azure Monitor here).
2. Then call Agent 365 `configure()` — it detects the existing TracerProvider
and adds its processors to it. Both backends receive spans.
3. Then install the OpenAI Agents SDK instrumentor (it requires A365 to be
configured first). It auto-instruments your agent — no manual span code.

Run with: ``python main.py``
"""

import json
import os

from dotenv import load_dotenv

load_dotenv()

# ---------------------------------------------------------------------------
# Step 1 — Existing OTel setup (Azure Monitor / Application Insights).
# This is what an app already has in production today.
# ---------------------------------------------------------------------------
from azure.monitor.opentelemetry import configure_azure_monitor

_app_insights_conn = os.environ.get("APPLICATIONINSIGHTS_CONNECTION_STRING")
if not _app_insights_conn:
raise SystemExit(
"APPLICATIONINSIGHTS_CONNECTION_STRING is not set. "
"Copy .env.template to .env and fill in your Application Insights "
"connection string. See README.md for setup steps."
)
configure_azure_monitor(connection_string=_app_insights_conn)

# ---------------------------------------------------------------------------
# Step 2 — Agent 365 SDK `configure()`.
# Detects the TracerProvider set in Step 1 and adds its processors to it.
# Both Azure Monitor and the Agent 365 exporter now receive spans.
# ---------------------------------------------------------------------------
from microsoft_agents_a365.observability.core import configure


def _stub_token_resolver(agent_id: str, tenant_id: str) -> str | None:
# In a real app, return a bearer token for the Agent 365 backend.
# See the observability-core docs for the production pattern.
return "stub-token"


_configure_ok = configure(
service_name=os.environ.get("AGENT_SERVICE_NAME", "sample-agent-azure-monitor"),
service_namespace="agent365-samples",
token_resolver=_stub_token_resolver,
)
if not _configure_ok:
raise SystemExit(
"Agent 365 observability configuration failed. See logs for details."
)

# ---------------------------------------------------------------------------
# Step 2b — Install the OpenAI Agents SDK instrumentor.
# Must run AFTER `configure()` — the instrumentor raises RuntimeError otherwise.
# ---------------------------------------------------------------------------
from microsoft_agents_a365.observability.extensions.openai import (
OpenAIAgentsTraceInstrumentor,
)

OpenAIAgentsTraceInstrumentor().instrument()

# ---------------------------------------------------------------------------
# Step 3 — Build the tool-calling agent (auto-instrumented).
# ---------------------------------------------------------------------------
from agents import Agent, Runner, function_tool


@function_tool
def get_weather(city: str) -> str:
"""Return the current weather for ``city`` as a JSON string."""
return json.dumps({"city": city, "temperature_f": 72, "conditions": "sunny"})


agent = Agent(
name="WeatherAgent",
instructions=(
"You are a helpful assistant that answers weather questions "
"using the get_weather tool."
),
tools=[get_weather],
)

# ---------------------------------------------------------------------------
# Step 4 — Run a single turn and exit, flushing spans on the way out.
# ---------------------------------------------------------------------------
from opentelemetry import trace


def main() -> None:
result = Runner.run_sync(agent, "What's the weather in Seattle?")
print(result.final_output)
# Force span flush so both Azure Monitor and Agent 365 exporters drain
# before the process exits.
trace.get_tracer_provider().force_flush()


if __name__ == "__main__":
main()
20 changes: 20 additions & 0 deletions python/observability-with-azure-monitor/pyproject.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
[project]
name = "observability-with-azure-monitor"
version = "0.1.0"
description = "Sample: Agent 365 SDK with existing Azure Monitor OpenTelemetry"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
"openai-agents>=0.2.6",
"azure-monitor-opentelemetry>=1.6.0",
"microsoft-agents-a365-observability-core",
Comment thread
juliomenendez marked this conversation as resolved.
"microsoft-agents-a365-observability-extensions-openai",
"python-dotenv>=1.0.0",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["."]
22 changes: 22 additions & 0 deletions python/observability-with-langgraph/.env.template
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
# OpenAI / Azure OpenAI (langchain-openai reads these)
OPENAI_API_KEY=<<YOUR_OPENAI_API_KEY>>
# OPENAI_MODEL=gpt-4o-mini
# For Azure OpenAI, use AzureChatOpenAI() in main.py and set:
# AZURE_OPENAI_API_KEY=
# AZURE_OPENAI_ENDPOINT=
# OPENAI_API_VERSION=2024-08-01-preview

# OpenTelemetry service identification
OTEL_SERVICE_NAME=sample-agent-langgraph
AGENT_SERVICE_NAME=sample-agent-langgraph

# Optional: only used if you swap ConsoleSpanExporter for OTLPSpanExporter in main.py.
# Examples:
# - Local OTLP/gRPC collector: http://localhost:4317
# - Google Cloud Trace (per the reference guide): https://telemetry.googleapis.com:443/v1/traces
# OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# REQUIRED: enable Agent 365 span emission. The SDK gates scope creation
# behind one of these flags; without either set to a truthy value, the
# scopes produce zero spans (silent failure mode).
ENABLE_OBSERVABILITY=true
6 changes: 6 additions & 0 deletions python/observability-with-langgraph/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.env
.venv/
__pycache__/
*.pyc
dist/
*.egg-info/
95 changes: 95 additions & 0 deletions python/observability-with-langgraph/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Observability — Agent 365 SDK with LangGraph

This sample shows how to add the [Microsoft Agent 365 Python SDK](https://github.com/microsoft/Agent365-python) to a [LangGraph](https://langchain-ai.github.io/langgraph/) agent that **already** uses OpenTelemetry. After running this, both your existing exporter and the Agent 365 backend receive every span produced by the agent.

The structure mirrors Google Cloud's [LangGraph + OpenTelemetry reference sample](https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-langgraph): an OTel TracerProvider is set up first, an auto-instrumentor handles the LLM and tool spans, and a manual top-level span wraps `agent.invoke(...)` so the trace tree has a clear "agent run" root.

> This is **not** a from-scratch tracing setup. For a full agent host with Microsoft 365 Agents SDK, see the [`python/openai/sample-agent`](../openai/sample-agent) sample.

## Demonstrates

- The recommended init order: existing OTel → Agent 365 `configure()` → LangChain instrumentor.
- Auto-instrumentation via `microsoft-agents-a365-observability-extensions-langchain` — every LangChain LLM and tool callback emits an OTel span automatically; no per-call wrapping in the agent body.
- A manual `InvokeAgentScope` around `agent.invoke(...)` for the top-level `invoke_agent <agent_name>` span (the Agent 365 equivalent of Google's `tracer.start_as_current_span("invoke agent")`).
- Default `ConsoleSpanExporter` for zero external setup, with a one-line swap to OTLP/gRPC for real backends (including Google Cloud Trace).

## Prerequisites

- Python 3.11+
- An OpenAI or Azure OpenAI key

No collector or external service is required — the sample defaults to `ConsoleSpanExporter` so spans print to stdout.

## Setup

1. Copy the env template and fill in your values:

```bash
cp .env.template .env
```

The template includes `ENABLE_OBSERVABILITY=true` — leave this as-is. Without it, the SDK silently emits zero spans.

2. Create a virtualenv and install:

```bash
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -e .
```

## Run

```bash
python main.py
```

Expected output (truncated):

- A one-line weather answer for Seattle.
- Multiple JSON span dumps printed by `ConsoleSpanExporter`. Look for spans named `invoke_agent WeatherAgent`, `chat ChatOpenAI` (typically twice — one per ReAct cycle), and `execute_tool get_weather`.

## Swap to a real OTLP endpoint

In `main.py`, comment out the `ConsoleSpanExporter` lines and uncomment the `OTLPSpanExporter` block. Set `OTEL_EXPORTER_OTLP_ENDPOINT` in `.env` (e.g. `http://localhost:4317` for a local OTLP/gRPC collector, or `https://telemetry.googleapis.com:443/v1/traces` for Google Cloud Trace per the [reference guide](https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-langgraph)).

For Cloud Trace specifically, the reference guide also configures gRPC channel credentials with Google ADC; copy that block into Step 1 if you target Google Cloud.

## What to look for

The console output (or your OTLP backend) should contain a span tree rooted at `invoke_agent WeatherAgent`. Inside it you'll see the LangGraph ReAct loop nested under the `agent` and `tools` graph nodes, with three spans that matter for Agent 365 telemetry:

- `invoke_agent WeatherAgent` (the outer span — one per user turn; emitted by `InvokeAgentScope`)
- `chat ChatOpenAI` — one per LLM call (twice for a tool-using turn); the LangChain instrumentor renames LLM runs to `chat <run_name>` when the underlying response carries a chat-completion id, matching the [OpenTelemetry GenAI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/gen-ai-spans/)
- `execute_tool get_weather` — the tool runs (renamed by the same instrumentor; carries `gen_ai.operation.name=execute_tool` and `gen_ai.tool.name=get_weather`)

The instrumentor also emits internal LangGraph spans (`LangGraph`, `agent`, `tools`, `call_model`, `should_continue`, `RunnableSequence`, `Prompt`) — those are normal and reflect the underlying graph execution. The Agent 365 backend receives the same spans (configured via the stub token resolver — replace with a real one for production).

## Where the integration happens

`main.py` is organized into the following sections (Step 2b is a sub-step that must run after Step 2):

1. **Step 1 — OTel SDK setup.** Build a `TracerProvider`, attach a `BatchSpanProcessor` with the exporter, call `trace.set_tracer_provider(...)`. This is the part of the file you'd already have in your real app.
2. **Step 2 — Agent 365 `configure()`.** Detects the TracerProvider set by Step 1 and adds its processors to it. Both your existing exporter and the Agent 365 exporter receive spans. Replace `_stub_token_resolver` with your production token resolver.
3. **Step 2b — `CustomLangChainInstrumentor`.** Must run after `configure()`; the constructor raises `RuntimeError` otherwise. Construction auto-calls `.instrument()`. After this, every LangChain LLM and tool callback flows through Agent 365's tracer.
4. **Step 3 — Build the agent.** Standard `langgraph.prebuilt.create_react_agent(...)` with a `langchain-openai` model and a `@tool`-decorated `get_weather` function. No observability code needed (the instrumentor handles it).
5. **Step 4 — Run + flush.** `InvokeAgentScope` wraps `agent.invoke(...)` so the run gets a top-level `invoke_agent <agent_name>` span; `force_flush()` is critical — without it, batched spans may not export before the process exits.

To diff against your own app: copy Steps 1, 2, and 2b into the file where your app currently initializes its TracerProvider, and apply the Step 4 `InvokeAgentScope` wrapping pattern around your `agent.invoke(...)` calls.

## Going further

- Integration patterns and pitfalls: [Integrating with existing OpenTelemetry](https://github.com/microsoft/Agent365-python/blob/main/docs/integrating-with-existing-opentelemetry.md) (in the SDK repo)
- Manual instrumentation example (no agent framework): [`python/observability-with-otlp`](../observability-with-otlp)
- Auto-instrumented OpenAI Agents SDK example: [`python/observability-with-azure-monitor`](../observability-with-azure-monitor)
- Google Cloud reference this sample mirrors: [LangGraph + OpenTelemetry on Stackdriver](https://docs.cloud.google.com/stackdriver/docs/instrumentation/ai-agent-langgraph)

## Troubleshooting

- **Sample runs without errors but no spans appear** — most commonly `ENABLE_OBSERVABILITY` is not set to a truthy value. The SDK gates span creation behind this env var and produces zero spans silently when it's missing. The sample's `.env.template` includes it; if you assembled `.env` manually, add `ENABLE_OBSERVABILITY=true`.
- **No spans printed to stdout** — `BatchSpanProcessor` may not have flushed; the sample calls `force_flush()` on exit, so make sure the script ran to completion.
- **`KeyError` or auth error from OpenAI** — verify `OPENAI_API_KEY` (or `AZURE_OPENAI_*` variables) in `.env`. `langchain-openai` reads these directly.
- **Spans missing from your OTLP backend (after swap)** — temporarily fall back to `ConsoleSpanExporter` to confirm the SDK is producing spans. If they appear on stdout but not in your backend, the issue is in the exporter / collector / network. See [the integration guide's verify recipe](https://github.com/microsoft/Agent365-python/blob/main/docs/integrating-with-existing-opentelemetry.md#verifying-the-integration).
- **`SystemExit: Agent 365 observability configuration failed`** — check logs for the failing step (most often a missing or unreachable token resolver in production; the sample uses a stub).
- **`RuntimeError: Tracing SDK is not configured`** — `CustomLangChainInstrumentor()` ran before `configure()`. Make sure Step 2 (`configure(...)`) executes successfully before Step 2b.
- **`TypeError: wrap_function_wrapper() got an unexpected keyword argument 'module'`** — the LangChain extension uses `wrapt`'s legacy keyword-argument call style, which `wrapt 2.x` removed. `pyproject.toml` pins `wrapt<2` to keep the extension working; if you assemble dependencies manually, do the same until the SDK ships a fix.
Loading
Loading