[SVLS-7018] Support profiling in Azure Functions by kathiehuang · Pull Request #40 · DataDog/serverless-components

kathiehuang · 2025-10-01T18:40:22Z

What does this PR do?

Implements functionality in Azure Functions to handle proxy payloads - for now, specifically profile payloads
Starts a background task that runs a proxy flusher
Adds a profiling endpoint that receives profile proxy payloads and sends them through an mpsc channel where the proxy flusher receives it and forwards it to Datadog, handling retries and exponential backoffs
Attaches additional Azure Function-specific tags to the profile
- _dd.origin:azurefunction
- functionname:<name-of-function>
Works in Python, Node.js, and Java, but the Java and .NET Continuous Profilers in serverless are not currently supported by the Profiling team

Motivation

Profiling is not yet supported in Azure Functions.

Additional Notes

Tested across all hosting plans. This works for Python and Node.js.

For Node.js, profiling only has prebuilt binaries for Linux arm64/x64, Alpine Linux x64, macOS arm64/x64, and Windows x64. The default for Azure Apps in Terraform is Windows x32, so we had to go to the Configuration tab in the Azure portal and change Platform to 64 Bit to get profiling to work.
- This is documented in the Node.js profiler troubleshooting page
This worked in Java as well in testing, but the Java Continuous Profiler in serverless is not currently supported by the Profiling team.
The .NET Continuous Profiler in serverless is also not currently supported by the Profiling team - could not test because the .NET tracer shuts down the profiler if it detects it's in an Azure Function.

Describe how to test/QA your changes

To test this, follow the Serverless Compatibility Layer instructions to deploy function apps across runtimes and hosting plans with the built binary. Add DD_PROFILING_ENABLED=true in Azure app settings. Generate load on the function app and check Datadog to see profiles show up, and click into the profile > Runtime Info to see the added tags!

Ticket for handling profile payloads: SVLS-7982

Ticket for adding tags to profiles: SVLS-7983

FRSLES-759

Follow up task: Talk with APM Serverless about changes in their tracer code that will need to be made. SVLS-7841

kathiehuang · 2025-10-27T16:59:33Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-10-27T17:06:41Z

+        // Extract headers and body
+        let (parts, body) = request.into_parts();
+
+        let body_bytes = match body.collect().await {
+            Ok(collected) => collected.to_bytes(),
+            Err(e) => {
+                return log_and_create_http_response(
+                    &format!("Error reading profiling request body: {e}"),
+                    StatusCode::BAD_REQUEST,
+                );
+            }
+        };
+
+        // Create proxy request
+        let proxy_request = ProxyRequest {
+            headers: parts.headers,
+            body: body_bytes,
+            target_url: config.profiling_intake.url.to_string(),
+        };


Forward profiling query string to Datadog

The profiling proxy handler builds the outgoing request URL from the static profiling_intake endpoint and ignores parts.uri.query(). Profiling clients pass service, environment, and other metadata via the query string when posting to /profiling/v1/input; dropping these parameters means the flusher forwards payloads without required metadata, causing ingestion failures or misattribution. Append the incoming query parameters to ProxyRequest.target_url before sending to the proxy flusher.

Useful? React with 👍 / 👎.

The lambda extension doesn't preserve query parameters: https://github.com/DataDog/datadog-lambda-extension/blob/main/bottlecap/src/traces/proxy_flusher.rs#L119

kathiehuang · 2025-10-27T19:10:12Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-10-27T19:14:39Z

+    /// Handles incoming proxy requests for profiling - can be abstracted into a generic proxy handler for other proxy requests in the future
+    async fn profiling_proxy_handler(
+        config: Arc<config::Config>,
+        request: hyper_migration::HttpRequest,
+        proxy_tx: Sender<ProxyRequest>,
+    ) -> http::Result<hyper_migration::HttpResponse> {
+        debug!("Trace Agent | Received profiling request");
+
+        // Extract headers and body
+        let (parts, body) = request.into_parts();
+
+        let body_bytes = match body.collect().await {
+            Ok(collected) => collected.to_bytes(),
+            Err(e) => {
+                return log_and_create_http_response(
+                    &format!("Error reading profiling request body: {e}"),
+                    StatusCode::BAD_REQUEST,
+                );


Enforce profiling payload size before buffering

The new profiling proxy endpoint reads the entire request body into memory (body.collect().await) without validating the Content-Length. The trace and stats handlers call verify_request_content_length to reject payloads larger than Config::max_request_content_length before allocation, but the profiling handler omits this guard. A client can therefore POST an arbitrarily large profiling payload and the mini agent will buffer it in RAM before forwarding, allowing a single request to exhaust memory and crash the function. Consider applying the same content length check as the other endpoints before collecting the body.

Useful? React with 👍 / 👎.

The lambda extension also doesn't validate the content-length, probably because profiling payloads are typically much larger and they're coming from the Datadog profiler

duncanpharvey

Looks good! Just a few minor comments.

Let's make sure to test this in all hosting plans and operating systems prior to releasing. I've seen issues in the past with TLS certificates. See DataDog/libdatadog#523

kathiehuang · 2025-11-10T20:13:48Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

duncanpharvey · 2025-12-23T20:07:20Z

@codex review

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2025-12-23T20:13:06Z

+        let profiling_intake_url = format!("https://intake.profile.{}/api/v2/profile", dd_site);
+
        // DD_APM_DD_URL env var will primarily be used for integration tests
        // overrides the entire trace/trace stats intake url prefix
        if let Ok(endpoint_prefix) = env::var("DD_APM_DD_URL") {


Honor DD_APM_DD_URL for profiling intake

When a custom APM endpoint is configured via DD_APM_DD_URL (used in integration tests and deployments without direct internet access), both trace and trace-stats intakes are redirected to that prefix, but the new profiling intake stays hardcoded to https://intake.profile.{dd_site}/api/v2/profile (lines 125-132). Profiling payloads will therefore bypass the override and attempt to reach the public intake, leading to failed requests in restricted networks or data being sent to the wrong backend. The profiling endpoint should apply the same DD_APM_DD_URL override as the other intakes.

Useful? React with 👍 / 👎.

It looks like the Go Agent uses DD_APM_PROFILING_DD_URL for this: https://github.com/DataDog/datadog-agent/blob/f166f77aad04d382e1e5dceb183c7548ba73f496/pkg/config/setup/apm.go#L141

Fixed in 9576d47 to respect the DD_APM_PROFILING_DD_URL override!

… should be the prefix of the url

kathiehuang · 2026-01-13T19:04:46Z

looks good to me. though i may not be the most experienced rust dev on our team. maybe check with duncan when he comes back? or ship it and have him review it later?

Duncan has reviewed it twice so I'll ship this now! I'll chat with him about the serverless scheduler in the tracers when he's back

kathiehuang changed the title ~~[SVLS-7018] Support profiling in Azure Functions~~ WIP [SVLS-7018] Support profiling in Azure Functions Oct 1, 2025

kathiehuang force-pushed the kathie.huang/receive-profile-payload branch from c0abc20 to d9ea13b Compare October 15, 2025 18:58

kathiehuang changed the title ~~WIP [SVLS-7018] Support profiling in Azure Functions~~ [SVLS-7018] Support profiling in Azure Functions Oct 16, 2025

kathiehuang changed the title ~~[SVLS-7018] Support profiling in Azure Functions~~ [SVLS-7673] Support profiling in Azure Functions Oct 16, 2025

kathiehuang marked this pull request as ready for review October 27, 2025 16:58

kathiehuang marked this pull request as draft October 27, 2025 16:59

chatgpt-codex-connector Bot reviewed Oct 27, 2025

View reviewed changes

kathiehuang marked this pull request as ready for review October 27, 2025 19:20

kathiehuang changed the title ~~[SVLS-7673] Support profiling in Azure Functions~~ [SVLS-7018] Support profiling in Azure Functions Oct 27, 2025

kathiehuang requested review from a team and duncanpharvey October 27, 2025 19:47

duncanpharvey approved these changes Nov 5, 2025

View reviewed changes

Comment thread crates/dogstatsd/src/datadog.rs Outdated

Comment thread crates/datadog-trace-agent/src/config.rs Outdated

Comment thread crates/datadog-trace-agent/src/http_utils.rs

chatgpt-codex-connector Bot reviewed Nov 10, 2025

View reviewed changes

Comment thread crates/datadog-trace-agent/src/mini_agent.rs

kathiehuang requested a review from duncanpharvey November 10, 2025 20:19

kathiehuang marked this pull request as draft November 12, 2025 15:58

kathiehuang force-pushed the kathie.huang/receive-profile-payload branch 5 times, most recently from 7671d2d to 0663d2c Compare December 18, 2025 16:58

kathiehuang marked this pull request as ready for review December 22, 2025 19:32

kathiehuang requested review from a team as code owners December 22, 2025 19:32

chatgpt-codex-connector Bot reviewed Dec 23, 2025

View reviewed changes

kathiehuang added 21 commits January 13, 2026 13:46

fix

5eeefb3

Reduce request timeout

f1ec0b8

Add rustls-tls

a17218b

Revert to original dogstatsd build_client function

9818997

Add additional tags

c23f3eb

Formatting

a678f23

Avoid panicking

082e21c

Add aas tags

bf61041

Fix aas tags

6049999

Fix additional tags in profiles

66d97a8

Remove unnecessary debug logs

d846870

Refactor

299488d

Comment out aas.* tags for testing

b58e2d7

Update license

e5fec36

Update tags

92655f4

Respect DD_APM_PROFILING_DD_URL for profiling intake url override

8fcf166

Detect environment when setting _dd.origin tag

9330986

Enable default features for datadog-fips

9c17914

Check for 202 success code

ebf4733

Make config variable names clearer

0f671f7

Update comment about DD_APM_PROFILING_DD_URL so that it is clearer it…

4e2a124

… should be the prefix of the url

kathiehuang force-pushed the kathie.huang/receive-profile-payload branch 2 times, most recently from 7d47f2e to a5d3ca6 Compare January 13, 2026 18:58

Add codeowners file

b749b9d

kathiehuang force-pushed the kathie.huang/receive-profile-payload branch from a5d3ca6 to b749b9d Compare January 13, 2026 18:59

Fix license

b33a25b

kathiehuang merged commit 05b14cd into main Jan 13, 2026
26 checks passed

kathiehuang deleted the kathie.huang/receive-profile-payload branch January 13, 2026 21:41

duncanpharvey mentioned this pull request Jan 23, 2026

Add Windows Named Pipes to the Trace Agent #58

Merged

2 tasks

Conversation

kathiehuang commented Oct 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Motivation

Additional Notes

Describe how to test/QA your changes

Uh oh!

kathiehuang commented Oct 27, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang commented Oct 27, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Oct 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

duncanpharvey left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kathiehuang commented Nov 10, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

Uh oh!

duncanpharvey commented Dec 23, 2025

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

duncanpharvey Dec 23, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang Dec 27, 2025

Choose a reason for hiding this comment

Uh oh!

kathiehuang commented Jan 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

kathiehuang commented Oct 1, 2025 •

edited

Loading

kathiehuang Oct 27, 2025 •

edited

Loading