Initial implementation of OTel process context support #266

jbachorik · 2025-08-22T11:35:47Z

What does this PR do?:
It adds the support for publishing process context in OTel friendly way.

Additional Notes:
See https://github.com/DataDog/fullhost-code-hotspots-wip/pull/2

How to test the change?:

For Datadog employees:

If this PR touches code that signs or publishes builds or packages, or handles
credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.
This PR doesn't touch any of that.
JIRA: PROF-12377

Unsure? Have a question? Request a review!

ivoanjo

Nice! Looks like it was very straightforward to wire this in, which was the point :)

Left a few notes.

ivoanjo · 2025-08-26T09:19:35Z

ddprof-lib/src/main/cpp/otel_process_ctx_macos.cpp

Ah, I can see why having a no-op fallback may be useful for implementers. I'll ~~borrow~~ ~~steal~~ build on this idea and add it on the reference implementation so we don't need to maintain it separately

Update: https://github.com/DataDog/fullhost-code-hotspots-wip/pull/2 now has a no-op implementation

ddprof-lib/src/main/java/com/datadoghq/profiler/OTelContext.java

ddprof-lib/src/main/cpp/javaApi.cpp

ddprof-lib/src/main/java/com/datadoghq/profiler/OTelContext.java

ddprof-test/src/test/java/com/datadoghq/profiler/context/ProcessContextTest.java

Co-authored-by: Ivo Anjo <ivo@ivoanjo.me>

- Add comprehensive gradle patching for otel_process_ctx.c to .cpp conversion - Add Linux preprocessor guards and C++ explicit casts for compilation - Fix gradle task dependencies and caching for proper file handling - Implement proper publish/update API usage in JNI setProcessCtx0 - Add native read functionality through JNI wrapper - Update ProcessContextTest for test resilience and native read testing - Resolve all compilation failures in gtest tasks 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>

ivoanjo

👍 LGTM left a few small notes, but overall looks very reasonable!

ddprof-lib/build.gradle

ddprof-lib/src/main/cpp/javaApi.cpp

ddprof-lib/src/main/java/com/datadoghq/profiler/OTelContext.java

ddprof-test/src/test/java/com/datadoghq/profiler/context/ProcessContextTest.java

ddprof-lib/src/main/cpp/otel_process_ctx_macos.cpp

ddprof-lib/src/main/java/com/datadoghq/profiler/OTelContext.java

ddprof-lib/src/main/cpp/javaApi.cpp

ivoanjo · 2025-08-29T08:30:49Z

I've opened #267 to bring the latest version of the reference library on top of this PR/branch :)

…ference library (#267)

jbachorik · 2025-08-29T10:06:56Z

@zhengyu123 I addressed the locking. If you could re-review and potentially approve ...

r1viollet · 2025-09-01T12:05:46Z

Exciting to see this! 👏

I addressed all concerns.

jbachorik · 2025-09-01T12:58:56Z

@r1viollet Can you 👍 this?
@zhengyu123 and @MattAlp are off today because of public holiday ...

r1viollet · 2025-09-04T08:21:49Z

ddprof-lib/src/main/cpp/otel_process_ctx.cpp

+  // (due to the MADV_DONTFORK) and we don't need to do anything to it.
+  if (state.mapping != NULL && state.mapping != MAP_FAILED && getpid() == state.publisher_pid) {
+    long mapping_size = size_for_mapping();
+    if (mapping_size == -1 || munmap(published_state.mapping, mapping_size) == -1) return false;


confusion between state.mapping and published_state.mapping

Really nice spot! I'll open a PR to fix this.

ddprof-lib/src/main/cpp/otel_process_ctx.cpp

r1viollet · 2025-09-04T08:33:43Z

ddprof-lib/src/main/cpp/otel_process_ctx.cpp

+        data_out->telemetry_sdk_name = value;
+      } else {
+        // Unknown key, put it into resources
+        char *key = (char *) calloc(key_len + 1, 1);


Could we avoid the manual allocs ? if we have to handle frees on error we will make mistakes
example not freeing other resources (data_out->resources )

Note that the resources are being cleared in the caller:

if (!otel_process_ctx_decode_payload(mapping->otel_process_payload, &data)) { otel_process_ctx_read_data_drop(data);

For avoiding manual allocs, any suggestions? Were you thinking something closer to -- allocate a big block of and then just memcpy into it?

(Also the reading support is mostly only for debugging/tests and can even be disabled individually, as it's not particularly interesting in actual production apps)

Disclaimer: when I initially wrote this I forgot that we had to be compatible with C 😄
In C++ the types would have defined the ownership.

In C, we can try to be more symmetrical on alloc / free code paths: having clear init / free functions to look for.
I'm not sure we need to polish this right now.

r1viollet · 2025-09-04T08:36:01Z

More generally I think we need to consider invalid inputs, size limits, null values (as these can come from users). Java might be handling some of this. This is not a blocker for prototype.

ivoanjo · 2025-09-04T10:57:18Z

More generally I think we need to consider invalid inputs, size limits, null values (as these can come from users). Java might be handling some of this. This is not a blocker for prototype.

I believe all of these are checked? validate_and_calculate_payload_size checks the size of every string + that it's not null while encoding; or do you mean in the reader?

This is not a blocker for prototype.

We definitely want to make this as production-quality as possible (especially the writer, but the reader as a bonus) so all feedback for that is welcome

jbachorik requested review from ivoanjo and zhengyu123 August 22, 2025 11:35

Initial implementation of OTel process context support

f0da6a1

jbachorik force-pushed the jb/ebf_ctx branch from f06aa29 to f0da6a1 Compare August 22, 2025 12:12

ivoanjo reviewed Aug 26, 2025

View reviewed changes

jbachorik and others added 4 commits August 26, 2025 11:41

Update ddprof-lib/src/main/java/com/datadoghq/profiler/OTelContext.java

c91f4a2

Co-authored-by: Ivo Anjo <ivo@ivoanjo.me>

Use the upstream ebpf-context lib sources

d8f0f5e

Merge remote-tracking branch 'origin/jb/ebf_ctx' into jb/ebf_ctx

5038112

jbachorik force-pushed the jb/ebf_ctx branch from c2ce3d5 to bf84dc1 Compare August 26, 2025 12:00

CI can't access the ebpf-context repo

60550a8

jbachorik requested a review from MattAlp August 26, 2025 15:21

ivoanjo approved these changes Aug 26, 2025

View reviewed changes

jbachorik added 4 commits August 27, 2025 09:00

Make OTelContext actually thread safe

6fd5615

Remove unused repo locks from build

60207ca

Typo

f2f3851

Simplify ProcessContextTest.java

f5a4274

zhengyu123 previously requested changes Aug 27, 2025

View reviewed changes

[PROF-12377] Update process context support with latest version of re…

2d52609

…ference library (#267)

jbachorik force-pushed the jb/ebf_ctx branch from 0baa232 to 2d5f54b Compare August 29, 2025 10:06

Optimize locking

ba7f386

jbachorik force-pushed the jb/ebf_ctx branch from 2d5f54b to ba7f386 Compare August 29, 2025 10:36

MattAlp approved these changes Sep 3, 2025

View reviewed changes

jbachorik merged commit 887508d into main Sep 3, 2025
182 of 183 checks passed

jbachorik deleted the jb/ebf_ctx branch September 3, 2025 16:10

github-actions bot added this to the 1.30.0 milestone Sep 3, 2025

r1viollet reviewed Sep 4, 2025

View reviewed changes

ddprof-lib/src/main/cpp/otel_process_ctx.cpp Show resolved Hide resolved

r1viollet reviewed Sep 4, 2025

View reviewed changes

Initial implementation of OTel process context support #266

Initial implementation of OTel process context support #266

Uh oh!

Conversation

jbachorik commented Aug 22, 2025 • edited by atlassian bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

ivoanjo Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

ivoanjo Aug 26, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivoanjo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ivoanjo commented Aug 29, 2025

Uh oh!

jbachorik commented Aug 29, 2025

Uh oh!

r1viollet commented Sep 1, 2025

Uh oh!

jbachorik commented Sep 1, 2025

Uh oh!

Uh oh!

r1viollet Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ivoanjo Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

r1viollet Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

ivoanjo Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

r1viollet Sep 4, 2025

Choose a reason for hiding this comment

Uh oh!

r1viollet commented Sep 4, 2025

Uh oh!

ivoanjo commented Sep 4, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

jbachorik commented Aug 22, 2025 •

edited by atlassian bot

Loading