diff --git a/content/en/tracing/trace_collection/proxy_setup/httpd.md b/content/en/tracing/trace_collection/proxy_setup/httpd.md index d6ffb9dde0c..5a11dd2e962 100644 --- a/content/en/tracing/trace_collection/proxy_setup/httpd.md +++ b/content/en/tracing/trace_collection/proxy_setup/httpd.md @@ -74,6 +74,68 @@ DatadogServiceName my-app DatadogSamplingRate 0.1 ``` +## Correlating traces to logs + +After you've enabled APM tracing, you can connect your traces to the corresponding Apache logs. This correlation links each trace and span to the specific log events generated during that request, allowing you to pivot between them to troubleshoot issues. + +### Prerequisites + +Before you begin, ensure that you have: +- Enabled APM tracing for Apache HTTP Server by following the steps earlier in this guide. +- Configured Datadog log collection for Apache HTTP Server. + +### Step 1: Inject trace and span IDs into Apache logs + +Modify your `LogFormat` directive to include the trace ID and the span ID using the Apache-specific variables `%{Datadog-Trace-ID}e` and `%{Datadog-Span-ID}e`. These values are `-` for requests that are not traced. + +Update your Apache configuration file (for example, `/etc/apache2/apache2.conf` or `/etc/httpd/httpd.conf`): + +```apache +LogFormat "%h %l %u %t \"%r\" %>s %b %D \"%{Referer}i\" \"%{User-Agent}i\" \"%{X-Forwarded-For}i\" dd.trace_id=\"%{Datadog-Trace-ID}e\" dd.span_id=\"%{Datadog-Span-ID}e\"" datadog_combined + +CustomLog /var/log/apache2/access.log datadog_combined +``` + +After saving your changes, reload the Apache configuration. For example: + +```sh +sudo systemctl reload apache2 +# or +sudo apachectl -k graceful +``` + +### Step 2: Configure the log pipeline to parse the trace and span IDs + +By default, logs capture `dd.trace_id` and `dd.span_id` in hexadecimal format: + +``` +172.237.96.114 - - [25/Sep/2025:14:48:52 +0000] "GET / HTTP/1.1" 200 617 28248 "-" "curl/8.7.1" "-" dd.trace_id="68d5565400000000970bf1a70782d612" dd.span_id="10884058624158782994" +``` + +To enable log and trace correlation in Datadog, configure a log processing pipeline to convert these IDs from hexadecimal to decimal. + +1. In Datadog, navigate to the [**Log Configuration**][4] page. +2. Hover over your active Apache pipeline and click the **Clone** icon to create an editable version. +3. Click the cloned pipeline. +4. Click **Add Processor**. +5. Select [Grok Parser][5] as the processor type. +6. Define the following parsing rule to extract the trace ID attribute from a log event: + ```text + extract_correlation_ids %{data} dd.trace_id="%{notSpace:dd.trace_id:nullIf("-")}" dd.span_id="%{notSpace:dd.span_id:nullIf("-")}" + ``` +7. Click **Create**. +8. Click **Add Processor** again. +9. Select [Trace ID Remapper][6] as the processor type. This processor associates the parsed ID with its corresponding APM trace. +10. In the **Set trace id attribute(s)** field, enter `dd.trace_id`. +11. Click **Create**. +12. Click **Add Processor** again. +13. Select [Span ID Remapper][7] as the processor type. This processor associates the parsed ID with its corresponding APM span. +14. In the **Set span id attribute(s)** field, enter `dd.span_id`. +15. Click **Create**. +16. Save and enable your new pipeline. + +Once the pipeline is active, new Apache logs are automatically correlated with their traces and spans. + ## Further Reading {{< partial name="whats-next/whats-next.html" >}} @@ -81,3 +143,7 @@ DatadogSamplingRate 0.1 [1]: https://github.com/DataDog/httpd-datadog [2]: https://httpd.apache.org/ [3]: https://github.com/DataDog/httpd-datadog/blob/main/doc/configuration.md +[4]: https://app.datadoghq.com/logs/pipelines +[5]: /logs/log_configuration/processors/?tab=ui#grok-parser +[6]: /logs/log_configuration/processors/?tab=ui#trace-remapper +[7]: /logs/log_configuration/processors/?tab=ui#span-remapper