Skip to content

Add S3 Downstream Span Pointers #587

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
nhulston merged 15 commits into from
Dec 6, 2024
Merged

Add S3 Downstream Span Pointers #587

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
4ac118d
Add span pointers downstream
nhulston Nov 14, 2024
a3b587b
Add unit tests
nhulston Nov 14, 2024
340ac84
Code review improvement
nhulston Nov 15, 2024
9e02a6a
Merge branch 'main' into nicholas.hulston/s3-span-pointers
nhulston Nov 18, 2024
d923f05
Merge branch 'main' into nicholas.hulston/s3-span-pointers
nhulston Nov 20, 2024
d30e722
Merge branch 'main' into nicholas.hulston/s3-span-pointers
nhulston Nov 22, 2024
b50916d
Only import tracer dependencies when we need it
nhulston Nov 22, 2024
0715e81
Fix tests
nhulston Nov 22, 2024
3f7858d
Remove noisy debug
nhulston Nov 22, 2024
0d1aecb
Code review improvements
nhulston Nov 26, 2024
44b737a
Code review improvements 2
nhulston Nov 26, 2024
86bab48
Add integration tests
nhulston Nov 27, 2024
996b6f6
snapshots
nhulston Nov 27, 2024
6c18199
Merge branch 'main' into nicholas.hulston/s3-span-pointers
nhulston Dec 5, 2024
bbf5575
Add/update Node 22 snapshots
nhulston Dec 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 39 additions & 0 deletions integration_tests/input_events/s3-completemultipartupload.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2024-11-27T15:13:12.668Z",
"eventName": "ObjectCreated:CompleteMultipartUpload",
"userIdentity": {
"principalId": "AWS:ABCDEFGHIJKLMNOPQRSTU:my-upstream-function"
},
"requestParameters": {
"sourceIPAddress": "1.234.56.78"
},
"responseElements": {
"x-amz-request-id": "V1234567890ABCDE",
"x-amz-id-2": "ABCDEFGHI+abcdefghijklmn/1234567890+1234567890="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "abcde-1234-5678-9012-abcde1234567890",
"bucket": {
"name": "my-bucket-name",
"ownerIdentity": {
"principalId": "ABCDEFGHIJKLMN"
},
"arn": "arn:aws:s3:::my-bucket-name"
},
"object": {
"key": "multipart_object.txt",
"size": 8388608,
"eTag": "09876543210987654321098765432109-2",
"versionId": "r_qtaksdfj9127489Uaksdfj",
"sequencer": "00112233445566AABB"
}
}
}
]
}
39 changes: 39 additions & 0 deletions integration_tests/input_events/s3-copyobject.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2024-11-27T15:13:08.307Z",
"eventName": "ObjectCreated:Copy",
"userIdentity": {
"principalId": "AWS:ABCDEFGHIJKLMNOPQRSTU:my-upstream-function"
},
"requestParameters": {
"sourceIPAddress": "1.234.56.78"
},
"responseElements": {
"x-amz-request-id": "V1234567890ABCDE",
"x-amz-id-2": "ABCDEFGHI+abcdefghijklmn/1234567890+1234567890="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "abcde-1234-5678-9012-abcde1234567890",
"bucket": {
"name": "my-bucket-name",
"ownerIdentity": {
"principalId": "ABCDEFGHIJKLMN"
},
"arn": "arn:aws:s3:::my-bucket-name"
},
"object": {
"key": "copied_object.txt",
"size": 100,
"eTag": "01234567890123456789012345678901",
"versionId": "r_qtaksdfj9127489Uaksdfj",
"sequencer": "00112233445566AABB"
}
}
}
]
}
39 changes: 39 additions & 0 deletions integration_tests/input_events/s3-putobject.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
{
"Records": [
{
"eventVersion": "2.1",
"eventSource": "aws:s3",
"awsRegion": "us-east-1",
"eventTime": "2024-11-27T15:13:03.715Z",
"eventName": "ObjectCreated:Put",
"userIdentity": {
"principalId": "AWS:ABCDEFGHIJKLMNOPQRSTU:my-upstream-function"
},
"requestParameters": {
"sourceIPAddress": "1.234.56.78"
},
"responseElements": {
"x-amz-request-id": "V1234567890ABCDE",
"x-amz-id-2": "ABCDEFGHI+abcdefghijklmn/1234567890+1234567890="
},
"s3": {
"s3SchemaVersion": "1.0",
"configurationId": "abcde-1234-5678-9012-abcde1234567890",
"bucket": {
"name": "my-bucket-name",
"ownerIdentity": {
"principalId": "ABCDEFGHIJKLMN"
},
"arn": "arn:aws:s3:::my-bucket-name"
},
"object": {
"key": "test_object.txt",
"size": 100,
"eTag": "abcdef0123456789abcdef01234567890",
"versionId": "r_qtaksdfj9127489Uaksdfj",
"sequencer": "00112233445566AABB"
}
}
}
]
}
6 changes: 6 additions & 0 deletions integration_tests/process-input-traced.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,12 @@ const getRecordIds = tracer.wrap("getRecordIds", (event) => {
eventType = "SNS";
recordIds.push(record.Sns.MessageId);
}
if (record.s3) {
eventType = "S3";
const bucketName = record.s3.bucket.name;
const objectKey = record.s3.object.key;
recordIds.push(`${bucketName}/${objectKey}`);
}
});
}

Expand Down
92 changes: 88 additions & 4 deletions integration_tests/snapshots/logs/async-metrics_node16.log
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@

START
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed APIGateway request
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
Expand All @@ -8,8 +9,8 @@ START
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"cold_start:true",
"memorysize:1024",
"cold_start:true",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
Expand All @@ -25,7 +26,6 @@ START
],
"v": 1
}
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed APIGateway request
{
"e": XXXX,
"m": "serverless.integration_test.execution",
Expand All @@ -38,6 +38,7 @@ XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed APIGat
}
END Duration: XXXX ms (init: XXXX ms) Memory Used: XXXX MB
START
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed undefined request
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
Expand All @@ -46,14 +47,97 @@ START
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"memorysize:1024",
"cold_start:false",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
"v": 1
}
{
"e": XXXX,
"m": "serverless.integration_test.execution",
"t": [
"tagkey:tagvalue",
"eventsource:undefined",
"dd_lambda_layer:datadog-nodev16.XX.X"
],
"v": 1
}
END Duration: XXXX ms Memory Used: XXXX MB
START
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
"t": [
"region:eu-west-1",
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"memorysize:1024",
"cold_start:false",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
"v": 1
}
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed undefined request
{
"e": XXXX,
"m": "serverless.integration_test.execution",
"t": [
"tagkey:tagvalue",
"eventsource:undefined",
"dd_lambda_layer:datadog-nodev16.XX.X"
],
"v": 1
}
END Duration: XXXX ms Memory Used: XXXX MB
START
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
"t": [
"region:eu-west-1",
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"memorysize:1024",
"cold_start:false",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
"v": 1
}
{
"e": XXXX,
"m": "serverless.integration_test.execution",
"t": [
"tagkey:tagvalue",
"eventsource:undefined",
"dd_lambda_layer:datadog-nodev16.XX.X"
],
"v": 1
}
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed undefined request
END Duration: XXXX ms Memory Used: XXXX MB
START
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SNS request
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
"t": [
"region:eu-west-1",
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"memorysize:1024",
"cold_start:false",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
"v": 1
}
{
"e": XXXX,
"m": "serverless.integration_test.records_processed",
Expand All @@ -76,7 +160,6 @@ XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SNS re
}
END Duration: XXXX ms Memory Used: XXXX MB
START
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SQS request
{
"e": XXXX,
"m": "aws.lambda.enhanced.invocations",
Expand All @@ -85,8 +168,8 @@ XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SQS re
"account_id:XXXX",
"functionname:integration-tests-js-XXXX-async-metrics_node16",
"resource:integration-tests-js-XXXX-async-metrics_node16",
"cold_start:false",
"memorysize:1024",
"cold_start:false",
"datadog_lambda:vX.X.X",
"runtime:nodejs16.x"
],
Expand All @@ -102,6 +185,7 @@ XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SQS re
],
"v": 1
}
XXXX-XX-XX XX:XX:XX.XXX INFO [dd.trace_id=XXXX dd.span_id=XXXX] Processed SQS request
{
"e": XXXX,
"m": "serverless.integration_test.records_processed",
Expand Down
Loading