Skip to content

fix(elasticsearch sink): Encode bulk action parameters as JSON #21293

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
jszwedko merged 7 commits into from
Jun 13, 2025
Merged

fix(elasticsearch sink): Encode bulk action parameters as JSON #21293

jszwedko merged 7 commits into from
Jun 13, 2025

Conversation

@jszwedko
Copy link
Member

@jszwedko jszwedko commented Sep 13, 2024

They are currently using string templating which, if there are special characters in the value, will
end up creating an invalid JSON payload; an issue that can be difficult to track down. This happened
in #21288.

Signed-off-by: Jesse Szwedko jesse.szwedko@datadoghq.com

@jszwedko
fix(elasticsearch): Encode bulk action parameters as JSON
e874265 
They are currently using string templating which, if there are special characters in the value, will
end up creating an invalid JSON payload; an issue that can be difficult to track down. This happened
in #21288.

Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
@jszwedko jszwedko requested a review from a team as a code owner September 13, 2024 14:47
@github-actions github-actions bot added the domain: sinks Anything related to the Vector's sinks label Sep 13, 2024
@jszwedko jszwedko enabled auto-merge September 13, 2024 14:50
@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Sep 13, 2024 •
edited
Loading

Datadog Report

Branch report: jszwedko/fix-elasticsearch-message-encoding
Commit report: 509120c
Test service: vector

❌ 9 Failed (0 Known Flaky), 2223 Passed, 0 Skipped, 1m 31.64s Total Time

❌ Failed Tests (9)

This report shows up to 5 failed tests.

  • sinks::elasticsearch::tests::allows_using_except_fields - vector - Details

    Expand for error 
     thread 'sinks::elasticsearch::tests::allows_using_except_fields' panicked at src/sinks/elasticsearch/tests.rs:494:5:
 assertion \`left == right\` failed
   left: "{\"index\":{\"_type\":\"_doc\",\"_index\":\"purple\"}}\n{\"foo\":\"bar\",\"message\":\"hello there\"}\n"
  right: "{\"index\":{\"_index\":\"purple\",\"_type\":\"_doc\"}}\n{\"foo\":\"bar\",\"message\":\"hello there\"}\n"
 stack backtrace:
    0:     0x5575c3570165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x5575c3570165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x5575c3570165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
 ...

  • sinks::elasticsearch::tests::allows_using_only_fields - vector - Details

    Expand for error 
     thread 'sinks::elasticsearch::tests::allows_using_only_fields' panicked at src/sinks/elasticsearch/tests.rs:529:5:
 assertion \`left == right\` failed
   left: "{\"index\":{\"_type\":\"_doc\",\"_index\":\"purple\"}}\n{\"foo\":\"bar\"}\n"
  right: "{\"index\":{\"_index\":\"purple\",\"_type\":\"_doc\"}}\n{\"foo\":\"bar\"}\n"
 stack backtrace:
    0:     0x557e6a475165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x557e6a475165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x557e6a475165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
 ...

  • sinks::elasticsearch::tests::encode_datastream_mode - vector - Details

    Expand for error 
     thread 'sinks::elasticsearch::tests::encode_datastream_mode' panicked at src/sinks/elasticsearch/tests.rs:251:5:
 assertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"synthetics-testing-default\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"default\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"synthetics-testing-default\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"default\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
 stack backtrace:
    0:     0x557f7366b165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x557f7366b165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x557f7366b165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
 ...

  • sinks::elasticsearch::tests::encode_datastream_mode_no_routing - vector - Details

    Expand for error 
     thread 'sinks::elasticsearch::tests::encode_datastream_mode_no_routing' panicked at src/sinks/elasticsearch/tests.rs:306:5:
 assertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"logs-generic-something\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"something\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"logs-generic-something\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"namespace\":\"something\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
 stack backtrace:
    0:     0x55dabe5bb165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x55dabe5bb165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x55dabe5bb165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
 ...

  • sinks::elasticsearch::tests::encode_datastream_mode_no_sync - vector - Details

    Expand for error 
     thread 'sinks::elasticsearch::tests::encode_datastream_mode_no_sync' panicked at src/sinks/elasticsearch/tests.rs:458:5:
 assertion \`left == right\` failed
   left: "{\"create\":{\"_type\":\"_doc\",\"_index\":\"synthetics-testing-something\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
  right: "{\"create\":{\"_index\":\"synthetics-testing-something\",\"_type\":\"_doc\"}}\n{\"@timestamp\":\"2020-12-01T01:02:03Z\",\"data_stream\":{\"dataset\":\"testing\",\"type\":\"synthetics\"},\"message\":\"hello there\"}\n"
 stack backtrace:
    0:     0x55ae51faf165 - std::backtrace_rs::backtrace::libunwind::trace::h649ab3318d3445c5
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/libunwind.rs:116:5
    1:     0x55ae51faf165 - std::backtrace_rs::backtrace::trace_unsynchronized::hf4bb60c3387150c3
                                at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/std/src/../../backtrace/src/backtrace/mod.rs:66:5
    2:     0x55ae51faf165 - std::sys::backtrace::_print_fmt::hd9186c800e44bd00
 ...

pront
pront approved these changes Sep 13, 2024
View reviewed changes
@pront
Copy link
Member

pront commented Oct 17, 2024

Note: Tests fail due the order of keys in the objects. The data look identical otherwise.

@jszwedko
Merge remote-tracking branch 'origin/master' into jszwedko/fix-elasti…
88d8a79 
…csearch-message-encoding
@jszwedko
Fix tests
6329720 
Signed-off-by: Jesse Szwedko <jesse.szwedko@datadoghq.com>
@jszwedko jszwedko added the no-changelog Changes in this PR do not need user-facing explanations in the release changelog label Nov 1, 2024
@jszwedko jszwedko changed the title fix(elasticsearch): Encode bulk action parameters as JSON fix(elasticsearch source): Encode bulk action parameters as JSON Nov 1, 2024
@jszwedko jszwedko changed the title fix(elasticsearch source): Encode bulk action parameters as JSON fix(elasticsearch sink): Encode bulk action parameters as JSON Nov 1, 2024
@datadog-vectordotdev
Copy link

datadog-vectordotdev bot commented Jun 13, 2025 •
edited
Loading

Datadog Report

Branch report: jszwedko/fix-elasticsearch-message-encoding
Commit report: dfadc0b
Test service: vector

✅ 0 Failed, 7 Passed, 0 Skipped, 25.26s Total Time

@jszwedko jszwedko added this pull request to the merge queue Jun 13, 2025
Merged via the queue into master with commit e74575a Jun 13, 2025
74 checks passed
@jszwedko jszwedko deleted the jszwedko/fix-elasticsearch-message-encoding branch June 13, 2025 20:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pront pront pront approved these changes

Assignees

No one assigned

Labels

domain: sinks Anything related to the Vector's sinks no-changelog Changes in this PR do not need user-facing explanations in the release changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@jszwedko @pront @thomasqueirozb