Fix truncation detectability for SQL array, object formats.#11685
Merged
gianm merged 1 commit intoapache:masterfrom Sep 14, 2021
Merged
Fix truncation detectability for SQL array, object formats.#11685gianm merged 1 commit intoapache:masterfrom
gianm merged 1 commit intoapache:masterfrom
Conversation
The SQL "array" and "object" formats are intended to return invalid JSON (lacking a ] terminator) if an error occurs midstream. This enables callers to detect truncated responses. But JsonGenerators, by default, close JSON arrays even when not explicitly told to. This patch disables automatic array closing, which fixes the problem with truncated response detection. It also adds tests for truncated responses for all result formats.
gianm
added a commit
to gianm/druid
that referenced
this pull request
Feb 10, 2024
This JsonGenerator feature is on by default. It causes problems with code
like this:
try (JsonGenerator jg = ...) {
jg.writeStartArray();
for (x : xs) {
jg.writeObject(x);
}
jg.writeEndArray();
}
If a jg.writeObject call fails due to some problem with the data it's
reading, the JsonGenerator will write the end array marker automatically
when closed as part of the try-with-resources. If the generator is writing
to a stream where the reader does not have some other mechanism to realize
that an exception was thrown, this leads the reader to believe that the
array is complete when it actually isn't.
Prior to this patch, we disabled AUTO_CLOSE_JSON_CONTENT for JSON-wrapped
SQL result formats in apache#11685, which fixed an issue where such results
could be erroneously interpreted as complete. This patch fixes a similar
issue with task reports, and all similar issues that may exist elsewhere,
by disabling the feature globally.
gianm
added a commit
that referenced
this pull request
Feb 16, 2024
* Globally disable AUTO_CLOSE_JSON_CONTENT.
This JsonGenerator feature is on by default. It causes problems with code
like this:
try (JsonGenerator jg = ...) {
jg.writeStartArray();
for (x : xs) {
jg.writeObject(x);
}
jg.writeEndArray();
}
If a jg.writeObject call fails due to some problem with the data it's
reading, the JsonGenerator will write the end array marker automatically
when closed as part of the try-with-resources. If the generator is writing
to a stream where the reader does not have some other mechanism to realize
that an exception was thrown, this leads the reader to believe that the
array is complete when it actually isn't.
Prior to this patch, we disabled AUTO_CLOSE_JSON_CONTENT for JSON-wrapped
SQL result formats in #11685, which fixed an issue where such results
could be erroneously interpreted as complete. This patch fixes a similar
issue with task reports, and all similar issues that may exist elsewhere,
by disabling the feature globally.
* Update test.
sreemanamala
pushed a commit
to sreemanamala/druid
that referenced
this pull request
Feb 20, 2024
* Globally disable AUTO_CLOSE_JSON_CONTENT.
This JsonGenerator feature is on by default. It causes problems with code
like this:
try (JsonGenerator jg = ...) {
jg.writeStartArray();
for (x : xs) {
jg.writeObject(x);
}
jg.writeEndArray();
}
If a jg.writeObject call fails due to some problem with the data it's
reading, the JsonGenerator will write the end array marker automatically
when closed as part of the try-with-resources. If the generator is writing
to a stream where the reader does not have some other mechanism to realize
that an exception was thrown, this leads the reader to believe that the
array is complete when it actually isn't.
Prior to this patch, we disabled AUTO_CLOSE_JSON_CONTENT for JSON-wrapped
SQL result formats in apache#11685, which fixed an issue where such results
could be erroneously interpreted as complete. This patch fixes a similar
issue with task reports, and all similar issues that may exist elsewhere,
by disabling the feature globally.
* Update test.
cryptoe
pushed a commit
to cryptoe/druid
that referenced
this pull request
Mar 18, 2024
* Globally disable AUTO_CLOSE_JSON_CONTENT.
This JsonGenerator feature is on by default. It causes problems with code
like this:
try (JsonGenerator jg = ...) {
jg.writeStartArray();
for (x : xs) {
jg.writeObject(x);
}
jg.writeEndArray();
}
If a jg.writeObject call fails due to some problem with the data it's
reading, the JsonGenerator will write the end array marker automatically
when closed as part of the try-with-resources. If the generator is writing
to a stream where the reader does not have some other mechanism to realize
that an exception was thrown, this leads the reader to believe that the
array is complete when it actually isn't.
Prior to this patch, we disabled AUTO_CLOSE_JSON_CONTENT for JSON-wrapped
SQL result formats in apache#11685, which fixed an issue where such results
could be erroneously interpreted as complete. This patch fixes a similar
issue with task reports, and all similar issues that may exist elsewhere,
by disabling the feature globally.
* Update test.
LakshSingla
pushed a commit
that referenced
this pull request
Mar 19, 2024
* Globally disable AUTO_CLOSE_JSON_CONTENT.
This JsonGenerator feature is on by default. It causes problems with code
like this:
try (JsonGenerator jg = ...) {
jg.writeStartArray();
for (x : xs) {
jg.writeObject(x);
}
jg.writeEndArray();
}
If a jg.writeObject call fails due to some problem with the data it's
reading, the JsonGenerator will write the end array marker automatically
when closed as part of the try-with-resources. If the generator is writing
to a stream where the reader does not have some other mechanism to realize
that an exception was thrown, this leads the reader to believe that the
array is complete when it actually isn't.
Prior to this patch, we disabled AUTO_CLOSE_JSON_CONTENT for JSON-wrapped
SQL result formats in #11685, which fixed an issue where such results
could be erroneously interpreted as complete. This patch fixes a similar
issue with task reports, and all similar issues that may exist elsewhere,
by disabling the feature globally.
* Update test.
Co-authored-by: Gian Merlino <gianmerlino@gmail.com>
gianm
added a commit
to gianm/druid
that referenced
this pull request
Sep 11, 2024
Following similar issues fixed in apache#11685 and apache#15880, this patch fixes a bug where QueryResource would write a closing array marker if it encountered an exception after starting to push results. This makes it difficult for callers to detect errors. The prior patches didn't catch this problem because QueryResource uses the ObjectMapper in a unique way, through writeValuesAsArray, which doesn't respect the global AUTO_CLOSE_JSON_CONTENT setting.
gianm
added a commit
that referenced
this pull request
Sep 14, 2024
* QueryResource: Don't close JSON content on error. Following similar issues fixed in #11685 and #15880, this patch fixes a bug where QueryResource would write a closing array marker if it encountered an exception after starting to push results. This makes it difficult for callers to detect errors. The prior patches didn't catch this problem because QueryResource uses the ObjectMapper in a unique way, through writeValuesAsArray, which doesn't respect the global AUTO_CLOSE_JSON_CONTENT setting. * Fix usage of customized ObjectMappers.
pranavbhole
pushed a commit
to pranavbhole/druid
that referenced
this pull request
Sep 17, 2024
* QueryResource: Don't close JSON content on error. Following similar issues fixed in apache#11685 and apache#15880, this patch fixes a bug where QueryResource would write a closing array marker if it encountered an exception after starting to push results. This makes it difficult for callers to detect errors. The prior patches didn't catch this problem because QueryResource uses the ObjectMapper in a unique way, through writeValuesAsArray, which doesn't respect the global AUTO_CLOSE_JSON_CONTENT setting. * Fix usage of customized ObjectMappers.
AmatyaAvadhanula
pushed a commit
to AmatyaAvadhanula/druid
that referenced
this pull request
Oct 9, 2024
* QueryResource: Don't close JSON content on error. Following similar issues fixed in apache#11685 and apache#15880, this patch fixes a bug where QueryResource would write a closing array marker if it encountered an exception after starting to push results. This makes it difficult for callers to detect errors. The prior patches didn't catch this problem because QueryResource uses the ObjectMapper in a unique way, through writeValuesAsArray, which doesn't respect the global AUTO_CLOSE_JSON_CONTENT setting. * Fix usage of customized ObjectMappers.
abhishekagarwal87
pushed a commit
that referenced
this pull request
Oct 9, 2024
* QueryResource: Don't close JSON content on error. Following similar issues fixed in #11685 and #15880, this patch fixes a bug where QueryResource would write a closing array marker if it encountered an exception after starting to push results. This makes it difficult for callers to detect errors. The prior patches didn't catch this problem because QueryResource uses the ObjectMapper in a unique way, through writeValuesAsArray, which doesn't respect the global AUTO_CLOSE_JSON_CONTENT setting. * Fix usage of customized ObjectMappers. Co-authored-by: Gian Merlino <gianmerlino@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The SQL "array" and "object" formats are intended to return invalid JSON
(lacking a ] terminator) if an error occurs midstream. This enables callers
to detect truncated responses. But JsonGenerators, by default, close JSON
arrays even when not explicitly told to.
This patch disables automatic array closing, which fixes the problem with
truncated response detection with these specific formats. It also adds tests
for truncated responses for all result formats.