Logs the BigQuery error that caused the load job failure (resolves #1… #36068
Conversation
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @mburuian6, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request enhances the error reporting for BigQuery load jobs by logging more specific details about the BigQuery error that caused the job to fail. This change aims to improve the diagnostic capabilities when encountering persistent load job failures, providing clearer insights into the underlying problems.
Highlights
- Enhanced Error Logging: Introduced detailed logging for BigQuery load job failures, specifically capturing the BigQuery
errorResultand the root cause error from theerrorsarray. - Improved Debuggability: The new logging provides more specific information when a BigQuery load job fails after all retries, aiding in quicker identification and resolution of issues.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment |
|
Assigning reviewers: R: @robertwb for label java. Note: If you would like to opt out of this review, comment Available commands:
The PR bot will only process comments in the main thread (not review comments). |
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request adds logging for BigQuery load job failures, which is a great improvement for debugging. The new logic correctly identifies the final error and the causative error from the job status.
My review focuses on improving the robustness and efficiency of the new logging method. I've suggested a revision to handle potential null values safely, avoid redundant method calls, and refine the log message formatting. These changes will make the error logging more reliable.
| void logBigQueryError(@Nullable Job job) { | ||
| if (job == null || !parseStatus(job).equals(Status.FAILED)) { | ||
| return; | ||
| } | ||
|
|
||
| List<ErrorProto> jobErrors = job.getStatus().getErrors(); | ||
| String finalError = job.getStatus().getErrorResult().getMessage(); | ||
| String causativeError = | ||
| jobErrors != null && !jobErrors.isEmpty() | ||
| ? String.format(" due to: %s", jobErrors.get(jobErrors.size() - 1).getMessage()) | ||
| : ""; | ||
|
|
||
| LOG.error(String.format("BigQuery Error : %s %s", finalError, causativeError)); | ||
| } |
There was a problem hiding this comment.
The new logBigQueryError method has a few issues that could lead to runtime errors and inefficient execution:
- Potential
NullPointerException:job.getStatus()can returnnull, which is not handled. The call toparseStatus(job)is unsafe asparseStatusitself doesn't handle a nullJobStatus. Even if it were safe, subsequent calls tojob.getStatus()within this method are also not null-safe. - Inefficiency:
job.getStatus()is called multiple times. It's better to call it once and store the result in a local variable. - Incomplete error handling: The logic assumes
job.getStatus().getErrorResult()is not null. If a job fails with errors in theerrorslist buterrorResultis null, this method would fail instead of logging the available error information. - Log message formatting: The use of
String.format("... %s %s", ...)can result in extraneous spaces in the log message.
Here is a revised implementation that addresses these points, making the method safer, more efficient, and more robust.
void logBigQueryError(@Nullable Job job) {
if (job == null) {
return;
}
final JobStatus status = job.getStatus();
if (status == null
|| (status.getErrorResult() == null
&& (status.getErrors() == null || status.getErrors().isEmpty()))) {
return;
}
String finalError;
if (status.getErrorResult() != null && status.getErrorResult().getMessage() != null) {
finalError = status.getErrorResult().getMessage();
} else {
finalError = "Job failed with an unknown error";
}
List<ErrorProto> jobErrors = status.getErrors();
String causativeErrorDetails = "";
if (jobErrors != null && !jobErrors.isEmpty()) {
causativeErrorDetails = " due to: " + jobErrors.get(jobErrors.size() - 1).getMessage();
}
LOG.error(String.format("BigQuery Error : %s%s", finalError, causativeErrorDetails));
}
There was a problem hiding this comment.
The null issues called out here aren't real, these methods all return non-null results.
The efficiency piece doesn't matter in the failure condition, and the formatting should be fine
|
Can we also add a test here? thanks |
| void logBigQueryError(@Nullable Job job) { | ||
| if (job == null || !parseStatus(job).equals(Status.FAILED)) { | ||
| return; | ||
| } | ||
|
|
||
| List<ErrorProto> jobErrors = job.getStatus().getErrors(); | ||
| String finalError = job.getStatus().getErrorResult().getMessage(); | ||
| String causativeError = | ||
| jobErrors != null && !jobErrors.isEmpty() | ||
| ? String.format(" due to: %s", jobErrors.get(jobErrors.size() - 1).getMessage()) | ||
| : ""; | ||
|
|
||
| LOG.error(String.format("BigQuery Error : %s %s", finalError, causativeError)); | ||
| } |
There was a problem hiding this comment.
The null issues called out here aren't real, these methods all return non-null results.
The efficiency piece doesn't matter in the failure condition, and the formatting should be fine
|
LGTM, but agreed with Derrick's comment, adding a test is a good idea |
|
@damccorm @derrickaw I think this is the test for the failure message , I'd probably have to change the FakeJobService to give all failed jobs a list of error protos that include the errorResult. |
|
Yeah, I think that is right. Looking at that test, it does look like we at least execute this code path, which seems probably good enough. Testing log messages is otherwise painful. So I think we can just take this as is |
3 participants
…8419)
Fixes #18419
errorResultand root cause error from theerrors[]field of the last attempted job - sample errorThank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, commentfixes #<ISSUE NUMBER>instead.CHANGES.mdwith noteworthy changes.See the Contributor Guide for more tips on how to make review process smoother.
To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md
GitHub Actions Tests Status (on master branch)
See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.