-
Notifications
You must be signed in to change notification settings - Fork 1.9k
Closed
Labels
enhancementNew feature or requestNew feature or request
Description
@devinjdangelo noted on #11345 that if an error happens deep in the parallel parquet writer it often is not returned to the user and instead we get a generic message "Unable to send array to writer!"
I think we need to update several map_err statements to propagate inner error messages rather than ignore them. E.g.
datafusion/datafusion/core/src/datasource/file_format/parquet.rs
Lines 880 to 884 in b96186f
| col_array_channels[next_channel] | |
| .send(c) | |
| .await | |
| .map_err(|_| { | |
| DataFusionError::Internal("Unable to send array to writer!".into()) |
change to something like
col_array_channels[next_channel]
.send(c)
.await
.map_err(|e| internal_datafusion_err!("Unable to send array to writer due to error {e}"))Originally posted by @devinjdangelo in #11345 (comment)
Metadata
Metadata
Assignees
Labels
enhancementNew feature or requestNew feature or request