tables_to_parquet() fails sporadically for large datasets

[galamit86]

In some cases, tables_to_parquet() fails on a large dataset, due to a restart of the task. The pathology is:

1. A long time (over an hour) between this message (indicating the heaviest dask computation is starting):

Dumping bag to textfiles for cbs.v3.84750NED_TypedDataSet

and the next one, (indicating this task has been restarted):

Task 'tables_to_parquet[8]': Starting task run...

2. The task continues, reaching the first message again (Dumping bag...)
3. This time around it finishes, logging:

Finished dumping bag to textfiles for cbs.v3.84750NED_TypedDataSet
Starting to concatanate files for cbs.v3.84750NED_TypedDataSet
Concluded concatanating files for cbs.v3.84750NED_TypedDataSet

4. And then throws an error:

Unexpected error: FileNotFoundError(2, "Failed to open local file '/tmp/cbs/v3/84750NED/20210320/json/cbs.v3.84750NED_TypedDataSet/cbs.v3.84750NED_TypedDataSet.json'. Detail: [errno 2] No such file or directory")
Traceback (most recent call last):
  File "/home/amitgalmail/nl-open-data/.venv/lib/python3.8/site-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/home/amitgalmail/nl-open-data/.venv/lib/python3.8/site-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/home/amitgalmail/nl-open-data/.venv/lib/python3.8/site-packages/prefect/utilities/executors.py", line 299, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/home/amitgalmail/nl-open-data/.venv/lib/python3.8/site-packages/statline_bq/utils.py", line 1390, in tables_to_parquet
    pq_path = convert_table_to_parquet(
  File "/home/amitgalmail/nl-open-data/.venv/lib/python3.8/site-packages/statline_bq/utils.py", line 629, in convert_table_to_parquet
    pa_table = pa_json.read_json(json_path)
  File "pyarrow/_json.pyx", line 238, in pyarrow._json.read_json
  File "pyarrow/_json.pyx", line 193, in pyarrow._json._get_reader
  File "pyarrow/io.pxi", line 1493, in pyarrow.lib.get_input_stream
  File "pyarrow/io.pxi", line 1464, in pyarrow.lib.get_native_file
  File "pyarrow/io.pxi", line 827, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 837, in pyarrow.lib.OSFile._open_readable
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/tmp/cbs/v3/84750NED/20210320/json/cbs.v3.84750NED_TypedDataSet/cbs.v3.84750NED_TypedDataSet.json'. Detail: [errno 2] No such file or directory

Full log from prefect attached.

[galamit86]

Flow fails with different pathology, but could be related - again the task is restarted, also with a relatively large dataset. View in Slack

[galamit86]

Attached logs of a currently running tables_to_parquet for a single dataset: cbs.v3.84750NED.

Same pathology (several restarts, and eventually a FileNotFound error).

The thrown error points to this line in statline-bq.

It occurs, as far as I can tell, only in larger datasets, So we can focus on a couple of directions:

1. Why is a long task being restarted in Prefect without any apparent warning? SIDE NOTE: This might have also happened to tasks that finished successfully altogether, but not sure.
2. What is the immediate cause of the error? Follow the error trail.
3. Why are the restarts causing this issue? Maybe adding timestamp to text file creation can help?

@dkapitan - I might like to schedule a deep dive with you, if you're interested and I don't work it out eventually myself.

[galamit86]

Another failed table_to_parquet task - the final error is different (JSONDecodeError ), but again a restart before the error.

log file

[galamit86]

Could (or could not) be related - this flow has been running for almost 12 hours, and still going - completed 5/10 datasets so far. Maybe it's simply many large datasets in this same flow, or something else is causing the long duration.

[dkapitan]

I think we have a tenacious bug. Can't wait for the moment that we find it, after which we will be slapping ourselves for not having thought of *that* earlier ... 😂

…

On Tue, 23 Mar 2021, 21:10 Amit Gal, ***@***.***> wrote: Could (or could not) be related - this flow <https://cloud.prefect.io/dataverbinders/task-run/9eb27f12-d6e5-4e7d-8c1e-a8daf9b9877f?mapped-runs=> has been running for almost 12 hours, and still going - completed 5/10 datasets so far. Maybe it's simply many large datasets in this same flow, or something else is causing the long duration. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GYDTKES3OCQQMX3LZO43TFDYTFANCNFSM4ZQ4W3SQ> .

[galamit86]

Yep, I agree. And it's been driving me crazy....

[dkapitan]

You get to do most of the slapping then 😂

…

On Tue, 23 Mar 2021, 21:36 Amit Gal, ***@***.***> wrote: Yep, I agree. And it's been driving me crazy.... — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GYDW43XG4PXI6SDNAQ6TTFD3TTANCNFSM4ZQ4W3SQ> .