`Exception while flushing logs: 429 Exceeded rate limits` using `jetstream preview ...`

[ncalexan]

I'm pretty sure jetstream preview ... shouldn't be flushing logs to BQ, full stop, and therefore shouldn't hit any limits. Is this not correct?

Tail of the log is:

$ jetstream preview --config-repos=/Users/nalexander/Mozilla/metric-hub --config-repos=/Users/nalexander/Mozilla/metric-hub/jetstream --platform fenix --sql-output-dir=output --experiment-slug=newly-dormant-push-programme-phase-1-android-retest
...
INFO:2026-05-12 15:37:01,659:jetstream.analysis:Metric query cost: 0.0642032
INFO:2026-05-12 15:37:03,502:jetstream.analysis:Metric query cost: 0.06291158333333334
WARNING:2026-05-12 15:37:04,943:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
WARNING:2026-05-12 15:37:04,987:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
WARNING:2026-05-12 15:37:05,029:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
WARNING:2026-05-12 15:37:10,721:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
WARNING:2026-05-12 15:37:10,810:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
WARNING:2026-05-12 15:37:10,894:jetstream.analysis:Covariate adjustment table newly_dormant_push_programme_phase_1_android_retest_enrollments_preenrollment_week_1 does not exist, falling back to unadjusted inferences
A preview is available at: https://mozilla.cloud.looker.com/dashboards/experimentation::jetstream_preview?Project=mozdata&Dataset=tmp&Slug=newly_dormant_push_programme_phase_1_android_retest
2026-05-12 15:37:29,769 - distributed.worker - ERROR - Failed to communicate with scheduler during heartbeat.
Traceback (most recent call last):
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/comm/tcp.py", line 226, in read
    frames_nosplit_nbytes_bin = await stream.read_bytes(fmt_size)
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
tornado.iostream.StreamClosedError: Stream is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/worker.py", line 1273, in heartbeat
    response = await retry_operation(
               ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/utils_comm.py", line 416, in retry_operation
    return await retry(
           ^^^^^^^^^^^^
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/utils_comm.py", line 395, in retry
    return await coro()
           ^^^^^^^^^^^^
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/core.py", line 1259, in send_recv_from_rpc
    return await send_recv(comm=comm, op=key, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/core.py", line 1018, in send_recv
    response = await comm.read(deserializers=deserializers)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/comm/tcp.py", line 237, in read
    convert_stream_closed_error(self, e)
  File "/Users/nalexander/Mozilla/jetstream/venv/lib/python3.11/site-packages/distributed/comm/tcp.py", line 137, in convert_stream_closed_error
    raise CommClosedError(f"in {obj}: {exc}") from exc
distributed.comm.core.CommClosedError: in <TCP (closed) ConnectionPool.heartbeat_worker local=tcp://127.0.0.1:49931 remote=tcp://127.0.0.1:49835>: Stream is closed
Exception while flushing logs: 429 Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas; reason: rateLimitExceeded, location: table.write, message: Exceeded rate limits: too many table update operations for this table. For more information, see https://cloud.google.com/bigquery/docs/troubleshoot-quotas
2026-05-12 15:37:33,756 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,759 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,760 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,765 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing
2026-05-12 15:37:33,765 - distributed.nanny - WARNING - Worker process still alive after 4.0 seconds, killing

┆Issue is synchronized with this Jira Task

[mikewilli]

The reason we flush logs to BQ is so that we can show them on the dashboard that jetstream preview creates. I think this is useful, but I suppose we could make that a parameter here (or re-use the --log-to-bigquery parameter that we use for other jetstream commands, but I think we'd want it to default to True for preview which makes it slightly tricky. The error is also safe to ignore if you don't care about the logs making their way to BigQuery, since we just print it and move on.

That said, this does seem bad in general -- we shouldn't be hitting rate limits on writing logs to BQ. I'll take a look.

[mikewilli]

I'm fairly sure that #3190 fixes this, but it's a bit tricky to test. If you are still seeing this issue on the latest jetstream (2026.5.2) then please feel free to re-open this.