wait_for_idle hangs on juju 3.6

[nsklikas]

Description

We are in the process of migrating our charms from juju 3.4 to 3.6. Most of the charm integration tests are passing without an issue, but integration tests on our charms are timing out when they call wait_for_idle (e.g. see https://github.com/canonical/iam-bundle/actions/runs/12180204254/job/33974029227).

When running it locally, I can see that all charms reach an active status, but the python code is stuck waiting for FullStatus to return.

This behavior is consistently reproducible, the tests are always failing after the 3rd iteration of wait_for_idle (could it be a rate limiting issue?). Tests on our other charms work fine, so my guess would be that it has to do something with calling wait_for_idle for many charms.

Urgency

Blocker for our release

Python-libjuju version

3.6.0.0

Juju version

3.6

Reproduce / Test

- install `microk8s=1.31-strict/stable` and `juju=3.6/stable`
- fetch `https://github.com/canonical/iam-bundle` 
- on `main` run `tox -e integration`

[dimaqq]

Looking at the linked job log, I see this:

WARNING  juju.client.connection:connection.py:579 RPC: Connection closed, reconnecting
ERROR    juju.client.connection:connection.py:587 RPC: Automatic reconnect failed

The whole test is done (failed) in 2.5 minutes, that's way too short for "it hangs" description.

Is the linked test log correct?

Is there anything in the test setup that would explain websocket connection dying, like if you are testing juju outage on purpose?

I see that you're already dumping Juju logs, that great! :)

Given that your model is rather large and it's just starting up, I can think of two possible issues:

1. websockets max frame size exceeded, see https://matrix.to/#/!xdClnUGkurzjxqiQcN:ubuntu.com/$49wTojb4U3svWiepHWj0L6M2BgpD57TGZQDZPzOSQVY?via=ubuntu.com&via=matrix.org&via=matrix.hya.sk
2. mongodb max transaction size exceeded, see https://bugs.launchpad.net/juju/+bug/2074315

The earlier can be solved client-side, I've opened #1237 to provide canonical advice.

The latter won't realistically be solved until Juju 4.0

[dimaqq]

I've also opened #1238 to log the earlier situation clearly.

Now that I think about it, the default in this library should match current internal juju limit, which is ~16MB BSON-encoded monogdb transaction, so perhaps ~40MB JSON to be on the safe side?

[nsklikas]

websockets max frame size exceeded

Not sure if I understand correctly. If it is a websocket issue, wouldn't it be happening regardless of juju versions? The size of the data transferred through the socket shouldn't have changed that much. Also the error I see is slightly different from the one linked in the chat.

[nsklikas]

So weird thing, there was a new microk8s version released today and after upgrading to that, the tests started passing. Not sure why, but I guess we can close this issue (?). But I have to say I don't feel comfortable with having no idea about why our tests are this flaky.

[dimaqq]

Well that’s only a conjecture, what could cause disconnect and failure to reconnect.

it could be something completely different

[dimaqq]

I've release pypi:juju 3.6.1.0 where the default websocket message limit has been bumped up to more is realistically needed. Please check if that solves your issue.

[nsklikas]

As I mentioned above, the issue was solved when a new microk8s version was released. But thanks for reaching out anyway.