Intermittently fail when pool release

[hyeongguen-song]

Describe the bug
When after upgrade tortoise-orm version to 0.19.0 intermittently occur exception after update_or_create(not raw query)

Stack trace
asyncpg.exceptions._base:InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7fee288912e0> 0x7fee222c5d00> is not a member of this pool

Traceback:
...
File "/x/.venv/lib/python3.9/site-packages/tortoise/models.py", line 1098, in update_or_create
File "/x/.venv/lib/python3.9/site-packages/tortoise/backends/base/client.py", line 283, in __aexit__
File "/x/.venv/lib/python3.9/site-packages/asyncpg/pool.py", line 812, in release

Additional context
When I use 0.17.7(before upgrade), never met this
I think it is related to #1001 .. (not sure)

[long2ice]

Strange, need more feedback

[hyeongguen-song]

@long2ice
on FastAPI BackgroundTasks with gunicorn worker,
It happened about 1 in 20,000 times.

async def blahbalh():
    await MyModel.update_or_create(status=0)
    # do awaitable things, sometimes it took 5~6 minutes
    # download csv from s3
    # update other model (with new connection and execute raw query) << I suspect this was the problem, but not sure
    await MyModel.update_or_create(status=200)

Full trace

"""
Stack trace
asyncpg.exceptions._base:InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f7a30148740> 0x7f7a300f4070> is not a member of this pool
Traceback(most recent call last):
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 231, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/app/base.py", line 72, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 202, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/workers.py", line 66, in init_process
File "/deploy/blue/.venv/lib/python3.9/site-packages/gunicorn/workers/base.py", line 142, in init_process
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/workers.py", line 84, in run
File "/.pyenv/versions/3.9.4/lib/python3.9/asyncio/runners.py", line 44, in run
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/protocols/http/httptools_impl.py", line 375, in run_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py", line 75, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 387, in nr_async_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/common/async_proxy.py", line 148, in next
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/common/async_proxy.py", line 120, in send
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 70, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/fastapi/applications.py", line 208, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/api/asgi_application.py", line 330, in nr_async_asgi
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/applications.py", line 112, in call
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/errors.py", line 159, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/hooks/framework_starlette.py", line 121, in middleware_wrapper
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 25, in call
File "/deploy/blue/myapp/app/config/middleware.py", line 14, in dispatch
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 45, in call_next
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/base.py", line 38, in coro
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/hooks/framework_starlette.py", line 121, in middleware_wrapper
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 18, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/gzip.py", line 35, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/newrelic/hooks/framework_starlette.py", line 121, in middleware_wrapper
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/middleware/cors.py", line 78, in call
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 82, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/exceptions.py", line 71, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/routing.py", line 580, in call
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/routing.py", line 241, in handle
File "/deploy/blue/.venv/lib/python3.9/site-packages/starlette/routing.py", line 52, in app
File "/deploy/blue/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 226, in app
File "", line 5, in wrapper
File "/deploy/blue/.venv/lib/python3.9/site-packages/fastapi/routing.py", line 159, in run_endpoint_function
File "/deploy/blue/myapp/app/sync/abc/routes.py", line 20, in sync_from_abc
File "/deploy/blue/myapp/app/sync/log/models.py", line 30, in put
File "/deploy/blue/.venv/lib/python3.9/site-packages/tortoise/models.py", line 1099, in update_or_create
File "/deploy/blue/.venv/lib/python3.9/site-packages/tortoise/models.py", line 1061, in get_or_create
File "/deploy/blue/.venv/lib/python3.9/site-packages/tortoise/backends/base/client.py", line 283, in aexit
File "/deploy/blue/.venv/lib/python3.9/site-packages/asyncpg/pool.py", line 812, in release
"""

[blazing-gig]

@hyeongguen-song Hey, could you please paste a snapshot of the code that caused this and potentially steps to reproduce the issue ? Also, would like to know whether you're using async features of fastAPI or sync ?

[hyeongguen-song]

@hyeongguen-song Hey, could you please paste a snapshot of the code that caused this and potentially steps to reproduce the issue ? Also, would like to know whether you're using async features of fastAPI or sync ?

@blazing-gig
Sorry for inaccurate description and code snippet🙏
I did my best to rewrite it
I'm using async of FastAPI.

I can't reproduce it myself.
I don't know exactly why it happens, and it doesn't happen every time.

[whitedemong]

#1213

[artefom]

Also getting this error. Do not know how to reproduce

Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f6e4075fa00> 0x7f6e405b01f0> is not a member of this pool

Full stack trace:

InterfaceError: Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7f6e4075fa00> 0x7f6e405b01f0> is not a member of this pool
  File "starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "starlette/middleware/errors.py", line 162, in __call__
    await self.app(scope, receive, _send)
  File "starlette/middleware/sessions.py", line 86, in __call__
    await self.app(scope, receive, send_wrapper)
  File "starlette/middleware/base.py", line 106, in __call__
    response = await self.dispatch_func(request, call_next)
  File "starlette_prometheus/middleware.py", line 57, in dispatch
    raise e from None
  File "starlette_prometheus/middleware.py", line 53, in dispatch
    response = await call_next(request)
  File "starlette/middleware/base.py", line 80, in call_next
    raise app_exc
  File "starlette/middleware/base.py", line 69, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "starlette/middleware/cors.py", line 84, in __call__
    await self.app(scope, receive, send)
  File "starlette/middleware/exceptions.py", line 79, in __call__
    raise exc
  File "starlette/middleware/exceptions.py", line 68, in __call__
    await self.app(scope, receive, sender)
  File "fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 706, in __call__
    await route.handle(scope, receive, send)
  File "starlette/routing.py", line 276, in handle
    await self.app(scope, receive, send)
  File "starlette/routing.py", line 66, in app
    response = await func(request)
  File "fastapi/routing.py", line 237, in app
    raw_response = await run_endpoint_function(
  File "fastapi/routing.py", line 163, in run_endpoint_function
    return await dependant.call(**values)
  File "broker/api/specification.py", line 138, in _handle_exceptions
    result = await func(*args, **kwargs)
  File "broker/api/implementation.py", line 889, in poll
    worker = await _update_worker_info(report)
  File "broker/api/implementation.py", line 83, in _update_worker_info
    worker, _ = await models.update_or_create_worker(
  File "broker/models.py", line 143, in update_or_create_worker
    return await Worker.update_or_create(
  File "tortoise/models.py", line 1104, in update_or_create
    async with in_transaction(connection_name=db.connection_name) as connection:
  File "tortoise/backends/base/client.py", line 283, in __aexit__
    await self.connection._parent._pool.release(self.connection._connection)
  File "asyncpg/pool.py", line 858, in release
    raise exceptions.InterfaceError(

Package versions:
fastapi: 0.89.1
asyncpg: 0.27.0
tortoise-orm: 0.19.3
starlette: 0.22.0
uvloop: 0.17.0

The error occurs in one of the api route handlers and is very rare.

No gunicorn is used.
The server itself is executed like that
in main.py

async def main():
    config = uvicorn.Config(
        app,
        host=host,
        port=port,
        log_config=LOGGING_CONFIG,
    )
    api_server = uvicorn.Server(config)
    await api_server.serve()
    
if __name__ == "__main__":
    asyncio.run(main())

and then
python main.py

In my application error occurs in this code:

    return await Worker.update_or_create( # this raises InterfaceError
        worker_id=worker_id,
        defaults={
            "strategy": strategy,
            "last_polled": last_polled,
            "max_instances": max_instances,
            "poll_interval_seconds": poll_interval_seconds,
            "terminated": terminated,
            "shutting_down": shutting_down,
            "hostname": hostname,
        },
    )

[hyeongguen-song]

After upgrade 0.17.7 to 0.20.0,
I never meet this again

[markmiscavage]

I'm on 0.20.1 and still get this somewhat frequently (about 2 a day in dev environment). I think for me it primarily occurs on FastAPI backgrounded tasks that call get_or_create().

InterfaceError Pool.release() received invalid connection: <PoolConnectionProxy <asyncpg.connection.Connection object at 0x7fcddd5dadc0> 0x7fcdddd0f610> is not a member of this pool

tortoise-orm==0.20.1
uvicorn ==0.18.3
asyncpg==0.26.0

[abondar]

You can try upgrading to 0.21.3, we had some fixes to get_or_create, although I am not sure how this error could have occurred in first place

Could you also share your configuration and how does your tasks look? Are all your tasks and views async or there are somewhere sync functions?