Possible thread leak

[chmoder]

Hi,

We investigated a bug about a process running out of threads in a long lasting process with thousands of requests being made. After researching we narrowed down the cause to key.get; we also can reproduce with query.fetch.

The below functions are ran in threading.py start when key.get is called. It seems like one or more of them may have a thread that needs to close at the end of the request?

Thank you for the help!

<function _run_channel_spin_thread.<locals>.channel_spin at 0x1119ec048>
<grpc._plugin_wrapping._Plugin object at 0x1119fc9e8>
<function _worker at 0x1119afae8>

Environment details

1. python-ndb
2. OS X Catalina
3. python 3.7.3
4. google-cloud-ndb==1.0.0

Steps to reproduce

1. invoke key.get()

Code example

 from google.cloud import ndb


class House(ndb.Model):
    room = ndb.StringProperty(default='kitchen')


class HouseHandler(webapp2.RequestHandler):
    def get(self):
        client = ndb.Client(project='project_id')
        with client.context():
            k = ndb.Key(House, 5508237726318592)
            o = k.get()

app = webapp2.WSGIApplication(
    [webapp2.Route('/v1/houses/1', HouseHandler)],
    debug=True
)

if __name__ == '__main__':
    from paste import httpserver

    httpserver.serve(app, host='127.0.0.1', port='8080')

When making the request, watch the number of threads in the python process. You should see it climb for each request. In the screenshot below it is the highlighted process with 19 threads.

ab -n 1 -H http://127.0.0.1:8080/v1/houses/1

Stack Trace

INFO:root:default: "POST /cloud_task_queue HTTP/1.1" 200 4 0.73653s
[2020-02-21 15:09:44 -0600] [3172] [DEBUG] POST /cloud_task_queue
ERROR:grpc._plugin_wrapping:AuthMetadataPluginCallback "<google.auth.transport.grpc.AuthMetadataPlugin object at 0x12d9d0978>" raised exception!
Traceback (most recent call last):
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_plugin_wrapping.py", line 79, in __call__
    callback_state, callback))
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/auth/transport/grpc.py", line 87, in __call__
    future = self._pool.submit(self._get_authorization_headers, context)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 160, in submit
    self._adjust_thread_count()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 181, in _adjust_thread_count
    t.start()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new threadΩ
ERROR:root:Unhandled exception: can't start new thread
Traceback (most recent call last):
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/logger/request_logger_clients/base_client.py", line 277, in process_exception
    raise exception
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/webapp2.py", line 604, in dispatch
    return method(*args, **kwargs)
  File "/Users/tcross/development/csgapi/handlers/cloud_task_handler.py", line 10, in post
    CloudTaskQueue.process_cloud_task(request_body)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/queues/cloud_task_queue.py", line 51, in process_cloud_task
    execute_task(func, **request_body)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/execute.py", line 14, in execute_task
    func(**kwargs)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/deferred_pickle/deferred.py", line 117, in run_pickle
    func_, args_ = manager.get(key)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/deferred_pickle/db_interface/ndb/ndb_manager.py", line 37, in get
    d_pickle = key.get()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_options.py", line 89, in wrapper
    return wrapped(*pass_args, **kwargs)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/utils.py", line 110, in positional_wrapper
    return wrapped(*args, **kwds)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/key.py", line 811, in get
    return self.get_async(_options=_options).result()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 190, in result
    self.check_success()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 137, in check_success
    raise self._exception
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/key.py", line 896, in get
    entity_pb = yield _datastore_api.lookup(self._key, _options)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_datastore_api.py", line 169, in lookup
    entity_pb = yield batch.add(key)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_retry.py", line 78, in retry_wrapper
    raise error
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_retry.py", line 73, in retry_wrapper
    result = yield result
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 312, in _advance_tasklet
    yielded = self.generator.send(send_value)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_datastore_api.py", line 108, in rpc_call
    call = method.future(request, timeout=timeout)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 595, in future
    (operations,), event_handler, self._context)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 863, in create
    _run_channel_spin_thread(state)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 827, in _run_channel_spin_thread
    channel_spin_thread.start()
  File "src/python/grpcio/grpc/_cython/_cygrpc/fork_posix.pyx.pxi", line 119, in grpc._cython.cygrpc.ForkManagedThread.start
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
ERROR    2020-02-21 15:09:44 _plugin_wrapping.py:83] Traceback (most recent call last):
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_plugin_wrapping.py", line 79, in __call__
    callback_state, callback))
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/auth/transport/grpc.py", line 87, in __call__
    future = self._pool.submit(self._get_authorization_headers, context)
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 160, in submit
    self._adjust_thread_count()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/concurrent/futures/thread.py", line 181, in _adjust_thread_count
    t.start()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
ERROR    2020-02-21 15:09:44 base_client.py:281] Traceback (most recent call last):
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/logger/request_logger_clients/base_client.py", line 277, in process_exception
    raise exception
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/webapp2.py", line 604, in dispatch
    return method(*args, **kwargs)
  File "/Users/tcross/development/csgapi/handlers/cloud_task_handler.py", line 10, in post
    CloudTaskQueue.process_cloud_task(request_body)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/queues/cloud_task_queue.py", line 51, in process_cloud_task
    execute_task(func, **request_body)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/execute.py", line 14, in execute_task
    func(**kwargs)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/deferred_pickle/deferred.py", line 117, in run_pickle
    func_, args_ = manager.get(key)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/csg/tasks/deferred_pickle/db_interface/ndb/ndb_manager.py", line 37, in get
    d_pickle = key.get()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_options.py", line 89, in wrapper
    return wrapped(*pass_args, **kwargs)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/utils.py", line 110, in positional_wrapper
    return wrapped(*args, **kwds)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/key.py", line 811, in get
    return self.get_async(_options=_options).result()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 190, in result
    self.check_success()
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 137, in check_success
    raise self._exception
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/key.py", line 896, in get
    entity_pb = yield _datastore_api.lookup(self._key, _options)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_datastore_api.py", line 169, in lookup
    entity_pb = yield batch.add(key)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 309, in _advance_tasklet
    self.generator.throw(type(error), error, traceback)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_retry.py", line 78, in retry_wrapper
    raise error
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_retry.py", line 73, in retry_wrapper
    result = yield result
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/tasklets.py", line 312, in _advance_tasklet
    yielded = self.generator.send(send_value)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/google/cloud/ndb/_datastore_api.py", line 108, in rpc_call
    call = method.future(request, timeout=timeout)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 595, in future
    (operations,), event_handler, self._context)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 863, in create
    _run_channel_spin_thread(state)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 827, in _run_channel_spin_thread
    channel_spin_thread.start()
  File "src/python/grpcio/grpc/_cython/_cygrpc/fork_posix.pyx.pxi", line 119, in grpc._cython.cygrpc.ForkManagedThread.start
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread
INFO:root:default: "POST /cloud_task_queue HTTP/1.1" 500 90 2.07239s
[2020-02-21 15:09:46 -0600] [3172] [DEBUG] POST /cloud_task_queue
INFO     2020-02-21 15:09:46 base_client.py:234] default: "POST /cloud_task_queue HTTP/1.1" 500 90 2.07239s
Exception in thread Thread-54699:
Traceback (most recent call last):
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 917, in _bootstrap_inner
    self.run()
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 865, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/tcross/development/virtual-environments/csgapi/lib/python3.7/site-packages/grpc/_channel.py", line 815, in channel_spin
    event = state.channel.next_call_event()
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 464, in grpc._cython.cygrpc.Channel.next_call_event
  File "src/python/grpcio/grpc/_cython/_cygrpc/channel.pyx.pxi", line 150, in grpc._cython.cygrpc._next_call_event
  File "src/python/grpcio/grpc/_cython/_cygrpc/completion_queue.pyx.pxi", line 67, in grpc._cython.cygrpc._latent_event
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 56, in grpc._cython.cygrpc._get_metadata
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 31, in grpc._cython.cygrpc._spawn_callback_async
  File "src/python/grpcio/grpc/_cython/_cygrpc/credentials.pyx.pxi", line 22, in grpc._cython.cygrpc._spawn_callback_in_thread
  File "src/python/grpcio/grpc/_cython/_cygrpc/fork_posix.pyx.pxi", line 119, in grpc._cython.cygrpc.ForkManagedThread.start
  File "/usr/local/Cellar/python/3.7.3/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 847, in start
    _start_new_thread(self._bootstrap, ())
RuntimeError: can't start new thread

possibly related to #336

[chrisrossi]

@chmoder #336 seems to be specifically related to App Engine. Are you running this in App Engine or something newer?

[chmoder]

This happens on my computer locally - I can see the thread count go up in a python process with each call to key.get.

We also use app engine standard37 and flexible python37.

[chrisrossi]

I'm trying to reproduce this and haven't so far.

Here's a sample app I'm using to test:
https://gist.github.com/chrisrossi/352efe8e4133de59e0ddb818c4138ad9

Here is a transcript of a session:

(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/threads
<_MainThread(MainThread, started 139848366495552)>
<Thread(Thread-52, started daemon 139848297903872)>
(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/make/hay
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>Redirecting...</title>
<h1>Redirecting...</h1>
<p>You should be redirected automatically to target URL: <a href="/show/5676714420076544">/show/5676714420076544</a>.  If not click the link.(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/show/5676714420076544
{'key': Key('SomeThing', 5676714420076544), 'properties': {'foo': 'hay'}}
(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/threads
<_MainThread(MainThread, started 139848366495552)>
<Thread(Thread-59, started daemon 139848297903872)>
(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/show/5676714420076544
{'key': Key('SomeThing', 5676714420076544), 'properties': {'foo': 'hay'}}
(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ curl http://localhost:5000/threads
<_MainThread(MainThread, started 139848366495552)>
<Thread(Thread-61, started daemon 139848297903872)>
(flask-ndb) chris@spirit:~/proj/google/flask-ndb$ 

The threads remain constant, despite the call to key.get().

One difference is my sample app creates the client only once and encapsulates the context in a middleware. I wonder if you still see this issue using the middleware in my sample app rather than instantiating a client in a Flask view?

Also, you can try upgrading to NDB 1.0.1.

[chmoder]

Thanks for the suggestion @chrisrossi! There is not a lot of good examples for how to properly make a middleware for webapp2. I'm trying to fit it into my sprint - I will let you know how it goes when we have time.

[chrisrossi]

Here is your example adapted to include the middleware:
https://gist.github.com/chrisrossi/559413684cebd0fbe6f50bba9101c48f

I've done some testing with this and haven't found a thread leak. I'll do a little more stress testing before putting this away, though.

[chrisrossi]

Ok, I just realized the cache was masking the issue. I have been able to reproduce the thread leak. Since NDB doesn't create threads, it's obviously an issue with something we're using. It's unclear at this time if there will be a solution on the NDB side, or if it will wind up being something for the maintainers of a library we depend on.

[chrisrossi]

I've also confirmed that the leak occurs with webapp2 but not Flask, which makes it more likely to be related to/the same as #336.

[andrewsg]

That's very interesting!! I wonder if this category of problems is an issue with webapp2 specifically and not GAE Py2 in general. If so it would be much easier to fix. Time permitting, let's try to verify that #336 is reproducible on webapp2 but not on flask, and, especially, if both this and #336 are reproducible locally, outside of GAE Py2.

[chmoder]

Thanks for the extra effort! I just verified your results; We still see the thread leak with the middleware as well.

More context: over the years we have seen memory leaks and thread leaks. Like variables not being destroyed after the handler finishes. Which leads to critical OOM errors on GAE.

My hunch is something to do with context managers. Like __exit__ not being called. Please feel free to ask for information, testing, or any kind of help!