[GAE python 2.7] query().filter().fetch() sometimes spawns threads preventing webapp2 requests finish within a deadline

[VladNF]

Certain cloud-ndb queries causes grpc to spawn threads that prevent accomplishing webapp2/django requests as GAE standard env for python 2 joins all request threads.
Requests failes with the following exception

Thread running after request. Creation traceback:
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 80, in Run
    target(*args, **kw)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 794, in __bootstrap
    self.__bootstrap_inner()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 821, in __bootstrap_inner
    self.run()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 774, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_channel.py", line 678, in channel_spin
    event = state.completion_queues[key].poll()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/third_party/grpcio-1.0.0/grpc/_plugin_wrapping.py", line 107, in __call__
    wrapped_context, AuthMetadataPluginCallback(wrapped_cygrpc_callback))
  File "/base/data/home/apps/.../lib/google/auth/transport/grpc.py", line 87, in __call__
    future = self._pool.submit(self._get_authorization_headers, context)
  File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 137, in submit
    self._adjust_thread_count()
  File "/base/data/home/apps/.../lib/concurrent/futures/thread.py", line 159, in _adjust_thread_count
    t.start()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_dist/lib/python2.7/threading.py", line 749, in start
    _start_new_thread(self.__bootstrap, ())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/ebb3af67a06047b6/python27/python27_lib/versions/1/google/appengine/runtime/runtime.py", line 83, in StartNewThread
    return base_start_new_thread(Run, ())

The following shows one example that after re-writing it in a get-by-id manner has no such issue.

Model1.query(Model1.repeatedKeyProp.IN(keys)).fetch()

Shutting down ThreadPoolExecutor from concurrent.future helped to localize the issue caused by additional threads caused by querying data.

Any ideas how to coupe with it? We cannot just move everything to Python 3 env in a one step, both envs are going to co-exist for sometime.

[andrewsg]

Thanks for this report. To clarify, in this case your query was fired and completed, and results delivered, entirely within the request; then, at the end of the request, there was an error? Or is the problem that the query was not completed at all and errored out immediately, even though the request had not completed?

[VladNF]

Yes, a query is completed and results are delivered. The things is AppEngine runtime joins all threads spawned within a request, somehow those additional threads after query with IN,AND etc are not finished. I tried to figure out what are these threads and seems like they are from grpc concurrent.future.thread.ThreadPoolExecutor. In our code we do not start any background thread, just using cloud-ndb to handle data (worked fine with AppEngine built-in ndb module before migration to cloud-ndb).

[oliverchang]

We also ran into this (google/clusterfuzz#1298 (comment)).

The workaround for us is to manually invoke the GC at the end of the request, which seems to clean up stray threads.

[VladNF]

@oliverchang thank you so much, seems to work for us

[chrisrossi]

Reopening this so we can investigate whether it makes sense to do something on the NDB side.

@oliverchang mentions this happens with MultiQuery but not other calls? Is it worth exploring why?

Would it make sense to add a call to gc.collect somewhere in NDB?