ext_proc fuzzer test trigger ENVOY_BUG when clear route cache by yanjunxiang-google · Pull Request #27657 · envoyproxy/envoy

yanjunxiang-google · 2023-05-26T19:11:29Z

This PR is to address an ENVOY_BUG crash exposed by ext_proc fuzzer test reported by:

https://clusterfuzz.corp.google.com/testcase-detail/4756668218736640

The steps to trigger this ENVOY_BUG is below:

When handling upstream response body message, ext_proc filter sends the response body to the ext_proc server.
The ext_proc server sends back a body response with header mutation and set the clear_route_cache flag.
Such clear_route_cache flag will trigger function call into connection_manager to clear route cache, which is set to be blocked in the stage when connection_manager is processing the upstream header response. Thus the ENVOY_BUG is fired.

Commit Message:
Additional Description:
Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

…tream response Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

KBaichoo · 2023-05-29T12:17:00Z

/assign @tyxia

As codeowner

/wait

For CI

tyxia

Thanks for working on this! I think it is the right fix overall.

Also, cc @wbpcode, I think before PR #26045, clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path ) rather than triggering the ENVOY_BUG.

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

…roc_clear_route_cache Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

yanjunxiang-google · 2023-05-30T19:28:11Z

/assign @yanavlasov @htuch @mpwarres @stevenzzzz

yanjunxiang-google · 2023-05-30T19:30:54Z

This is the OSS fuzzer report:
https://clusterfuzz.corp.google.com/testcase-detail/4756668218736640

Here is the crash trace back decode:

Command: /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test

| Time ran: 57.5866904258728
|
| INFO: google3-libFuzzer:
| google3 flags should be passed as --flag=value
| libFuzzer flags should be passed as -flag=value
| INFO: Running in fuzzing mode, google3 logging disabled
| INFO: Use --minloglevel=0 to re-enable logging
| INFO: found LLVMFuzzerCustomMutator (0x5aa5109285c0). Disabling -len_control by default.
| INFO: Running with entropic power schedule (0xFF, 100).
| INFO: Seed:4231534795
| INFO: Loaded 1 modules (1767040 inline 8-bit counters): 1767040 [0x5aa51a28ec50, 0x5aa51a43e2d0),
| INFO: Loaded 1 PC tables (1767040 PCs): 1767040 [0x5aa51a43e2d0,0x5aa51bf34ad0),
| /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test: Running 1 inputs 100 time(s) each.
| Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
| MemorySanitizer:DEADLYSIGNAL
| ==1458622==ERROR: MemorySanitizer: ABRT on unknown address 0x0539001641be (pc 0x79b7c07162fa bp 0x79b7b4756b60 sp 0x79b7b47569b8 T1458646)
| #0 0x79b7c07162fa in raise (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d)
| #1 0x5aa516aa81a3 in Envoy::Http::ConnectionManagerImpl::ActiveStream::routeCacheBlocked() const third_party/envoy/src/source/common/http/conn_manager_impl.h:310:7
| #2 0x5aa516ab95a2 in Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc:1966:7
| #3 0x5aa516ab99f3 in non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc
| #4 0x5aa510a2ad96 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::clearRouteCache(envoy::service::ext_proc::v3::CommonResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:375:47
| #5 0x5aa510a2f2f1 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::handleBodyResponse(envoy::service::ext_proc::v3::BodyResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:328:7
| #6 0x5aa510a153b6 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc:628:41
| #7 0x5aa510a18c93 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc
| #8 0x5aa510a01537 in onReceiveMessage third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc:53:14
| #9 0x5aa510a01537 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorStreamImpl::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc
| #10 0x5aa510a01fb8 in Envoy::Grpc::AsyncStreamCallbacksenvoy::service::ext_proc::v3::ProcessingResponse::onReceiveMessageRaw(std::__msan::unique_ptr<Envoy::Buffer::Instance, std::__msan::default_deleteEnvoy::Buffer::Instance>&&) third_party/envoy/src/source/common/grpc/typed_async_client.h:98:5
| #11 0x5aa51314e664 in Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc:153:21
| #12 0x5aa51314e836 in non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc
| #13 0x5aa51316009e in Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/async_client_impl.cc:142:21
| #14 0x5aa51379db66 in Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc:1606:15
| #15 0x5aa51379e016 in non-virtual thunk to Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc
| #16 0x5aa5137c7ad9 in Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.cc:311:11
| #17 0x5aa5137dc48f in Envoy::Router::UpstreamRequestFilterManagerCallbacks::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.h:273:23
| #18 0x5aa516af1223 in Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState) third_party/envoy/src/source/common/http/filter_manager.cc:1356:29
| #19 0x5aa516af19d9 in encodeData third_party/envoy/src/source/common/http/filter_manager.cc:463:11
| #20 0x5aa516af19d9 in non-virtual thunk to Envoy::Http::ActiveStreamDecoderFilter::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/filter_manager.cc
| #21 0x5aa5137ebda8 in Envoy::Router::UpstreamCodecFilter::CodecBridge::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_codec_filter.cc:162:23
| #22 0x5aa5130bc9b4 in Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_wrappers.h:32:12
| #23 0x5aa51701a99b in Envoy::Http::Http2::ConnectionImpl::StreamImpl::decodeData() third_party/envoy/src/source/common/http/http2/codec_impl.cc:519:17
| #24 0x5aa51703799f in Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1175:13
| #25 0x5aa517057040 in operator() third_party/envoy/src/source/common/http/http2/codec_impl.cc:1695:64
| #26 0x5aa517057040 in Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_5::__invoke(nghttp2_session*, nghttp2_frame const*, void*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1694:19
| #27 0x5aa517380334 in external_quiche_http2::adapter::CallbackVisitor::OnDataForStream(int, std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/callback_visitor.cc:259:14
| #28 0x5aa51734e773 in external_quiche_http2::adapter::callbacks::OnDataChunk(nghttp2_session*, unsigned char, int, unsigned char const*, unsigned long, void*) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_callbacks.cc:277:32
| #29 0x5aa5173b2b51 in nghttp2_session_mem_recv third_party/nghttp2/src/lib/nghttp2_session.c:7170:18
| #30 0x5aa51734aefd in external_quiche_http2::adapter::NgHttp2Session::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_session.cc:35:10
| #31 0x5aa517344b19 in external_quiche_http2::adapter::NgHttp2Adapter::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_adapter.cc:74:45
| #32 0x5aa51703170f in Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc:956:20
| #33 0x5aa5170342b0 in virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc
| #34 0x5aa513801292 in Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/codec_client.cc:173:33
| #35 0x5aa5138073e1 in Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_client.h:211:15
| #36 0x5aa5171af927 in Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) third_party/envoy/src/source/common/network/filter_manager_impl.cc:95:48
| #37 0x5aa5171b0097 in Envoy::Network::FilterManagerImpl::onRead() third_party/envoy/src/source/common/network/filter_manager_impl.cc:105:3
| #38 0x5aa5171887a0 in Envoy::Network::ConnectionImpl::onRead(unsigned long) third_party/envoy/src/source/common/network/connection_impl.cc:345:19
| #39 0x5aa51719926f in Envoy::Network::ConnectionImpl::onReadReady() third_party/envoy/src/source/common/network/connection_impl.cc:651:5
| #40 0x5aa5171929b8 in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) third_party/envoy/src/source/common/network/connection_impl.cc:602:5
| #41 0x5aa5171a7f05 in operator() third_party/envoy/src/source/common/network/connection_impl.cc:94:54
| #42 0x5aa5171a7f05 in __invoke<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
| #43 0x5aa5171a7f05 in __call<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
| #44 0x5aa5171a7f05 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
| #45 0x5aa5171a7f05 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__msan::unique_ptr<Envoy::Network::ConnectionSocket, std::__msan::default_deleteEnvoy::Network::ConnectionSocket>&&, std::__msan::unique_ptr<Envoy::Network::TransportSocket, std::__msan::default_deleteEnvoy::Network::TransportSocket>&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
| #46 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
| #47 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
| #48 0x5aa517130576 in operator() third_party/envoy/src/source/common/event/dispatcher_impl.cc:184:9
| #49 0x5aa517130576 in __invoke<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
| #50 0x5aa517130576 in __call<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
| #51 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
| #52 0x5aa517130576 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__msan::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
| #53 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
| #54 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
| #55 0x5aa51713710d in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) third_party/envoy/src/source/common/event/file_event_impl.cc:161:3
| #56 0x5aa5171377fb in operator() third_party/envoy/src/source/common/event/file_event_impl.cc:82:16
| #57 0x5aa5171377fb in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) third_party/envoy/src/source/common/event/file_event_impl.cc:66:7
| #58 0x5aa5175af2c7 in event_persist_closure third_party/libevent/src/event.c:1639:9
| #59 0x5aa5175af2c7 in event_process_active_single_queue third_party/libevent/src/event.c:1698:4
| #60 0x5aa5175a2132 in event_process_active third_party/libevent/src/event.c:1799:9
| #61 0x5aa5175a2132 in event_base_loop third_party/libevent/src/event.c:2041:12
| #62 0x5aa517574e53 in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/libevent_scheduler.cc:60:3
| #63 0x5aa51712739a in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/dispatcher_impl.cc:299:19
| #64 0x5aa511f0bbce in Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&) third_party/envoy/src/source/server/worker_impl.cc:148:16
| #65 0x5aa511f0d318 in operator() third_party/envoy/src/source/server/worker_impl.cc:115:42
| #66 0x5aa511f0d318 in __invoke<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
| #67 0x5aa511f0d318 in __call<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
| #68 0x5aa511f0d318 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
| #69 0x5aa511f0d318 in void std::__msan::__function::__policy_invoker<void ()>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&)::$_0, void ()>>(std::__msan::__function::__policy_storage const*) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
| #70 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
| #71 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
| #72 0x5aa5178a74d2 in operator() third_party/envoy/src/source/common/common/posix/thread_impl.cc:49:11
| #73 0x5aa5178a74d2 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__msan::function<void ()>, std::__msan::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::__invoke(void*) third_party/envoy/src/source/common/common/posix/thread_impl.cc:48:9
| #74 0x79b7c0838d62 in start_thread (/usr/grte/v5/lib64/libpthread.so.0+0xbd62) (BuildId: 598ce51a74ecc39ad11a39e9c549e191)
|
| MemorySanitizer can not provide additional info.
| SUMMARY: MemorySanitizer: ABRT (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d) in raise
| ==1458622==ABORTING

Command: /mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test -runs=100 /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
Time ran: 57.5866904258728
INFO: google3-libFuzzer:
google3 flags should be passed as --flag=value
libFuzzer flags should be passed as -flag=value
INFO: Running in fuzzing mode, google3 logging disabled
INFO: Use --minloglevel=0 to re-enable logging
INFO: found LLVMFuzzerCustomMutator (0x5aa5109285c0). Disabling -len_control by default.
INFO: Running with entropic power schedule (0xFF, 100).
INFO: Seed:4231534795
INFO: Loaded 1 modules (1767040 inline 8-bit counters): 1767040 [0x5aa51a28ec50, 0x5aa51a43e2d0),
INFO: Loaded 1 PC tables (1767040 PCs): 1767040 [0x5aa51a43e2d0,0x5aa51bf34ad0),
/mnt/scratch0/clusterfuzz/bot/builds/clusterfuzz-g3-builds_third_party-envoy-src-test-extensions-filters-http-ext_proc_libfuzzer_memory_ext_proc_grpc_fuzz_test_77651789446b3c3a04b9f492ff141f003d437347/revisions/ext_proc_grpc_fuzz_test: Running 1 inputs 100 time(s) each.
Running: /mnt/scratch0/clusterfuzz/bot/inputs/fuzzer-testcases/input.test
MemorySanitizer:DEADLYSIGNAL
==1458622==ERROR: MemorySanitizer: ABRT on unknown address 0x0539001641be (pc 0x79b7c07162fa bp 0x79b7b4756b60 sp 0x79b7b47569b8 T1458646)
#0 0x79b7c07162fa in raise (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d)
#1 0x5aa516aa81a3 in Envoy::Http::ConnectionManagerImpl::ActiveStream::routeCacheBlocked() const third_party/envoy/src/source/common/http/conn_manager_impl.h:310:7
#2 0x5aa516ab95a2 in Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc:1966:7
#3 0x5aa516ab99f3 in non-virtual thunk to Envoy::Http::ConnectionManagerImpl::ActiveStream::clearRouteCache() third_party/envoy/src/source/common/http/conn_manager_impl.cc
#4 0x5aa510a2ad96 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::clearRouteCache(envoy::service::ext_proc::v3::CommonResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:375:47
#5 0x5aa510a2f2f1 in Envoy::Extensions::HttpFilters::ExternalProcessing::ProcessorState::handleBodyResponse(envoy::service::ext_proc::v3::BodyResponse const&) third_party/envoy/src/source/extensions/filters/http/ext_proc/processor_state.cc:328:7
#6 0x5aa510a153b6 in Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc:628:41
#7 0x5aa510a18c93 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::Filter::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/ext_proc.cc
#8 0x5aa510a01537 in onReceiveMessage third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc:53:14
#9 0x5aa510a01537 in non-virtual thunk to Envoy::Extensions::HttpFilters::ExternalProcessing::ExternalProcessorStreamImpl::onReceiveMessage(std::__msan::unique_ptr<envoy::service::ext_proc::v3::ProcessingResponse, std::__msan::default_deleteenvoy::service::ext_proc::v3::ProcessingResponse>&&) third_party/envoy/src/source/extensions/filters/http/ext_proc/client_impl.cc
#10 0x5aa510a01fb8 in Envoy::Grpc::AsyncStreamCallbacksenvoy::service::ext_proc::v3::ProcessingResponse::onReceiveMessageRaw(std::__msan::unique_ptr<Envoy::Buffer::Instance, std::__msan::default_deleteEnvoy::Buffer::Instance>&&) third_party/envoy/src/source/common/grpc/typed_async_client.h:98:5
#11 0x5aa51314e664 in Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc:153:21
#12 0x5aa51314e836 in non-virtual thunk to Envoy::Grpc::AsyncStreamImpl::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/grpc/async_client_impl.cc
#13 0x5aa51316009e in Envoy::Http::AsyncStreamImpl::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/async_client_impl.cc:142:21
#14 0x5aa51379db66 in Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc:1606:15
#15 0x5aa51379e016 in non-virtual thunk to Envoy::Router::Filter::onUpstreamData(Envoy::Buffer::Instance&, Envoy::Router::UpstreamRequest&, bool) third_party/envoy/src/source/common/router/router.cc
#16 0x5aa5137c7ad9 in Envoy::Router::UpstreamRequest::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.cc:311:11
#17 0x5aa5137dc48f in Envoy::Router::UpstreamRequestFilterManagerCallbacks::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_request.h:273:23
#18 0x5aa516af1223 in Envoy::Http::FilterManager::encodeData(Envoy::Http::ActiveStreamEncoderFilter*, Envoy::Buffer::Instance&, bool, Envoy::Http::FilterManager::FilterIterationStartState) third_party/envoy/src/source/common/http/filter_manager.cc:1356:29
#19 0x5aa516af19d9 in encodeData third_party/envoy/src/source/common/http/filter_manager.cc:463:11
#20 0x5aa516af19d9 in non-virtual thunk to Envoy::Http::ActiveStreamDecoderFilter::encodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/filter_manager.cc
#21 0x5aa5137ebda8 in Envoy::Router::UpstreamCodecFilter::CodecBridge::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/router/upstream_codec_filter.cc:162:23
#22 0x5aa5130bc9b4 in Envoy::Http::ResponseDecoderWrapper::decodeData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_wrappers.h:32:12
#23 0x5aa51701a99b in Envoy::Http::Http2::ConnectionImpl::StreamImpl::decodeData() third_party/envoy/src/source/common/http/http2/codec_impl.cc:519:17
#24 0x5aa51703799f in Envoy::Http::Http2::ConnectionImpl::onFrameReceived(nghttp2_frame const*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1175:13
#25 0x5aa517057040 in operator() third_party/envoy/src/source/common/http/http2/codec_impl.cc:1695:64
#26 0x5aa517057040 in Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_5::__invoke(nghttp2_session*, nghttp2_frame const*, void*) third_party/envoy/src/source/common/http/http2/codec_impl.cc:1694:19
#27 0x5aa517380334 in external_quiche_http2::adapter::CallbackVisitor::OnDataForStream(int, std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/callback_visitor.cc:259:14
#28 0x5aa51734e773 in external_quiche_http2::adapter::callbacks::OnDataChunk(nghttp2_session*, unsigned char, int, unsigned char const*, unsigned long, void*) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_callbacks.cc:277:32
#29 0x5aa5173b2b51 in nghttp2_session_mem_recv third_party/nghttp2/src/lib/nghttp2_session.c:7170:18
#30 0x5aa51734aefd in external_quiche_http2::adapter::NgHttp2Session::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_session.cc:35:10
#31 0x5aa517344b19 in external_quiche_http2::adapter::NgHttp2Adapter::ProcessBytes(std::__msan::basic_string_view<char, std::__msan::char_traits>) third_party/envoy/external_quiche/src/http2/adapter/nghttp2_adapter.cc:74:45
#32 0x5aa51703170f in Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc:956:20
#33 0x5aa5170342b0 in virtual thunk to Envoy::Http::Http2::ConnectionImpl::dispatch(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/http2/codec_impl.cc
#34 0x5aa513801292 in Envoy::Http::CodecClient::onData(Envoy::Buffer::Instance&) third_party/envoy/src/source/common/http/codec_client.cc:173:33
#35 0x5aa5138073e1 in Envoy::Http::CodecClient::CodecReadFilter::onData(Envoy::Buffer::Instance&, bool) third_party/envoy/src/source/common/http/codec_client.h:211:15
#36 0x5aa5171af927 in Envoy::Network::FilterManagerImpl::onContinueReading(Envoy::Network::FilterManagerImpl::ActiveReadFilter*, Envoy::Network::ReadBufferSource&) third_party/envoy/src/source/common/network/filter_manager_impl.cc:95:48
#37 0x5aa5171b0097 in Envoy::Network::FilterManagerImpl::onRead() third_party/envoy/src/source/common/network/filter_manager_impl.cc:105:3
#38 0x5aa5171887a0 in Envoy::Network::ConnectionImpl::onRead(unsigned long) third_party/envoy/src/source/common/network/connection_impl.cc:345:19
#39 0x5aa51719926f in Envoy::Network::ConnectionImpl::onReadReady() third_party/envoy/src/source/common/network/connection_impl.cc:651:5
#40 0x5aa5171929b8 in Envoy::Network::ConnectionImpl::onFileEvent(unsigned int) third_party/envoy/src/source/common/network/connection_impl.cc:602:5
#41 0x5aa5171a7f05 in operator() third_party/envoy/src/source/common/network/connection_impl.cc:94:54
#42 0x5aa5171a7f05 in __invoke<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#43 0x5aa5171a7f05 in __call<(lambda at third_party/envoy/src/source/common/network/connection_impl.cc:94:20) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#44 0x5aa5171a7f05 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#45 0x5aa5171a7f05 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::Dispatcher&, std::__msan::unique_ptr<Envoy::Network::ConnectionSocket, std::__msan::default_deleteEnvoy::Network::ConnectionSocket>&&, std::__msan::unique_ptr<Envoy::Network::TransportSocket, std::__msan::default_deleteEnvoy::Network::TransportSocket>&&, Envoy::StreamInfo::StreamInfo&, bool)::$_6, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#46 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#47 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#48 0x5aa517130576 in operator() third_party/envoy/src/source/common/event/dispatcher_impl.cc:184:9
#49 0x5aa517130576 in __invoke<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#50 0x5aa517130576 in __call<(lambda at third_party/envoy/src/source/common/event/dispatcher_impl.cc:182:7) &, unsigned int> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#51 0x5aa517130576 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#52 0x5aa517130576 in void std::__msan::__function::__policy_invoker<void (unsigned int)>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Event::DispatcherImpl::createFileEvent(int, std::__msan::function<void (unsigned int)>, Envoy::Event::FileTriggerType, unsigned int)::$_0, void (unsigned int)>>(std::__msan::__function::__policy_storage const*, unsigned int) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#53 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#54 0x5aa51713710d in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#55 0x5aa51713710d in Envoy::Event::FileEventImpl::mergeInjectedEventsAndRunCb(unsigned int) third_party/envoy/src/source/common/event/file_event_impl.cc:161:3
#56 0x5aa5171377fb in operator() third_party/envoy/src/source/common/event/file_event_impl.cc:82:16
#57 0x5aa5171377fb in Envoy::Event::FileEventImpl::assignEvents(unsigned int, event_base*)::$_0::__invoke(int, short, void*) third_party/envoy/src/source/common/event/file_event_impl.cc:66:7
#58 0x5aa5175af2c7 in event_persist_closure third_party/libevent/src/event.c:1639:9
#59 0x5aa5175af2c7 in event_process_active_single_queue third_party/libevent/src/event.c:1698:4
#60 0x5aa5175a2132 in event_process_active third_party/libevent/src/event.c:1799:9
#61 0x5aa5175a2132 in event_base_loop third_party/libevent/src/event.c:2041:12
#62 0x5aa517574e53 in Envoy::Event::LibeventScheduler::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/libevent_scheduler.cc:60:3
#63 0x5aa51712739a in Envoy::Event::DispatcherImpl::run(Envoy::Event::Dispatcher::RunType) third_party/envoy/src/source/common/event/dispatcher_impl.cc:299:19
#64 0x5aa511f0bbce in Envoy::Server::WorkerImpl::threadRoutine(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&) third_party/envoy/src/source/server/worker_impl.cc:148:16
#65 0x5aa511f0d318 in operator() third_party/envoy/src/source/server/worker_impl.cc:115:42
#66 0x5aa511f0d318 in __invoke<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:394:23
#67 0x5aa511f0d318 in __call<(lambda at third_party/envoy/src/source/server/worker_impl.cc:115:7) &> third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/invoke.h:487:9
#68 0x5aa511f0d318 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:239:12
#69 0x5aa511f0d318 in void std::__msan::__function::__policy_invoker<void ()>::__call_impl<std::__msan::__function::__default_alloc_func<Envoy::Server::WorkerImpl::start(Envoy::Server::GuardDog&, std::__msan::function<void ()> const&)::$_0, void ()>>(std::__msan::__function::__policy_storage const*) third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:723:16
#70 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:854:16
#71 0x5aa5178a74d2 in operator() third_party/crosstool/v18/stable/toolchain/bin/../include/c++/v1/__functional/function.h:1166:12
#72 0x5aa5178a74d2 in operator() third_party/envoy/src/source/common/common/posix/thread_impl.cc:49:11
#73 0x5aa5178a74d2 in Envoy::Thread::ThreadImplPosix::ThreadImplPosix(std::__msan::function<void ()>, std::__msan::optionalEnvoy::Thread::Options const&)::'lambda'(void*)::__invoke(void*) third_party/envoy/src/source/common/common/posix/thread_impl.cc:48:9
#74 0x79b7c0838d62 in start_thread (/usr/grte/v5/lib64/libpthread.so.0+0xbd62) (BuildId: 598ce51a74ecc39ad11a39e9c549e191)
MemorySanitizer can not provide additional info.
SUMMARY: MemorySanitizer: ABRT (/usr/grte/v5/lib64/libc.so.6+0x7e2fa) (BuildId: d17ba7518112927980b056b9456fb30d) in raise
==1458622==ABORTING

yanjunxiang-google · 2023-05-30T19:31:42Z

This is the fuzzer test case:

ext_proc_data: "scterpc_cre:csp_yilooo0\n\000*!pV1:ae!FoFFF,F\n"

wbpcode · 2023-05-31T04:58:53Z

clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path

Nope, the refresh in response phase still may changes the roue and effects the per filter config/metadata... etc that filters may need to use. Apparently it's wrong, so, we should never do this refresh in the response phase.

tyxia · 2023-05-31T12:09:30Z

clearRouteCache on response path to downstream client is just a no-op (since route selection is not needed on response path

Nope, the refresh in response phase still may changes the roue and effects the per filter config/metadata... etc that filters may need to use. Apparently it's wrong, so, we should never do this refresh in the response phase.

Yes, per filter config can be used in response path. What I meant about "no-op" is that before your PR #26045, such behavior will not trigger the Envoy_Bug. Thanks for sharing more context!

yanavlasov

/wait

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

repokitteh-read-only · 2023-05-31T19:28:51Z

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @markdroth
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #27657 was synchronize by yanjunxiang-google.

see: more, trace.

yanavlasov

/wait

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

tyxia

LGTM, modulo nit.

Thanks!

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

tyxia

Thanks!

yanjunxiang-google · 2023-06-05T13:44:21Z

@envoyproxy/api-shepherds PTAL

yanavlasov

/wait-any

… it completely anyway Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

yanjunxiang-google · 2023-06-09T14:34:45Z

Kind ping!

stevenzzzz · 2023-06-09T14:54:07Z

LGTM

yanjunxiang-google · 2023-06-12T20:29:09Z

Kind ping!

…roxy#27657) * ext_proc fuzzer test trigger ENVOY_BUG when clear route cache for upstream response Signed-off-by: Yanjun Xiang <yanjunxiang@google.com> Signed-off-by: asheryer <asheryer@amazon.com>

…roxy#27657) * ext_proc fuzzer test trigger ENVOY_BUG when clear route cache for upstream response Signed-off-by: Yanjun Xiang <yanjunxiang@google.com> Signed-off-by: Ryan Eskin <ryan.eskin89@protonmail.com>

ext_proc fuzzer test trigger ENVOY_BUG when clear route cache for ups…

d7da5f5

…tream response Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

yanjunxiang-google requested a review from snowp as a code owner May 26, 2023 19:11

repokitteh-read-only Bot assigned tyxia May 29, 2023

repokitteh-read-only Bot added the waiting label May 29, 2023

tyxia requested changes May 30, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/ext_proc.h Outdated

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

removing outbound stats

cc634a9

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

repokitteh-read-only Bot removed the waiting label May 30, 2023

yanjunxiang-google commented May 30, 2023

View reviewed changes

Comment thread test/extensions/filters/http/ext_proc/ext_proc_integration_test.cc

Merge branch 'main' of https://github.com/envoyproxy/envoy into ext_p…

f866784

…roc_clear_route_cache Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

repokitteh-read-only Bot assigned htuch, mpwarres, stevenzzzz and yanavlasov May 30, 2023

yanjunxiang-google commented May 30, 2023

View reviewed changes

Comment thread test/extensions/filters/http/ext_proc/ext_proc_integration_test.cc

yanavlasov requested changes May 31, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

repokitteh-read-only Bot added the waiting label May 31, 2023

tyxia reviewed May 31, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

addressing comments

488332c

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

repokitteh-read-only Bot added api and removed waiting labels May 31, 2023

repokitteh-read-only Bot assigned markdroth May 31, 2023

yanavlasov requested changes Jun 1, 2023

View reviewed changes

Comment thread api/envoy/service/ext_proc/v3/external_processor.proto Outdated

repokitteh-read-only Bot added the waiting label Jun 1, 2023

removing logs

8a2cf5b

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

repokitteh-read-only Bot removed the waiting label Jun 2, 2023

move clear route cache to decoder processing only

3e80bed

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

yanjunxiang-google mentioned this pull request Jun 2, 2023

Changing clear_route_cache from ext_proc server CommoneResponse into a more meaningful name #27772

Closed

tyxia reviewed Jun 2, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc Outdated

Comment thread source/extensions/filters/http/ext_proc/processor_state.h Outdated

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc

yanjunxiang-google added 2 commits June 2, 2023 19:40

addressing comments

2528ca0

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

adding back logs

b09a844

Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

tyxia previously approved these changes Jun 5, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc

yanavlasov requested changes Jun 5, 2023

View reviewed changes

Comment thread api/envoy/service/ext_proc/v3/external_processor.proto Outdated

repokitteh-read-only Bot added waiting:any and removed waiting:any labels Jun 5, 2023

backing out the change in external_processor.proto as we will change…

d8418bb

… it completely anyway Signed-off-by: Yanjun Xiang <yanjunxiang@google.com>

yanjunxiang-google dismissed tyxia’s stale review via d8418bb June 6, 2023 01:38

stevenzzzz reviewed Jun 6, 2023

View reviewed changes

Comment thread source/extensions/filters/http/ext_proc/processor_state.cc

stevenzzzz mentioned this pull request Jun 6, 2023

[Ext_proc]: only allow clear_route_cache on the INBOUND responses. #27828

Closed

yanjunxiang-google mentioned this pull request Jun 7, 2023

Refactoring ext_proc clear_route_cache API. #27852

Closed

tyxia approved these changes Jun 9, 2023

View reviewed changes

yanjunxiang-google requested a review from yanavlasov June 12, 2023 20:29

yanavlasov approved these changes Jun 12, 2023

View reviewed changes

yanavlasov merged commit ce456d5 into envoyproxy:main Jun 12, 2023

yanjunxiang-google deleted the ext_proc_clear_route_cache branch June 13, 2023 01:24

Conversation

yanjunxiang-google commented May 26, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KBaichoo commented May 29, 2023

Uh oh!

tyxia left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

yanjunxiang-google commented May 30, 2023

Uh oh!

yanjunxiang-google commented May 30, 2023

Uh oh!

yanjunxiang-google commented May 30, 2023

Uh oh!

Uh oh!

wbpcode commented May 31, 2023

Uh oh!

tyxia commented May 31, 2023

Uh oh!

yanavlasov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

repokitteh-read-only Bot commented May 31, 2023

Uh oh!

yanavlasov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

tyxia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tyxia left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

yanjunxiang-google commented Jun 5, 2023

Uh oh!

yanavlasov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

yanjunxiang-google commented Jun 9, 2023

Uh oh!

stevenzzzz commented Jun 9, 2023

Uh oh!

yanjunxiang-google commented Jun 12, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

yanjunxiang-google commented May 26, 2023 •

edited

Loading

tyxia left a comment •

edited

Loading