Hi.
the conditions below using httpCall API, crach occurs.
- When service exists but Pod does not exist
$ kubectl get service -o yaml test-service
apiVersion: v1
kind: Service
metadata:
name: test-service
labels:
app: test-server
spec:
selector:
app: test-server
type: NodePort
ports:
- name: http2
port: 8080
nodePort: 32530
targetPort: 9090
$ kubectl get pod test-service
Error from server (NotFound): pods "test-service" not found
The backtrace is as follows
2021-01-07T04:20:30.141218Z warning envoy wasm [external/envoy/source/extensions/common/wasm/context.cc:1110] wasm log outbound|8080||test-service.default.svc.cluster.local
2021-01-07T04:20:30.141362Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x148
2021-01-07T04:20:30.141385Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
2021-01-07T04:20:30.141392Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: a20ed1b7b98d8a7a8480c5100c0752578e31d659/1.14.4/Clean/RELEASE/BoringSSL
2021-01-07T04:20:30.141897Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7fad273038a0]
2021-01-07T04:20:30.154680Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #1: Envoy::Http::AsyncRequestImpl::onComplete() [0x55b896955ccf]
2021-01-07T04:20:30.165000Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #2: Envoy::Http::AsyncStreamImpl::encodeData() [0x55b8969556c2]
2021-01-07T04:20:30.174887Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #3: Envoy::Http::Utility::sendLocalReply() [0x55b896a744ee]
2021-01-07T04:20:30.184720Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #4: Envoy::Http::AsyncStreamImpl::sendLocalReply() [0x55b896956975]
2021-01-07T04:20:30.200819Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #5: Envoy::Router::Filter::sendNoHealthyUpstreamResponse() [0x55b89695e98a]
2021-01-07T04:20:30.211038Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #6: Envoy::Router::Filter::decodeHeaders() [0x55b89695ce9c]
2021-01-07T04:20:30.221311Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #7: Envoy::Http::AsyncStreamImpl::sendHeaders() [0x55b896955a47]
2021-01-07T04:20:30.231357Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #8: Envoy::Http::AsyncRequestImpl::initialize() [0x55b8969549e7]
2021-01-07T04:20:30.241355Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #9: Envoy::Http::AsyncClientImpl::send() [0x55b89695444b]
2021-01-07T04:20:30.253443Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #10: Envoy::Extensions::Common::Wasm::Context::httpCall() [0x55b8959aacd2]
2021-01-07T04:20:30.263850Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #11: Envoy::Extensions::Common::Wasm::Exports::http_call() [0x55b8959c2b45]
2021-01-07T04:20:30.284883Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #12: Envoy::Extensions::Common::Wasm::ConvertFunctionWordToUint32<>::convertFunctionWordToUint32() [0x55b8959cea06]
2021-01-07T04:20:30.300589Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #13: Envoy::Extensions::Common::Wasm::V8::V8::registerHostFunctionImpl<>()::{lambda()#1}::operator()() [0x55b8959f96f2]
2021-01-07T04:20:30.310606Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #14: Envoy::Extensions::Common::Wasm::V8::V8::registerHostFunctionImpl<>()::{lambda()#1}::__invoke() [0x55b8959f9597]
2021-01-07T04:20:30.322101Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #15: wasm::FuncData::v8_callback() [0x55b895a04e8f]
2021-01-07T04:20:30.322529Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #16: [0x39fdc0a41647]
2021-01-07T04:20:30.323159Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #17: [0x39fdc0712f74]
2021-01-07T04:20:30.323801Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #18: [0x39fdc07b275f]
2021-01-07T04:20:30.324442Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #19: [0x39fdc075c935]
2021-01-07T04:20:30.325075Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #20: [0x39fdc0895348]
2021-01-07T04:20:30.325714Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #21: [0x16f6000835df]
2021-01-07T04:20:30.335913Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #22: v8::internal::Execution::CallWasm() [0x55b895add963]
2021-01-07T04:20:30.351839Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #23: wasm::Func::call() [0x55b895a041f3]
2021-01-07T04:20:30.362223Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #24: Envoy::Extensions::Common::Wasm::V8::V8::getModuleFunctionImpl<>()::{lambda()#1}::operator()() [0x55b8959ecc40]
2021-01-07T04:20:30.373415Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #25: Envoy::Extensions::Common::Wasm::Context::onRequestBody() [0x55b8959addb9]
2021-01-07T04:20:30.383636Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: Envoy::Extensions::Common::Wasm::Context::decodeData() [0x55b8959b03dc]
2021-01-07T04:20:30.394001Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #27: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeData() [0x55b8969ca131]
2021-01-07T04:20:30.403919Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #28: Envoy::Http::ConnectionManagerImpl::ActiveStream::decodeData() [0x55b8969c9f08]
2021-01-07T04:20:30.415622Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #29: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x55b8969e4589]
2021-01-07T04:20:30.428600Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #30: nghttp2_session_on_data_received [0x55b896bb109c]
2021-01-07T04:20:30.444517Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #31: nghttp2_session_mem_recv [0x55b896bb309d]
2021-01-07T04:20:30.455400Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #32: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55b8969e3c1a]
2021-01-07T04:20:30.465503Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #33: Envoy::Http::Http2::ServerConnectionImpl::dispatch() [0x55b8969e7660]
2021-01-07T04:20:30.475754Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #34: Envoy::Http::ConnectionManagerImpl::onData() [0x55b8969c4a52]
2021-01-07T04:20:30.490730Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #35: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55b896763883]
2021-01-07T04:20:30.502170Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #36: Envoy::Network::ConnectionImpl::onReadReady() [0x55b89675f9c5]
2021-01-07T04:20:30.512263Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #37: Envoy::Network::ConnectionImpl::onFileEvent() [0x55b89675ed1d]
2021-01-07T04:20:30.522525Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #38: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x55b896759d46]
2021-01-07T04:20:30.532490Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #39: event_process_active_single_queue [0x55b896ba2f4b]
2021-01-07T04:20:30.542948Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #40: event_base_loop [0x55b896ba17de]
2021-01-07T04:20:30.559206Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #41: Envoy::Server::WorkerImpl::threadRoutine() [0x55b89674db84]
2021-01-07T04:20:30.569170Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #42: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x55b896c579c3]
2021-01-07T04:20:30.569285Z critical envoy backtrace [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #43: start_thread [0x7fad272f86db]
ActiveStream 0x55b89951db00, stream_id_: 17940336530524321975, state_.has_continue_headers_: 0, state_.is_head_request_: 0, state_.decoding_headers_only_: 0, state_.encoding_headers_only_: 0
request_headers_:
':authority', 'test.domain.com:32530'
':method', 'PUT'
':path', '/namf-comm/v1/ue-contexts/nfsimulator'
':scheme', 'http'
'content-type', 'application/json'
'content-length', '5624'
'accept-encoding', 'gzip'
'user-agent', 'Go-http-client/2.0'
'x-forwarded-proto', 'http'
'x-request-id', '5173ce30-091d-4402-9ffb-533748f70aaf'
'x-envoy-decorator-operation', 'test.domain.com:32530/*'
'x-envoy-peer-metadata-id', 'sidecar~10.244.1.38~nfsim-server-8bcf66cf4-hqhbg.default~default.svc.cluster.local'
request_trailers_: null
response_headers_: null
response_trailers_: null
&stream_info_:
StreamInfoImpl 0x55b89951dc48, protocol_: 2, response_code_: null, response_code_details_: null, health_check_request_: 0, route_name_:
2021-01-07T04:20:30.585441Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2021-01-07T04:20:30.585478Z info sds resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2021-01-07T04:20:30.585503Z info sds resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2021-01-07T04:20:30.585543Z error sds Remote side closed connection
2021-01-07T04:20:30.585547Z error sds Remote side closed connection
2021-01-07T04:20:30.586437Z error Epoch 0 exited with error: signal: segmentation fault
2021-01-07T04:20:30.586458Z info No more active epochs, terminating
It can be solved by modifying it like the code below. Please review.
envoy/source/extensions/common/wasm.context.cc
WasmResult Context::httpCall(absl::string_view cluster, const Pairs& request_headers,
absl::string_view request_body, const Pairs& request_trailers,
int timeout_milliseconds, uint32_t* token_ptr) {
--- skip ----
#ifdef __FIX_ENVOY_CORE__ /* by eunbin */
handler.context_ = this;
handler.token_ = token;
*token_ptr = token;
#endif
// set default hash policy to be based on :authority to enable consistent hash
Http::AsyncClient::RequestOptions options;
options.setTimeout(timeout);
Protobuf::RepeatedPtrField<HashPolicy> hash_policy;
hash_policy.Add()->mutable_header()->set_header_name(Http::Headers::get().Host.get());
options.setHashPolicy(hash_policy);
auto http_request = clusterManager()
.httpAsyncClientForCluster(cluster_string)
.send(std::move(message), handler, options);
if (!http_request) {
http_request_.erase(token);
return WasmResult::InternalFailure;
}
#ifdef __FIX_ENVOY_CORE__ /* by eunbin */
handler.request_ = http_request;
#else
handler.context_ = this;
handler.token_ = token;
handler.request_ = http_request;
*token_ptr = token;
return WasmResult::Ok;
#endif
}
Hi.
the conditions below using httpCall API, crach occurs.
The backtrace is as follows
It can be solved by modifying it like the code below. Please review.
envoy/source/extensions/common/wasm.context.cc