access logging: gRPC logger retry to establish underlying stream connection

[Shikugawa]

Signed-off-by: Shikugawa rei@tetrate.io

Commit Message: Add a retry mechanism to the grpc access logger. This retry mechanism currently only supports a simple retry count. Also, at the moment, retries are only fired when the gRPC stream fails to be established, and nothing happens if the stream is successfully established once and a reset is issued.
Additional Description:
Risk Level: Low
Testing: Unit
Docs Changes: Required
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Deprecated:]
[Optional API Considerations:]

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to api/envoy/.
envoyproxy/api-shepherds assignee is @lizan
CC @envoyproxy/api-watchers: FYI only for changes made to api/envoy/.

🐱

Caused by: #17469 was opened by Shikugawa.

see: more, trace.

[dmitri-d]

lgtm!

[lizan]

/wait

[zuercher]

/wait

[htuch]

@lizan friendly ping.

[Shikugawa]

/retest

[repokitteh-read-only]

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #17469 (comment) was created by @Shikugawa.

see: more, trace.

[lizan]

@htuch can you do a non-tetrands review?

[htuch]

Looks good, just one Q: how does this behave when we are using the Google gRPC async client vs. the in-built Envoy async gRPC client?

[Shikugawa]

Looks good, just one Q: how does this behave when we are using the Google gRPC async client vs. the in-built Envoy async gRPC client?

It seems opaque behavior on Google gRPC with this API. Maybe we should configure channel args for retry and buffer on google_grpc configuration.

envoy/source/common/grpc/google_async_client_impl.cc

Lines 172 to 202 in 3b0a4a0

	// TODO(htuch): figure out how to propagate "this request should be buffered for
	// retry" bit to Google gRPC library.
	void GoogleAsyncStreamImpl::initialize(bool /*buffer_body_for_retry*/) {
	parent_.stats_.streams_total_->inc();
	gpr_timespec abs_deadline =
	options_.timeout
	? gpr_time_add(gpr_now(GPR_CLOCK_REALTIME),
	gpr_time_from_millis(options_.timeout.value().count(), GPR_TIMESPAN))
	: gpr_inf_future(GPR_CLOCK_REALTIME);
	ctxt_.set_deadline(abs_deadline);
	// Fill service-wide initial metadata.
	auto initial_metadata = Http::RequestHeaderMapImpl::create();
	parent_.metadata_parser_->evaluateHeaders(*initial_metadata, options_.parent_context.stream_info);
	callbacks_.onCreateInitialMetadata(*initial_metadata);
	initial_metadata->iterate([this](const Http::HeaderEntry& header) {
	ctxt_.AddMetadata(std::string(header.key().getStringView()),
	std::string(header.value().getStringView()));
	return Http::HeaderMap::Iterate::Continue;
	});
	// Invoke stub call.
	rw_ = parent_.stub_->PrepareCall(&ctxt_, "/" + service_full_name_ + "/" + method_name_,
	&parent_.tls_.completionQueue());
	if (rw_ == nullptr) {
	notifyRemoteClose(Status::WellKnownGrpcStatus::Unavailable, nullptr, EMPTY_STRING);
	call_failed_ = true;
	return;
	}
	parent_.tls_.registerStream(this);
	rw_->StartCall(&init_tag_);
	++inflight_tags_;
	}

If we'd like to integrate between Envoy gRPC and Google gRPC for retry and buffer, we should have API migration for envoy's retry policy and Google gRPC's channel args. Is there any plan to work on this?

[htuch]

I don't think there is a solid story for how to get the retry/buffering semantics aligned across the gRPC implementations. This really touches on the fact that we have not made any progress on unifying the gRPC implementations more broadly.

I think for this PR, maybe just clarify in the release notes and comments that this only works with the Envoy gRPC client and we can merge.

[repokitteh-read-only]

CC @envoyproxy/api-shepherds: Your approval is needed for changes made to (api/envoy/|docs/root/api-docs/).
envoyproxy/api-shepherds assignee is @lizan
CC @envoyproxy/api-watchers: FYI only for changes made to (api/envoy/|docs/root/api-docs/).

🐱

Caused by: #17469 was synchronize by Shikugawa.

see: more, trace.

[Shikugawa]

/retest

[repokitteh-read-only]

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #17469 (comment) was created by @Shikugawa.

see: more, trace.

[Shikugawa]

/retest

[repokitteh-read-only]

Retrying Azure Pipelines:
Retried failed jobs in: envoy-presubmit

🐱

Caused by: a #17469 (comment) was created by @Shikugawa.

see: more, trace.

[lizan]

@htuch ping?

[htuch]

LGTM, thanks!

[durd07]

@htuch @Shikugawa the merge bring back the generated_api_shadow into main repo.
can we add a hook to check whether generated_api_shadow added by mistake?
I have seen serval times generated_api_shadow back.

[Shikugawa]

@htuch @Shikugawa the merge bring back the generated_api_shadow into main repo. can we add a hook to check whether generated_api_shadow added by mistake? I have seen serval times generated_api_shadow back.

@durd07 Thank you for the notification. This is no longer needed and just my mistake. #18911

[htuch]

@durd07 what other examples are there? If this is common I can add a hook, otherwise I think we will soon forget about generated_api_shadow/ and this won't sneak back in.

[durd07]

@htuch , yes, you're right. maybe we will forget about it soon. I met this situation at least twice. but I think it is not necessary to emphasis the alreay not exist codes. please forget it.