feat: data plane & envoyproxy resilience test suite#4862
feat: data plane & envoyproxy resilience test suite#4862guydc merged 54 commits intoenvoyproxy:mainfrom
Conversation
Signed-off-by: Alexander Volchok <alex.volchok@sap.com>
alexwo
changed the title
alexwo
changed the title
alexwo
changed the title
Signed-off-by: Alexander Volchok <alex.volchok@sap.com>
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #4862 +/- ##
=======================================
Coverage 66.28% 66.28%
=======================================
Files 209 209
Lines 31942 31942
=======================================
+ Hits 21172 21174 +2
+ Misses 9517 9516 -1
+ Partials 1253 1252 -1 ☔ View full report in Codecov by Sentry. |
Signed-off-by: Alexander Volchok <alex.volchok@sap.com>
alexwo
changed the title
alexwo
changed the title
alexwo
changed the title
|
/retest |
1 similar comment
|
/retest |
| TIMEOUT=120 # seconds | ||
| ELAPSED_TIME=0 | ||
|
|
||
| if [ "$CUSTOM_CNI" = "true" ]; then |
There was a problem hiding this comment.
can you share more details about why we need cilium?
There was a problem hiding this comment.
The default network plugin in Kubernetes-in-Docker (Kind) does not support enforcing network policies that restrict connectivity to local services, such as the Kubernetes API server. However, by disabling the default CNI and deploying Cilium as the CNI plugin, we can implement network policies that effectively control access to the API server. This approach allows for precise management of network traffic within the cluster, including the ability to block connectivity to the Kubernetes API server as needed.
There was a problem hiding this comment.
I recall kind support network policy as kubernetes-sigs/kind#842 closed.
There was a problem hiding this comment.
@zirain That’s correct, network policies are supported now. However, even the new local kind network adapter is currently not enforcing network policies that restrict connectivity to local services like the API server. As a result, the default kind adapter isn’t suitable for this specific use case.
There was a problem hiding this comment.
There was a problem hiding this comment.
I will give it a try, maybe it can do the work 👍
There was a problem hiding this comment.
@zirain I tested with kube-network-policies, but it was unable to block connectivity to the API server. In fact, I tried all the other available options as well. Currently, it seems this functionality is only achievable with the Cilium CNI.
|
why we need a new test suite instead of reuse existing one? |
Having a dedicated resilience test suite can be helpful due to the unique network setup and tests involved in resilience testing, which differ significantly from those of existing test suites. For instance, we need a custom CNI only for this test case, other tests can still use the default. Fault Injection: Resilience testing involves deliberately introducing failures to assess the system’s response and recovery capabilities. This approach is not typically encompassed by standard functional or performance test suites. By maintaining a separate resilience test suite, we can effectively simulate and manage system stressors, and separate this concern from the other test cases we have. Running this suite in parallel will keep us at good execution time. |
|
/retest |
|
/retest |
| t.Log("Marking the identified pod as leader") | ||
| suite.Kube().MarkAsLeader(namespace, name) | ||
|
|
||
| t.Log("Simulating API server connection failure for all pods") |
There was a problem hiding this comment.
If all pods loose connectivity, don't we expect that they would all fail eventually once becoming leaders?
There was a problem hiding this comment.
Pods rely on connectivity to the API server to participate in leader election processes. Without this connectivity, they cannot become leaders, in this test we won't bring it back. The secondary pods will continue to operate using their last known good configuration (xDS) and share it with envoy proxies accordingly.
|
@alexwo Could you please add a few lines of the comments for each test? (Like you explained in the comments of this PR) This will help other contributors understand why we need the test and how the test works. Thanks! |
Co-authored-by: Huabing (Robin) Zhao <zhaohuabing@gmail.com> Signed-off-by: Alex Volchok <alex.volchok@sap.com>
|
#4767 has been merged, that should unblock this PR now ? |
|
/retest |
Signed-off-by: Alex Volchok <alex.volchok@sap.com>
5 participants
What type of PR is this?
This pr adds a resilience test suite to cover the expected behavior of the data plane for both existing and new proxy instances during specific edge conditions.
Why do we use Cilium?
The default network plugin in Kubernetes-in-Docker (Kind) does not support enforcing network policies that restrict connectivity to local services, such as the Kubernetes API server. However, by disabling the default CNI and deploying Cilium as the CNI plugin, we can implement network policies that effectively control access to the API server. This approach allows for precise management of network traffic within the cluster, including the ability to block connectivity to the Kubernetes API server as needed.
New test suite
Having a new resilience test suite can be helpful due to the unique network setup and tests involved in resilience testing, which differ significantly from those of existing test suites.
For instance, we need a custom CNI only for this test case, other tests can still use the default.
Fault Injection: Resilience testing involves deliberately introducing failures to assess the system’s response and recovery capabilities. This approach is not typically encompassed by standard functional or performance test suites.
By maintaining a separate resilience test suite, we can effectively simulate and manage system stressors, and separate this concern from the other test cases we have.
Test suite is expected to pass once #4767 is merged.
Issue:
#4861