clh: Add support to unplug block devices by likebreath · Pull Request #2833 · kata-containers/runtime

likebreath · 2020-07-16T02:42:28Z

This patch enables kata+clh to unplug block devices, which is required
to pass cri-o integration tests.

Signed-off-by: Bo Chen chen.bo@intel.com

auto-comment · 2020-07-16T02:42:31Z

Thank you for raising your pull request. Please note that the main development of Kata Containers has moved to the 2.0-dev branch of https://github.com/kata-containers/kata-containers repository. The kata-containers/runtime repository is kept for 1.x release maintenance. Please check twice if your change should go to the 2.0-dev branch directly.

If it is strongly required for adding the change to Kata Containers 1.x releases, please ping @kata-containers/runtime to assign a dedicated developer to be responsible for porting the change to 2.0-dev branch. Thanks!

This patch extends the existing scripts to cover a new Jenkins job on testing k8s with crio for cloud-hypervisor. Fixes: kata-containers#2546 Depends-on: github.com/kata-containers/runtime#2833 Signed-off-by: Bo Chen <chen.bo@intel.com>

likebreath · 2020-07-16T02:59:32Z

/test-clh

codecov · 2020-07-16T03:56:12Z

Codecov Report

Merging #2833 into master will increase coverage by 0.00%.
The diff coverage is 51.85%.

@@           Coverage Diff           @@
##           master    #2833   +/-   ##
=======================================
  Coverage   51.40%   51.41%           
=======================================
  Files         118      118           
  Lines       17411    17434   +23     
=======================================
+ Hits         8950     8963   +13     
- Misses       7379     7388    +9     
- Partials     1082     1083    +1

likebreath · 2020-07-16T04:45:32Z

/test-ubuntu

likebreath · 2020-07-28T02:24:52Z

/test-clh

likebreath · 2020-07-28T02:25:00Z

/test-ubuntu

likebreath · 2020-07-28T02:28:05Z

/test-clh

likebreath · 2020-07-28T02:28:16Z

/test-ubuntu

This patch extends the existing scripts to cover a new Jenkins job on testing k8s with crio for cloud-hypervisor. Fixes: kata-containers#2546 Depends-on: github.com/kata-containers/runtime#2833 Signed-off-by: Bo Chen <chen.bo@intel.com>

likebreath · 2020-07-28T14:41:18Z

@jcvenegas @GabyCT The metrics CI is failing on Failed to load JSON error. Can you please share your insights about it?

19:34:43 Report Summary:
19:34:43 +-----+----------------------+---------------------+---------------+------+-----+-----+-----+-----+-----+-----+
19:34:43 | P/F |         NAME         |         FLR         |     MEAN      | CEIL | GAP | MIN | MAX | RNG | COV | ITS |
19:34:43 +-----+----------------------+---------------------+---------------+------+-----+-----+-----+-----+-----+-----+
19:34:43 | *F* | boot-times           | Failed to load JSON | exit status 2 |      |     |     |     |     |     |     |
19:34:43 | *F* | memory-footprint     | Failed to load JSON | exit status 2 |      |     |     |     |     |     |     |
19:34:43 | *F* | memory-footprint-ksm | Failed to load JSON | exit status 2 |      |     |     |     |     |     |     |
19:34:43 | *F* | blogbench            | Failed to load JSON | exit status 2 |      |     |     |     |     |     |     |
19:34:43 | *F* | blogbench            | Failed to load JSON | exit status 2 |      |     |     |     |     |     |     |
19:34:43 +-----+----------------------+---------------------+---------------+------+-----+-----+-----+-----+-----+-----+

GabyCT · 2020-07-28T15:10:24Z

@likebreath if you see the log, there are issues of unexpected processes

21:34:40 setting KSM to aggressive mode
21:34:40 
21:34:40 ===== starting test [memory footprint ksm] =====
21:34:40 command: docker: yes
21:34:40 27346
21:34:40 ERROR: Found unexpected /usr/bin/cloud-hypervisor present
21:34:40 disabling KSM
21:34:41 
21:34:41 ===== starting test [memory footprint] =====
21:34:41 command: docker: yes
21:34:41 27346
21:34:41 ERROR: Found unexpected /usr/bin/cloud-hypervisor present
21:34:41 
21:34:41 ===== starting test [memory footprint inside container] =====
21:34:41 command: docker: yes
21:34:41 27346
21:34:41 ERROR: Found unexpected /usr/bin/cloud-hypervisor present
21:34:41 Executing test: boot times image=ubuntu runtime=kata-runtime units=seconds
21:34:41 command: bc: yes
21:34:41 command: awk: yes
21:34:41 
21:34:41 ===== starting test [boot times] =====
21:34:41 command: docker: yes
21:34:42 27346
21:34:42 ERROR: Found unexpected /usr/bin/cloud-hypervisor present
21:34:42 
21:34:42 ===== starting test [blogbench] =====
21:34:42 command: docker: yes
21:34:42 27346
21:34:42 ERROR: Found unexpected /usr/bin/cloud-hypervisor present
21:34:43 27346
21:34:43 ERROR: Found unexpected /usr/bin/cloud-hypervisor present

GabyCT · 2020-07-28T15:11:04Z

that corrupted the tests and that is why the failure, that process is there before running the metrics

likebreath · 2020-07-28T15:23:56Z

21:34:41 ERROR: Found unexpected /usr/bin/cloud-hypervisor present

@GabyCT Thank you for the inputs. Any thoughts on why this error is happening and how it can be fixed?

GabyCT · 2020-07-28T15:31:40Z

@likebreath no, check the logs maybe you will get an info there of what is happening

likebreath · 2020-07-29T03:10:02Z

/test-clh-docker

jodh-intel

Thanks @likebreath.

lgtm

likebreath · 2020-07-29T15:26:49Z

/test-clh-metrics

likebreath · 2020-07-29T21:42:05Z

I re-ran the metrics CI again, and it is now not failing on the original setup stage in the CI. For now it is failing on the memory foot-print metrics, as listed below:

08:41:39 Report Summary:
08:41:39 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
08:41:39 | P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   |  RNG  | COV  | ITS |
08:41:39 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
08:41:39 | P   | boot-times           | 83.3% | 87.6%  | 116.7% | 33.3% | 79.2%  | 93.8%  | 18.5% | 4.7% |  20 |
08:41:39 | *F* | memory-footprint     | 95.0% | 95.0%  | 105.0% | 10.0% | 95.0%  | 95.0%  | 0.0%  | 0.0% |   1 |
08:41:39 | *F* | memory-footprint-ksm | 95.0% | 95.0%  | 105.0% | 10.0% | 95.0%  | 95.0%  | 0.0%  | 0.0% |   1 |
08:41:39 | P   | blogbench            | 80.0% | 117.2% | 120.0% | 40.0% | 117.2% | 117.2% | 0.0%  | 0.0% |   1 |
08:41:39 | P   | blogbench            | 80.0% | 103.6% | 120.0% | 40.0% | 103.6% | 103.6% | 0.0%  | 0.0% |   1 |
08:41:39 +-----+----------------------+-------+--------+--------+-------+--------+--------+-------+------+-----+
08:41:39 Fails: 2, Passes 3

With pointer from @jcvenegas, the root cause of this failure is: 08:41:39 time="2020-07-29T15:41:39Z" level=warning msg="Failed Minval ( 269727.515000(minimal mean expected) > 269672.400000(minimal mean on dataset)) for [memory-footprint]".

A similar metrics CI failure is also observed from PR #2863 (@amshinde). From the log: 17:52:12 time="2020-07-29T00:52:12Z" level=warning msg="Failed Minval ( 269727.515000(minimal mean expected) > 269595.700000(minimal mean on dataset)) for [memory-footprint]".

@GabyCT Can you please share any insights about the above obversations? Also, do you have pointers to explain the metrics CI in general? I am interested to know how those reference data are generated, and what kind of data are we tracking as the part of the metrics (e.g. why are we monitoring minimum memory-footprint, etc). Thank you.

GabyCT · 2020-07-29T21:57:48Z

@likebreath https://github.com/kata-containers/tests/tree/master/metrics, https://github.com/kata-containers/tests/tree/master/cmd/checkmetrics and https://github.com/kata-containers/tests/tree/master/cmd/checkmetrics/ci_slaves

To support unplug block device, we need to set the 'Id' explicitly while hotplugging devices with cloud-hypervisor HTTP API. Fixes: kata-containers#2832 Signed-off-by: Bo Chen <chen.bo@intel.com>

likebreath · 2020-08-13T23:45:24Z

/test-clh

likebreath · 2020-08-14T16:19:28Z

/test-clh

This patch enables kata+clh to unplug block devices, which is required to pass cri-o integration tests. Fixes: kata-containers#2832 Signed-off-by: Bo Chen <chen.bo@intel.com>

likebreath · 2020-08-14T16:40:01Z

/test-clh-metrics

likebreath · 2020-08-14T18:08:19Z

/test-ubuntu

likebreath · 2020-08-14T19:58:15Z

@amshinde @jcvenegas This PR is ready to be landed. Please take a look and ACK. Thanks.

likebreath force-pushed the fix_2832 branch from 3d859b6 to 57e68e4 Compare July 16, 2020 02:53

likebreath mentioned this pull request Jul 16, 2020

[DNM] clh: Add a ci job for crio tests kata-containers/tests#2621

Closed

sboeuf reviewed Jul 16, 2020

View reviewed changes

Comment thread virtcontainers/clh.go Outdated

sboeuf reviewed Jul 16, 2020

View reviewed changes

Comment thread virtcontainers/clh.go Outdated

amshinde reviewed Jul 17, 2020

View reviewed changes

Comment thread virtcontainers/clh.go Outdated

Comment thread virtcontainers/clh.go Outdated

likebreath force-pushed the fix_2832 branch from 57e68e4 to bf8a55b Compare July 28, 2020 02:23

likebreath force-pushed the fix_2832 branch from bf8a55b to c14cd16 Compare July 28, 2020 02:27

likebreath mentioned this pull request Jul 28, 2020

versions: Update qemu-virtiofs to 5.0 and cloud-hypervisor 0.9 #2840

Merged

This was referenced Jul 28, 2020

clh: Potential regression test failure VmInfoGet failed #2864

Closed

clh: Add some error handling for clh #2863

Merged

jodh-intel approved these changes Jul 29, 2020

View reviewed changes

likebreath mentioned this pull request Jul 30, 2020

[DNM] dummy: clh: test PR #1099 from kata-containers/packaging kata-containers/tests#2733

Closed

clh: Set 'Id' explicitly while hotplugging block device

03fb9c5

To support unplug block device, we need to set the 'Id' explicitly while hotplugging devices with cloud-hypervisor HTTP API. Fixes: kata-containers#2832 Signed-off-by: Bo Chen <chen.bo@intel.com>

likebreath force-pushed the fix_2832 branch from c14cd16 to a150e76 Compare August 13, 2020 23:45

likebreath force-pushed the fix_2832 branch from a150e76 to 44b58e4 Compare August 14, 2020 16:18

clh: Add support to unplug block devices

44b58e4

This patch enables kata+clh to unplug block devices, which is required to pass cri-o integration tests. Fixes: kata-containers#2832 Signed-off-by: Bo Chen <chen.bo@intel.com>

jcvenegas approved these changes Aug 14, 2020

View reviewed changes

jcvenegas merged commit 7827155 into kata-containers:master Aug 14, 2020

jodh-intel mentioned this pull request Aug 26, 2020

[forward port]: clh: Add support to unplug block devices kata-containers/kata-containers#604

Closed

Conversation

likebreath commented Jul 16, 2020

Uh oh!

auto-comment Bot commented Jul 16, 2020

Uh oh!

likebreath commented Jul 16, 2020

Uh oh!

codecov Bot commented Jul 16, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

likebreath commented Jul 16, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

GabyCT commented Jul 28, 2020

Uh oh!

GabyCT commented Jul 28, 2020

Uh oh!

likebreath commented Jul 28, 2020

Uh oh!

GabyCT commented Jul 28, 2020

Uh oh!

likebreath commented Jul 29, 2020

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

likebreath commented Jul 29, 2020

Uh oh!

likebreath commented Jul 29, 2020

Uh oh!

GabyCT commented Jul 29, 2020

Uh oh!

likebreath commented Aug 13, 2020

Uh oh!

likebreath commented Aug 14, 2020

Uh oh!

likebreath commented Aug 14, 2020

Uh oh!

likebreath commented Aug 14, 2020

Uh oh!

likebreath commented Aug 14, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Jul 16, 2020 •

edited

Loading