Rewrite virtio-net IO to be more efficient by mtjhrc · Pull Request #530 · containers/libkrun

mtjhrc · 2026-02-04T12:48:49Z

This PR

introduces new abstractions for dealing with virtio queues (RxQueueProducer, TxQueueConsumer) in an efficient manner
rewrite all 3 virtio-net device backends to use the abstractions (copy straight from guest memory into the backend, no intermediary buffer)
new integration tests for virtio-net over tap, passt, gvproxy

TODO:

more code cleanup
at least some basic benchmark data (e.g. iperf)
more manual testing besides the simple integration smoke tests

Adresses: #385, #405
supersedes: #493

Signed-off-by: Matej Hrica <mhrica@redhat.com>

Use correct platform-specific library directory (lib vs lib64) and library path env variable (DYLD_LIBRARY_PATH vs LD_LIBRARY_PATH). Signed-off-by: Matej Hrica <mhrica@redhat.com>

Signed-off-by: Matej Hrica <mhrica@redhat.com>

Make TcpTester::run_server also leak the file descriptor for the stream socket. Closing the fd caused the tests to fail on macOS (they randomly worked on Linux I suppose). Signed-off-by: Matej Hrica <mhrica@redhat.com>

Explicitly specify a /tmp directory, this fixes an issue on macOS where the default tmp path that gets used could be very long, causing unix domain socket tests to fail due to path length. Signed-off-by: Matej Hrica <mhrica@redhat.com>

This flag should be used to indicate to libkrun that downstream network backend wants to receive and transmit the virtio-net header along with Ethernet frames. Network backends using this flag can then forward unmodified headers to another VM or build a sensible virtio_net_hdr (e.g. with GSO fields correctly set) such that receiving VM handles GSO'd frames properly. Signed-off-by: Albin Kerouanton <albin.kerouanton@docker.com>

Signed-off-by: Matej Hrica <mhrica@redhat.com>

Introduce TxQueueConsumer and RxQueueProducer utilities, which allow consuming virtio queues as a bunch of iovec vectors. Notably these utilities are different than the preexisiting descriptor_utilis. The Reader and Writer in descriptor utilis operate on the order of single descriptor chains and don't allow the multiple descriptor chains to be processed at once due to borrowing issues, wheras these the TxQueueConsumer/RxQueueProducer operate on the order of all descriptors of all descriptor chains at once allowing for batch processing of the whole queue at once. Signed-off-by: Matej Hrica <mhrica@redhat.com>

Rewrite the all of the backend (unixstream, unixgram, tap) in terms of the new RxQueueProducer/TxQueueConsumer abstractions. Signed-off-by: Matej Hrica <mhrica@redhat.com>

Signed-off-by: Matej Hrica <mhrica@redhat.com>

Restore accidentally deleted guest-agent crate. Signed-off-by: Matej Hrica <mhrica@redhat.com>

Signed-off-by: Matej Hrica <mhrica@redhat.com>

nirs

Exciting change! I did not review everything but I added comments related to testing vmnet-helper.

nirs · 2026-02-19T22:08:21Z

tests/test_cases/src/test_net/vmnet_helper.rs

            config.mac.as_mut_ptr(),
-            COMPAT_NET_FEATURES,
-            0, // no VFKIT flag - vmnet-helper uses raw datagrams
+            0, // no offloading - vmnet-helper uses raw ethernet frames


you can use offloading if you add vmnet-helper options --enable-tso --enable-check-offload.
https://github.com/nirs/vmnet-helper/blob/455d17220e07e0301206050438e1f02620bf54aa/vmnet/helper.py#L158

nirs · 2026-02-19T22:09:44Z

tests/test_cases/src/test_net/vmnet_helper.rs


+    // Increase socket buffer sizes so libkrun's Unixgram backend (which uses
+    // the fd path and does NOT set these) can batch frames without drops.
+    let buf_size: libc::c_int = 7 * 1024 * 1024;


This is not needed, 4m is large enough for recv buffer.

nirs · 2026-02-19T22:12:27Z

tests/test_cases/src/test_net/vmnet_helper.rs

+        libc::setsockopt(
+            our_fd,
+            libc::SOL_SOCKET,
+            libc::SO_SNDBUF,


Send buffer is not used for datagram socket on macOS. The size should be larger than the maximum packet size. See https://github.com/nirs/vmnet-helper/blob/455d17220e07e0301206050438e1f02620bf54aa/config.h.in#L9

nirs · 2026-02-19T22:18:36Z

tests/test_cases/src/test_net/mod.rs

+            #[cfg(feature = "guest")]
+            gateway: Some([192, 168, 105, 1]),
+            // HACK: hardcoded host LAN IP for testing; guest needs a default
+            // route via the vmnet gateway (192.168.105.1) to reach it.


No need to hard code the IPs. If you test with a real vm you can use cloud-init to report the vm ip address in the serial log. Hacky works well for testing.

ip is written here:
https://github.com/nirs/vmnet-helper/blob/455d17220e07e0301206050438e1f02620bf54aa/vmnet/cidata.py#L72

and read from serial log here:
https://github.com/nirs/vmnet-helper/blob/455d17220e07e0301206050438e1f02620bf54aa/vmnet/vm.py#L114

mtjhrc force-pushed the vectored-io-net branch 2 times, most recently from 5220656 to db2af02 Compare February 6, 2026 13:43

slp mentioned this pull request Feb 9, 2026

vsock fails when guest sends large packets to host #535

Open

akerouanton mentioned this pull request Feb 11, 2026

net: add a new flag NET_FLAG_INCLUDE_VNET_HEADER #493

Open

mtjhrc force-pushed the vectored-io-net branch 4 times, most recently from bccf5ac to 3e444c0 Compare February 16, 2026 18:33

mtjhrc and others added 13 commits February 16, 2026 19:34

tests/run.sh: Add macOS cross-compilation support for guest-agent

ccb09b1

Signed-off-by: Matej Hrica <mhrica@redhat.com>

Makefile: add macOS support to test target

5bb56f5

Use correct platform-specific library directory (lib vs lib64) and library path env variable (DYLD_LIBRARY_PATH vs LD_LIBRARY_PATH). Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests/README.md: document running tests on macOS

d84eeaa

Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests: Add support for skipping tests

5e75488

Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests: Fix TcpTester::run_server closing TCP stream too early

8ae32b6

Make TcpTester::run_server also leak the file descriptor for the stream socket. Closing the fd caused the tests to fail on macOS (they randomly worked on Linux I suppose). Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests: Explicitly specify a /tmp directory

03d29c1

Explicitly specify a /tmp directory, this fixes an issue on macOS where the default tmp path that gets used could be very long, causing unix domain socket tests to fail due to path length. Signed-off-by: Matej Hrica <mhrica@redhat.com>

examples/chroot_vm: Add ability to attach a TAP for networking

3b69f88

Signed-off-by: Matej Hrica <mhrica@redhat.com>

utils: Introduce a set_non_blocking method for file descriptors

34a23cd

Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests: Introduce virtio-net tests (tap, passt, gvproxy)

ca8cc4f

Signed-off-by: Matej Hrica <mhrica@redhat.com>

devices/net: Rewrite using new RxQueueProducer/TxQueueConsumer

9834acf

Rewrite the all of the backend (unixstream, unixgram, tap) in terms of the new RxQueueProducer/TxQueueConsumer abstractions. Signed-off-by: Matej Hrica <mhrica@redhat.com>

fixup: (different branch) do not run unit tests on macos

a0d2bb3

Signed-off-by: Matej Hrica <mhrica@redhat.com>

mtjhrc force-pushed the vectored-io-net branch from 3e444c0 to 398016b Compare February 16, 2026 18:35

mtjhrc added 4 commits February 16, 2026 20:23

fixup! tests: Introduce virtio-net tests (tap, passt, gvproxy)

9c77d1b

Restore accidentally deleted guest-agent crate. Signed-off-by: Matej Hrica <mhrica@redhat.com>

tests: WIP introduce performance tests

3363b34

Signed-off-by: Matej Hrica <mhrica@redhat.com>

wip

678f7ab

Signed-off-by: Matej Hrica <mhrica@redhat.com>

wip

d09264b

Signed-off-by: Matej Hrica <mhrica@redhat.com>

mtjhrc force-pushed the vectored-io-net branch from 6e14f63 to d09264b Compare February 16, 2026 19:33

tests: Add vmnet-helper tests

567b33d

Signed-off-by: Matej Hrica <mhrica@redhat.com>

mtjhrc force-pushed the vectored-io-net branch from 1621b2c to 567b33d Compare February 17, 2026 11:13

fixup accidentally deleted something again

cc9f886

Signed-off-by: Matej Hrica <mhrica@redhat.com>

mtjhrc added 3 commits February 18, 2026 14:39

fixup: net: reeanble event_idx

62cd51e

Signed-off-by: Matej Hrica <mhrica@redhat.com>

fixup: queue consumer/producer handle event_idx

8cceb7f

Signed-off-by: Matej Hrica <mhrica@redhat.com>

wip human cleanup

89e3f09

Signed-off-by: Matej Hrica <mhrica@redhat.com>

nirs reviewed Feb 19, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comments

Rewrite virtio-net IO to be more efficient #530

Rewrite virtio-net IO to be more efficient #530
mtjhrc wants to merge 22 commits intocontainers:mainfrom
mtjhrc:vectored-io-net

mtjhrc commented Feb 4, 2026

Uh oh!

nirs left a comment

Uh oh!

nirs Feb 19, 2026

Uh oh!

nirs Feb 19, 2026

Uh oh!

nirs Feb 19, 2026

Uh oh!

nirs Feb 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

Conversation

mtjhrc commented Feb 4, 2026

Uh oh!

nirs left a comment

Choose a reason for hiding this comment

Uh oh!

nirs Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

nirs Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

nirs Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

nirs Feb 19, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants