From 4415dfed91c3bb5d4326fe78f643f929dd997cce Mon Sep 17 00:00:00 2001 From: Ashraf Fouda Date: Thu, 12 Mar 2026 17:06:04 +0200 Subject: [PATCH] updates network docs Signed-off-by: Ashraf Fouda --- docs/internals/network-light/readme.md | 334 +++++++++++++++++++------ docs/internals/network/readme.md | 272 +++++++++++++++++++- 2 files changed, 522 insertions(+), 84 deletions(-) diff --git a/docs/internals/network-light/readme.md b/docs/internals/network-light/readme.md index b8179709..1c9d7484 100644 --- a/docs/internals/network-light/readme.md +++ b/docs/internals/network-light/readme.md @@ -1,109 +1,194 @@ +# Network Light Module -# Network Light +## ZBus -> WIP: This is still a draft proposal +Network light module is available on zbus over the following channel -Network light is a new version of network daemon that simplifies zos setup on most of cloud providers so ZOS can be deployed and hosted on the cloud. This version of networking has a small set of requirements and features as follows: +| module | object | version | +|--------|--------|---------| +| network | [network](#interface) | 0.0.1 | -- VMs have (Always) two IPs. - - A local private IP. This only allows the VM to have access to the public internet via NATting. So far it's not relevant if that IP subnet and IP is assigned by the user or ZOS itself - - A mycelium IP. The mycelium IP subnet is chosen by the user during deployment for his entire *network space* on that node (by providing the seed) and then he can chose individual IPs per VM that is deployed in that space. -- Different *network space* has isolated networks. VMs from different spaces can't communicate over their private Ips, but only over their mycelium Ips. -- If the user wants his VMs across multiple node to communicate it's up to him to setup mycelium ips white lists or other means of authentication. -- There is NO public IP support. -- ZDBs only have mycelium IPs. -- No yggdrasil -- No public config +## Home Directory -## Network layout +| directory | path | +|-----------|------| +| root | `/var/cache/modules/networkd` (volatile) | +| networks | `{root}/networks/` — persisted NR configs | +| link | `{root}/networks/link/` — workload-to-network symlinks | +| ndmz leases | `{root}/ndmz-lease/` — IPv4 IPAM allocations | +| mycelium seeds | `/tmp/network/mycelium/` — per-NR mycelium seed files | -This is to layout the simplified version of networkd. This version makes sure that ZOS gets only a single IP (via the cloud provider) and then all traffic from that node is basically NATted over this IP. +## Introduction -### Host namespace +Network light is a simplified version of the [network module](../network/readme.md) designed for cloud-hosted ZOS nodes. It uses Mycelium as the primary overlay network and drops features that require complex infrastructure (Yggdrasil, public IPs, dual-NIC detection with IPv6 SLAAC). -![host](png/host.png) +### Key differences from the full network module -- The physical (only) nic is directly attached to the ZOS bridge. -- There is a dhcp client instance running on zos bridge that gets an IP from the physica network. -- There is created an **ndmz** bridge that is set to UP and statically assigned an IP address `100.127.0.1` - - This bridge will be wired later to other namespaces -- The `br-my` bridge is created. - - This bridge will be wired later to `zdb` namespaces so they can get mycelium IPs. -- nft rule to drop connection to `100.172.0.1` +| Feature | Network Light | Full Network | +|---------|--------------|--------------| +| Primary overlay | Mycelium (`400::/7`) | Yggdrasil (`200::/7`) + Mycelium | +| Yggdrasil | Not available | Full support with `br-ygg` bridge | +| Public IPs for VMs | Not supported | Supported via `br-pub` + nftables | +| NDMZ | Simple bridge at `100.127.0.1/16` | Full namespace with dual-stack (IPv4 DHCP + IPv6 SLAAC) | +| ZDB connectivity | Mycelium only (via `br-hmy`) | Public IPv6 + Yggdrasil + Mycelium | +| WireGuard | Optional per-NR | Always present per-NR | +| NIC setup | Single NIC assumed | Auto-detects single/dual NIC | -The usage of `br-ndmz` and `br-my` bridges will be clear later +### What VMs get -> **SAFETY:** All incoming connections to ip `100.127.0.1` is dropped via nft rule. This will prevent VMs from being able to connect to the node directly. +- A **local private IP** (e.g., `10.x.x.x`) for internet via NAT through the node's IP +- A **mycelium IP** (`400::/7`) for end-to-end encrypted communication. The user controls the subnet via their seed +- Optional **WireGuard** access for direct private network connectivity -## Ndmz namespace +## Architecture -![ndmz](png/ndmz.png) -The ndmz namespace is a much simplified version of the previous layout. -The key difference here is that ndmz does **NOT** have an IP from the physical LAN instead it is also natted over the same node IP. +### Host Namespace -The reason we need it is we can safely run services that need to be accessed remotely over the node mycelium IP. Like the NODE openRPC api. But also prevent attackers from reaching to other services directly running inside the host +``` + ┌──────────┐ + │ NIC │ + └────┬─────┘ + │ attached + ▼ + ┌──────────────────────────────────────────────────────────────────┐ + │ Host │ + │ │ + │ ┌──────────┐ ┌─────┐ ┌──────────┐ ┌──────────┐ │ + │ │ br-hmy │ │ my0 │ │ zos │- - - - - │ br-ndmz │ │ + │ │ (ZDB) │ │(tun)│ │ (DHCP) │ NAT'd │100.127.0.1 │ + │ └──────────┘ └─────┘ └──────────┘ └──────────┘ │ + │ │ │ │ + │ │ mycelium-host │ + │ │ zinit service │ + └──────────────────────────────────────────────────────────────────┘ +``` + +- The physical NIC is attached to the `zos` bridge +- DHCP client runs on `zos` to get the node's IP +- `br-ndmz` bridge is created with static IP `100.127.0.1/16` +- `my0` tunnel and `br-hmy` bridge run directly on the host (moved from ndmz namespace in the original design) +- `mycelium-host` zinit service creates the `my0` tunnel for host-level mycelium +- `br-hmy` connects ZDB namespaces to the host mycelium +- nftables rule drops inbound connections to `100.127.0.1` -The node mycelium instance will also be running inside `ndmz` this process will then create the `my0` interface and give it the seed associated ip address. For example let's assume `my0` gets ip `58a:1059:b9a9:bc3:a1e8:5909:8957:d956/7` +### Bridges -According to mycelium docs the entire `/64` prefix is assigned also to that node. We then can do the following: +| Bridge | Purpose | +|--------|---------| +| `zos` | Management bridge; DHCP from physical network | +| `br-pub` | Public bridge; exit NIC or veth to zos | +| `br-ndmz` | NDMZ routing bridge at `100.127.0.1/16`; all NR namespaces connect here | +| `br-hmy` | Host mycelium bridge; ZDB namespaces connect here for mycelium IPs | +| `r` | Per-NR private bridge; VMs attach TAPs here | +| `m` | Per-NR mycelium bridge; VMs attach mycelium TAPs here | -- Assign the IP `58a:1059:b9a9:bc3::1/64` to nmy -- make sure forwarding is enabled for ipv6 -- set routing of `58a:1059:b9a9:bc3::/64` over dev `nmy` -- make sure that `200::/7` is routed over `my0` -- make sure default gw is set to `100.127.0.1` (the br-ndmz on the host) -- enable ipv4 forwarding, and set any needed masquerading rules as needed. `oifname "public" masquerade fully-random` +### Namespaces -Later, containers wired to `br-my` can then get IPs inside the range `58a:1059:b9a9:bc3::/64` and they then can send traffic (and receive traffic) over mycelium network +| Namespace | Purpose | +|-----------|---------| +| `public` | Optional — only when farmer sets a public config | +| `n` | Per-user network resource with private bridge, mycelium, optional WireGuard | +| `n` | Per-ZDB container; wired to `br-hmy` for mycelium access | -> Note: Again the reason we still have ndmz is to run mycelium isolated from the host, and run API services isolated from rest of the system. It's still possible to simply create the same setup on the host name space directly. Jan? +Unlike the full network module, network light has **no ndmz namespace**. The `br-ndmz` bridge lives directly on the host at `100.127.0.1/16` and provides NAT'd internet to all NR namespaces through the node's IP. Mycelium runs inside each NR namespace individually (not in a shared daemon). -## NR (Network Resource) +## Network Resources (NR) -![nr](png/nr.png) +``` + ┌──────┐ + │ VM │ + └──┬┬──┘ + ││ + ┌─────────────┘└──────────┐ + │ tap (b-) tap │ (m-) + ▼ ▼ + r- m- br-ndmz + ○ ○ ○ + │ veth │ veth │ veth + │ │ │ + ┌─┼──────────────────────────────────┼─────────────┼──┐ + │ │ n- │ │ │ + │ ▼ ▼ ▼ │ + │ ┌─────────┐ ┌──────────┐ ┌─────────┐ │ + │ │ private │ │ mycelium │ │ public │ │ + │ │.1 │ │ (myc daemon │100.127. │ │ + │ │ gw │ │ runs here) │ x.x/16 │ │ + │ └─────────┘ └──────────┘ └─────────┘ │ + │ ▲ │ │ + │ VM private default route │ + │ traffic via 100.127.0.1 │ + └─────────────────────────────────────────────────────┘ +``` -The network resource is yet a simplified version of NR namesapce on original setup but here it looks and behaves more like `ndmz` in the sense that it only to isolate VMs from rest of the system. All public traffic is natter over `public` interface, which is wired to br-ndmz +Each user network gets an isolated namespace `n` with: -The `my0`, `nmy`, and `m-` work in exactly similar way as the ones in `ndmz`. The only difference is that those are user network specific. The user provides the seed and hence control what IP range, and Ips assigned to the VMs. All routing rules has to be done here. +**Interfaces:** +- `private` veth → `r` bridge (private VM traffic). Gateway at `.1` +- `public` veth → `br-ndmz` (internet via NAT). IP from `100.127.0.2–100.127.255.254` +- `mycelium` veth → `m` bridge (per-NR mycelium) +- `w-` WireGuard interface (optional) -The `b-` is where all VMs will be connected. Each VM must get an IP range from a private range as follows: +**Mycelium:** +Each NR runs its own mycelium daemon inside its namespace with the user's seed. This gives the user control over the IP range. VMs get deterministic IPs from the NR's `/64` subnet: `::ff0f:/64`. -- the private range doesn't really matter, it can be fixed by ZOS forever (say range `10.0.0.0/8`) or allowed to be chosen by the user. - - I personally prefer to make it fixed to `10.0.0.0/8` range and allow the user to choose individual VMs ips in the private range. -- `private` interface becomes the `gw` hence it can get the first ip in the range as per convention. So `10.0.0.1` -- make sure routing is set as follows: - - default gw is `100.127.0.1` (the br-ndmz on the host) -- enable ipv4 forwarding, and set any needed masquerading rules as needed. `oifname "public" masquerade fully-random` +**Private networking:** +- VMs in the same NR communicate directly over the `r` bridge +- Different NRs are completely isolated — overlapping private IPs are fine +- Internet access is NAT'd through `public` → `br-ndmz` → node IP -What happens now is: +**Firewall:** nftables masquerade on traffic from the `private` interface so VMs can reach the internet via NAT. + +## ZDB Networking + +``` + br-hmy + ○ + │ veth + │ + ┌────┼──────────────────┐ + │ │ n- │ + │ ▼ │ + │ ┌──────┐ │ + │ │ eth0 │ │ + │ └──────┘ │ + │ │ │ + │ mycelium IP from │ + │ DMZ /64 subnet │ + │ route 400::/7 via gw │ + └────────────────────────┘ +``` -- VMs inside a single space can communicate directly over their bridge. -- Different networks resource can (and well) have conflicting IP and ranges but with no issue since each network is completely isolated from the other ones. +ZDB containers get a dedicated namespace with mycelium-only connectivity: -## Private Networks +1. Namespace `n` is created +2. A veth pair connects the namespace to `br-hmy` (host mycelium bridge) +3. A mycelium IP from the DMZ's `/64` subnet is assigned +4. Route `400::/7` via the mycelium gateway -To reach vms on local nodes using wireguard you need to: +ZDBs are accessible exclusively over mycelium. -- Deploy a networkwith valid pairs so you can be able to connect to the vm from your machine and add a container to this network. -For example: +## Private Networks (WireGuard) + +WireGuard is optional in network light. To access VMs on a node using WireGuard: + +1. Deploy a network with valid WireGuard peers: ```go WGPrivateKey: wgKey, WGListenPort: 3011, Peers: []zos.Peer{ - { - Subnet: gridtypes.MustParseIPNet("10.1.2.0/24"), - WGPublicKey: "4KTvZS2KPWYfMr+GbiUUly0ANVg8jBC7xP9Bl79Z8zM=", - - AllowedIPs: []gridtypes.IPNet{ - gridtypes.MustParseIPNet("10.1.2.0/24"), - gridtypes.MustParseIPNet("100.64.1.2/32"), - + { + Subnet: gridtypes.MustParseIPNet("10.1.2.0/24"), + WGPublicKey: "4KTvZS2KPWYfMr+GbiUUly0ANVg8jBC7xP9Bl79Z8zM=", + AllowedIPs: []gridtypes.IPNet{ + gridtypes.MustParseIPNet("10.1.2.0/24"), + gridtypes.MustParseIPNet("100.64.1.2/32"), + }, + }, +}, ``` -> **Note:** make sure to use valid two wg key pairs for the container and your local machine. - -- After the deployment the network can be accessed through wg with the following config. +2. Configure WireGuard on your local machine: ```conf [Interface] @@ -117,15 +202,110 @@ PersistentKeepalive = 25 Endpoint = 192.168.123.32:3011 ``` -- Bring wireguard interface up `wg-quick up ` -- Test the connection `wg` -![image](https://github.com/user-attachments/assets/ca0d37e2-d586-4e0f-ae98-2d70188492bd) - -- Then you should be able to ping/access the container `ping 10.1.1.2` -![image](https://github.com/user-attachments/assets/d625a573-3d07-4980-afc0-4570acd7a21f) +3. Bring the interface up: `wg-quick up ` +4. Test: `ping 10.1.1.2` -- Then you should be able to ping to the container `ping 10.1.1.2` +WireGuard IPs use CGNAT space: `100.64../16` derived from the NR subnet. ### Full Picture -![full](png/full.png) +``` + ┌──────┐ + │ VM │ + └─┬──┬─┘ + │ │ + tap (b-) │ │ tap (m-) + │ │ + ┌──────────────────────────┼──┼──────────────────────────────────────┐ + │ Host │ │ │ + │ │ │ │ + │ ┌─────────┐ r- │ │ m- ┌──────────┐ ┌──────────┐ │ + │ │my0 (tun)│ ○───────┘ └────○ │ br-ndmz │ │ br-hmy │ │ + │ │mycelium │ │ │ │100.127. │ └────┬─────┘ │ + │ │ -host │ │ │ │ 0.1/16 │ │ │ + │ └─────────┘ │ │ └────┬─────┘ │ │ + │ ┌────┘ ┌─────┘ │ │ │ + │ ┌────┐ │ veth │ veth veth │ veth │ │ + │ │NIC │ │ │ │ │ │ + │ └─┬──┘ │ │ │ │ │ + │ │att. │ │ │ │ │ + │ ▼ │ │ │ │ │ + │ ┌──────┐ │ │ │ │ │ + │ │ zos │ │ │ │ │ │ + │ │(DHCP)│ │ │ │ │ │ + │ └──────┘ │ │ │ │ │ + └────────────┼───────────────┼────────────────┼─────────────┼───────┘ + │ │ │ │ + ┌────────────┼───────────────┼────────────────┼──┐ ┌─────┼─────┐ + │ ▼ n- ▼ ▼ │ │ n- │ + │ ┌─────────┐ ┌──────────┐ ┌─────────┐ │ │ ▼ │ + │ │ private │ │ mycelium │ │ public │ │ │ ┌──────┐ │ + │ │.1 │ │ (myc │ │100.127. │ │ │ │ eth0 │ │ + │ │ gw │ │ daemon) │ │ x.x/16 │ │ │ └──────┘ │ + │ └─────────┘ └──────────┘ └─────────┘ │ │ myc IP │ + │ │ │ └───────────┘ + │ default route │ + │ via 100.127.0.1 │ + └────────────────────────────────────────────────┘ +``` + +## VM TAP Devices + +VMs connect via TAP devices created by the networker: + +| TAP name | Bridge | Purpose | +|----------|--------|---------| +| `b-` | `r` | Private NR network (IPv4) | +| `m-` | `m` | Per-NR mycelium (IPv6 in `400::/7`) | + +TAP names are derived from MD5 hash of the workload ID, Base58-encoded, truncated to 13 characters. + +## Subpackages + +| Package | Purpose | +|---------|---------| +| `bootstrap/` | Node boot: detect NICs, create `zos` bridge, DHCP probing | +| `bridge/` | Bridge create/delete/attach with VLAN filtering | +| `dhcp/` | DHCP probing via `udhcpc` and `dhcpcd` zinit service | +| `ifaceutil/` | Veth pairs, deterministic MAC/device-name from input bytes | +| `ipam/` | IPv4 IPAM from `100.127.0.3–100.127.255.254` via CNI host-local | +| `iperf/` | iperf3 service in public namespace | +| `macvlan/` | Macvlan device creation and IP/route installation | +| `macvtap/` | Macvtap device creation | +| `namespace/` | Network namespace create/delete/list/monitor | +| `options/` | sysctl wrappers: IPv6 forwarding, accept_ra, proxy_arp | +| `public/` | Public namespace setup, config persistence, exit NIC detection | +| `resource/` | NR lifecycle: bridges, namespace, mycelium daemon, WireGuard, nftables | +| `tuntap/` | TAP device creation for VM network interfaces | +| `types/` | Bridge/namespace name constants | + +## Interface + +```go +type NetworkerLight interface { + Create(name string, wl gridtypes.WorkloadWithID, network Network) (string, error) + Delete(name string) error + + AttachPrivate(name, id string, vmIp net.IP) (TapDevice, error) + AttachMycelium(name, id string, seed []byte) (TapDevice, error) + Detach(id string) error + + AttachZDB(id string) (string, error) + ZDBIPs(nsName string) ([]net.IP, error) + + GetSubnet(networkID NetID) (net.IPNet, error) + GetNet(networkID NetID) (net.IPNet, string, error) + GetDefaultGwIP(networkID NetID) (net.IP, error) + + Interfaces(iface string, netns string) (Interfaces, error) + ZOSAddresses(ctx context.Context) (<-chan NetlinkAddresses, error) + WireguardPorts() ([]uint, error) + Namespace(id string) (string, error) + + SetPublicConfig(cfg PublicConfig) error + LoadPublicConfig() (*PublicConfig, error) + UnSetPublicConfig() error + + Ready() error +} +``` diff --git a/docs/internals/network/readme.md b/docs/internals/network/readme.md index a4e7427c..b6bc28b3 100644 --- a/docs/internals/network/readme.md +++ b/docs/internals/network/readme.md @@ -1,9 +1,267 @@ -# zos networking +# Network Module -## Index +## ZBus -- [definitions of the vocabulary used in the documentation](definitions.md) -- [Introduction to networkd, the network manager of 0-OS](introduction.md) -- [Detail about the wireguard mesh used to interconnect 0-OS nodes](mesh.md) -- [Documentation for farmer on how to setup the network of their farm](setup_farm_network.md) -- [VLANS](vlans.md) +Network module is available on zbus over the following channel + +| module | object | version | +|--------|--------|---------| +| network | [network](#interface) | 0.0.1 | + +## Home Directory + +| directory | path | +|-----------|------| +| root | `/var/cache/modules/networkd` | +| networks | `{root}/networks/` — persisted NR configs | +| link | `{root}/networks/link/` — workload-to-network symlinks | +| ndmz leases | `{root}/ndmz-lease/` — IPv4 IPAM allocations | +| mycelium keys | `{root}/mycelium-key/` — per-NR mycelium key files | + +## Related Documentation + +- [Definitions of vocabulary](definitions.md) +- [Introduction to networkd](introduction.md) +- [WireGuard mesh](mesh.md) +- [Farm network setup](setup_farm_network.md) +- [VLANs](vlans.md) +- [Yggdrasil](yggdrasil.md) + +## Introduction + +The network module (`networkd`) manages all networking infrastructure on the node. It creates and manages bridges, network namespaces, WireGuard tunnels, overlay networks (Yggdrasil and Mycelium), and provides network connectivity for all workloads (VMs, containers, ZDB, QSFS). + +On boot, the node first establishes internet connectivity by finding a NIC with a working DHCP lease and creating the `zos` management bridge. Once connected, `networkd` starts and sets up the DMZ namespace, overlay networks, and registers itself on zbus to accept workload networking requests from the [provision engine](../provision/readme.md). + +### zinit unit + +```yaml +exec: networkd --broker unix:///var/run/redis.sock +after: + - boot +``` + +## Architecture + +### Network Layers + +``` + ┌─────────────────────────────────────────────┐ + │ Physical NIC(s) │ + └──────┬──────────────────┬───────────────────┘ + │ │ + ┌──────▼──────┐ ┌──────▼──────┐ + │ zos bridge │ │ br-pub │ (exit NIC or veth to zos) + │ (mgmt/dhcp)│ │ (public) │ + └──────┬──────┘ └──────┬──────┘ + │ │ + ┌───────────────┼──────────────────┼────────────────┐ + │ │ │ │ + ┌──────▼──────┐ ┌──────▼──────┐ ┌─────────▼────────┐ ┌────▼─────┐ + │ ndmz │ │ public │ │ zdb-ns-* │ │ br-ygg │ + │ namespace │ │ namespace │ │ ZDB namespaces │ │ br-my │ + │ (NAT gw) │ │ (pub IP) │ │ (pub+ygg+myc) │ │ (overlay)│ + └──────┬──────┘ └─────────────┘ └──────────────────┘ └────┬─────┘ + │ │ + ┌──────▼──────┐ │ + │ br-ndmz │ │ + │ (routing) │ │ + └──────┬──────┘ │ + │ │ + ┌──────▼───────────────────────────────────────────────────▼──┐ + │ n- namespaces (per-user network resources) │ + │ ├── macvlan on b- bridge (internal NR traffic) │ + │ ├── macvlan "public" on br-ndmz (internet via NAT) │ + │ ├── WireGuard w- (encrypted mesh to other nodes) │ + │ └── mycelium on m- bridge (optional overlay) │ + └──────┬─────────────────────────────────────────────────────┘ + │ + ┌──────▼──────┐ + │ VM TAPs │ t- on b- (private) + │ │ p- on br-pub (public IP) + │ │ t- on br-ygg (yggdrasil) + │ │ t- on m- (mycelium) + └─────────────┘ +``` + +### Bridges + +| Bridge | Name constant | Purpose | +|--------|--------------|---------| +| `zos` | DefaultBridge | Management bridge; gets DHCP from router; default internet uplink | +| `br-pub` | PublicBridge | Public traffic; wired to exit NIC (dual-NIC) or zos via veth (single-NIC) | +| `br-ndmz` | NdmzBridge | Routing bridge connecting NR namespaces to the ndmz for NAT'd internet | +| `br-ygg` | YggBridge | Yggdrasil overlay; shared by all ygg interfaces (`200::/7`) | +| `br-my` | MyceliumBridge | Mycelium overlay; shared by all mycelium interfaces (`400::/7`) | +| `b-` | — | Per-NR bridge; connects VMs/containers within that private network | +| `m-` | — | Per-NR mycelium bridge; connects VMs' mycelium tap devices in that NR | + +### Namespaces + +| Namespace | Purpose | +|-----------|---------| +| `ndmz` | Node's DMZ — NAT gateway for all NR namespaces. Has `npub4` (IPv4 via zos) and `npub6` (IPv6 via br-pub) | +| `public` | Optional — only when farmer sets a public config. Hosts the node's public IP, yggdrasil daemon, iperf | +| `n-` | Per-user private network resource with WireGuard mesh, NAT via ndmz, optional mycelium | +| `zdb-ns-` | Per-ZDB container; wired to br-pub + ygg + mycelium | +| `qfs-ns-` | Per-QSFS mount; wired to ndmz + ygg + mycelium | + +## NDMZ (DMZ Namespace) + +The `ndmz` namespace is the node's NAT gateway providing internet access to all NR namespaces. + +**Setup:** +1. Creates `br-ndmz` bridge on the host +2. Creates macvlan `tonrs` inside ndmz on `br-ndmz` with IP `100.127.0.1/16` +3. Creates macvlan `npub6` inside ndmz on `br-pub` for IPv6 (SLAAC) +4. Creates macvlan `npub4` inside ndmz on `zos` for IPv4 (DHCP) +5. Applies nftables: masquerade on `npub4`/`npub6`, exempt yggdrasil traffic +6. Starts DHCP monitoring goroutine for IPv4 lease renewal + +**NR attachment:** Each NR namespace gets a macvlan `public` on `br-ndmz` with an IP allocated from `100.127.0.2–100.127.255.254/16` (IPAM via CNI host-local backend). Default gateway points to `100.127.0.1`. + +## Public Namespace + +The `public` namespace is optional and created only when the farmer configures a public IP for the node. + +**Setup:** +1. Creates `br-pub` bridge and detects the exit NIC (physical NIC with public IPv6, or falls back to `zos`) +2. Creates `public` namespace with macvlan on `br-pub` +3. Assigns static IPv4/IPv6 from the public config +4. Starts yggdrasil daemon inside the namespace +5. Starts iperf service for network testing + +**Single-NIC vs dual-NIC:** When no dedicated public NIC is found, `br-pub` is connected to `zos` via a veth pair (single-NIC mode). When a separate NIC with public IPv6 is detected, it's attached directly to `br-pub` (dual-NIC mode). + +## Network Resources (NR) + +Each user private network gets a dedicated namespace `n-` with: + +- **Internal bridge** (`b-`): VMs connect here via TAP devices +- **NAT interface** (`public` macvlan on `br-ndmz`): Internet via ndmz +- **WireGuard** (`w-`): Encrypted mesh to the same network on other nodes. IP in `100.64.0.0/16` CGNAT space +- **Mycelium** (optional, `m-` bridge): Per-NR mycelium instance with its own key, running as a zinit service + +**IP addressing within NR:** +- NR gateway: `.1` (e.g., `10.1.2.1/24`) +- IPv6 (ULA): derived from MD5 of netID + IPv4 (`fd::/8` range) +- WireGuard: `100.64../16` derived from subnet + +## Overlay Networks + +### Yggdrasil + +Address space: `200::/7`. Ports: TCP 9943, TLS 9944, link-local 9945. + +Runs as a zinit service inside the `public` namespace (or `ndmz` if no public namespace). The node's yggdrasil identity is derived from its Ed25519 private key. + +Each ZDB/QSFS/VM gets a deterministic yggdrasil IP by hashing its identifier through the node's `/64` subnet. + +### Mycelium + +Address space: `400::/7`. Port: TCP 9651. + +Runs as a zinit service at the host level. Uses X25519 key derived from the node's Ed25519 key. Each NR can optionally run its own mycelium instance with a separate key. + +Each ZDB/QSFS/VM gets a deterministic mycelium IP by hashing its identifier through the node's `/64` subnet. + +## ZDB Networking + +ZDB containers get a dedicated namespace (`zdb-ns-`) with: +- `eth0`: veth pair to `br-pub` (public IPv6 via SLAAC) +- `ygg0`: veth pair to `br-ygg` (yggdrasil access) +- `my0`: veth pair to `br-my` (mycelium access) + +## QSFS Networking + +QSFS mounts get a dedicated namespace (`qfs-ns-`) with: +- `public`: macvlan on `br-ndmz` (internet via NAT, IPv4 from IPAM) +- `ygg0`: veth pair to `br-ygg` +- `my0`: veth pair to `br-my` +- Strict nftables: drops all inbound except established, loopback, prometheus on ygg, and ICMPv6 + +## VM TAP Devices + +VMs connect to networks via TAP devices created by the networker: + +| Method | TAP name | Bridge | Purpose | +|--------|----------|--------|---------| +| `SetupPrivTap` | `t-` | `b-` | Private NR network | +| `SetupPubTap` | `p-` | `br-pub` | Public IPv4/IPv6 | +| `SetupYggTap` | `t-` | `br-ygg` | Yggdrasil overlay | +| `SetupMyceliumTap` | `t-` | `m-` | Per-NR mycelium | + +## Public IP Filtering + +When a VM has a public IP, the networker sets up nftables bridge filter rules on `br-pub` to enforce anti-spoofing: only traffic from the assigned MAC+IP pair is allowed through the VM's tap device. + +## Subpackages + +| Package | Purpose | +|---------|---------| +| `bootstrap/` | Node boot: detect NICs, create `zos` bridge, find internet connectivity | +| `bridge/` | Bridge create/delete/attach with VLAN filtering support | +| `dhcp/` | DHCP probing via `udhcpc` and `dhcpcd` zinit service management | +| `ifaceutil/` | Veth pair creation, MAC/IPv6 derivation from input bytes | +| `iperf/` | iperf3 service in public namespace for network testing | +| `macvlan/` | Macvlan device creation, IP/route installation | +| `macvtap/` | Macvtap device creation for VMs | +| `mycelium/` | Mycelium daemon management, config generation, namespace plumbing | +| `namespace/` | Linux network namespace create/delete/list, address monitoring | +| `ndmz/` | DMZ namespace setup, DHCP monitoring, IPAM, nftables | +| `nr/` | Network resource lifecycle: bridge, namespace, WireGuard, mycelium, firewall | +| `options/` | sysctl wrappers: IPv6 forwarding, accept_ra, proxy_arp | +| `portm/` | WireGuard port allocator with filesystem-backed persistence | +| `public/` | Public namespace setup, public config persistence, exit NIC detection | +| `tuntap/` | TAP device creation for VM network interfaces | +| `types/` | Bridge/namespace name constants, interface info types | +| `yggdrasil/` | Yggdrasil daemon management, config generation, namespace plumbing | + +## Interface + +```go +type Networker interface { + CreateNR(wl gridtypes.WorkloadWithID, network Network) (string, error) + DeleteNR(wl gridtypes.WorkloadWithID) error + + SetPublicConfig(cfg PublicConfig) error + GetPublicConfig() (PublicConfig, error) + UnsetPublicConfig() + + EnsureZDBPrepare(id string) (string, error) + ZDBDestroy(ns string) error + + QSFSPrepare(id string) (string, string, error) + QSFSDestroy(id string) error + + SetupPrivTap(networkID NetID, name string) (string, error) + SetupPubTap(name string) (string, error) + SetupYggTap(name string) (YggTapConfig, error) + SetupMyceliumTap(name string, netID string, cfg MyceliumTapConfig) (string, error) + RemoveTap(name string) error + RemovePubTap(name string) error + DisconnectPubTap(name string) error + + SetupPubIPFilter(filterName, tapName, mac string, ip4 net.IP, ip6 net.IP) error + RemovePubIPFilter(filterName, tapName string) error + PubIPFilterExists(filterName string) bool + + GetSubnet(networkID NetID) (net.IPNet, error) + GetNet(networkID NetID) (net.IPNet, string, error) + GetDefaultGwIP(networkID NetID) (net.IP, error) + GetPublicIPv6Subnet() (net.IPNet, error) + GetPublicIPV6Gateway() (net.IP, error) + + Interfaces(iface string, netns string) (Interfaces, error) + Addrs(iface string, netns string) ([][]byte, string, error) + ZOSAddresses(ctx context.Context) (<-chan NetlinkAddresses, error) + DMZAddresses(ctx context.Context) (<-chan NetlinkAddresses, error) + YggAddresses(ctx context.Context) (<-chan NetlinkAddresses, error) + PublicAddresses(ctx context.Context) (<-chan OptionPublicConfig, error) + WireguardPorts() ([]uint, error) + + Metrics() (NetResourceMetrics, error) + Namespace(networkID NetID) (string, error) + GetIPv6From4(networkID NetID, ip4 net.IP) (net.IPNet, error) +} +```