Release 0.3.0

HTTPS-by-default for k7-api, cluster-wide Cilium isolation, first-class
--docker on Kata and k7d, and RuntimeClass k7-fc. Playbook pins k7d 0.6.0.
This commit is contained in:
G
2026-09-13 17:30:37 +02:00
parent 1420780f19
commit 76a8139774
92 changed files with 9016 additions and 901 deletions
+180
View File
@@ -348,3 +348,183 @@ that includes `-c` trips its self-execution guard.
**Reference:** none (Typer + PyInstaller).
**Time lost:** ~20 min (misread as sidecar/egress failure).
---
## 13. k7d `--docker`: BuildKit TLS vs dockerd pull; `-p` is guest-host netns (spec 37a-inc2)
**Symptom:** `docker run --rm hello-world` and `docker run alpine:3.21`
succeeded in a `--docker` k7d sandbox, but `docker build -t myapp .`
from alpine:3.21 failed with `x509: certificate signed by unknown
authority` on `auth.docker.io`, and `wget http://127.0.0.1:8080` from
`k7 exec` never saw an nginx published with `-p 8080:80`.
`K7Core.exec_command` also always reports `exit_code=0`, so a failed
`docker build` looked like a successful tag that `docker run myapp`
then could not find.
**Root cause:** dockerd's pull path uses the payload CA bundle
(`guest/docker/payload/etc/ssl/certs/ca-certificates.crt`). Default
BuildKit metadata fetch (docker driver, buildx 0.20 / buildkit v0.18)
does not, so `docker build` / compose `build:` fail TLS while `docker
pull` works. `docker run -p 8080:80` publishes in the **guest host**
netns (k7d inc1 probes it via agent vsock wget). `k7 exec` is the CRI
container netns, so localhost:8080 is the wrong place. `exec_command`
never reads the kubectl-exec exit code.
**Fix:** tests use `DOCKER_BUILDKIT=0 docker build` (classic builder →
dockerd pull + CA) and probe the inner nginx with `docker exec web wget
http://127.0.0.1:80`. Guest exit codes are asserted via an `__K7_EC:$?`
marker. Do not "fix" this by setting `DOCKER_BUILDKIT=0` as a product
default.
**Reference:** k7d `crates/k7d/tests/test_docker_service.rs`
`test_docker_warm_fork_nginx_survives` (`agent_sh` wget 8080);
k7d `guest/docker/payload/etc/ssl/certs/ca-certificates.crt`.
**Time lost:** ~40 min (first integration pass).
---
## 14. k3s containerd does not stamp the runtime-handler; `k7d_version` 0.5.0 would clobber the new shim (spec 38a-inc5)
**Symptom:** RuntimeClass `k7-fc` is not enough for the shim to see
handler `k7-fc` — this node's k3s 1.36 / containerd 2.3.3 never sets
`io.kubernetes.cri.runtime-handler` on the sandbox OCI spec (k7d
CHALLENGES #228). Also, `k7 install --backend k7d-fc` with the playbook
default `k7d_version: 0.5.0` would download the public GitHub tarball
and overwrite `/usr/local/bin/containerd-shim-k7-v1` / `k7d` with a
build that does not know ConfigPath.
**Root cause:** ConfigPath on `runtimes.k7-fc` is the working signal.
Kata Firecracker lives at `/opt/kata/bin` (playbook pin ~v1.14);
k7d-fc installs upstream v1.16.1 at `/usr/local/bin` — different
paths, do not share binaries. `grep runtimes.k7` matches `k7-fc`.
**Fix:** playbook writes `/etc/k7d/shim-k7-fc.toml` and a ConfigPath
`[options]` table **without** `BinaryName`. Install Firecracker via
vendored `src/k7/deploy/k7d-fc/install-firecracker.sh` (sha fatal).
Smoke-test the playbook with `--tags k7d-fc` (facts tagged `always`)
and `--k7d-artifact` pointing at a tarball built from the k7d tree
that just passed `make remote-check`, never the 0.5.0 GitHub URL.
`k7_has_k7d` is true for `k7d` **or** `k7d-fc`. The allowed-backend
`difference()` list must include `k7d-fc` or a k7d-fc-only install
fails validation before any task runs.
A fifth trap: the playbook **replaces** the whole containerd template
from the selected backends. A full `k7 install --backend k7d,k7d-fc`
on a mixed k7d-dev node would drop kata/nvidia/wasm runtime blocks.
`--tags k7d-fc` installs Firecracker + toml + RuntimeClass + labels
and does not rewrite the template (`runtimes.k7-fc` is also
self-patched by k7d's `ensure_k7_fc_runtime_registered`).
**Reference:** k7d CHALLENGES #228, `docs/backends.md`.
**Time lost:** named in the spec before implementation (~15 min
confirming live sandbox inspect on the node).
---
## 15. CRI exec into a k7-fc guest hung; kube Ready never flipped (spec 38a-inc5 / fixed 38a-inc6)
**Symptom:** `tests/integration/test_k7d_fc.py` created a Running
`runtimeClassName: k7-fc` pod (shim log `backend=Firecracker`) then
hung forever on `K7Core.exec_command`. `timeout 15 k3s kubectl exec
… -- echo hello` exited 124. The sandbox's exec readiness probe
(`/bin/sh -c true`, 5s) also timed out, so the container never
became Ready.
**Root cause:** Firecracker guests are reached over the jail vsock
UDS (`FcUds`), not host `AF_VSOCK`. The daemon publishes
`guest_cid = 0` (`NO_HOST_CID`). A host that dialled CID 0 blocked
in `connect()` with no timeout, so CRI Wait never saw an exit code.
This was not ConfigPath / RuntimeClass selection (k7d #228 / #230)
and not virtiofs (k7d-fc has none).
**Fix (inc6):** k7d refuses `AF_VSOCK` CIDs 0–2 and proves
`timeout 15 kubectl exec … -- echo hello` plus kube Ready on
`k7-fc` (`test_k7_fc_exec_probe_ready_and_native_exec`). Install that
artifact with `--k7d-artifact` (not GitHub `latest`). k7-fc tests
wait on Ready again; `test_k7d_fc_exec_echo_hello` and
`TestDockerK7dFc` exec into the guest. Ready is a valid signal.
**Reference:** k7d CHALLENGES #228 / #230 (selection) and #232 (CID 0).
**Time lost:** ~45 min in inc5 (pytest sat on exec until the SSH
session dropped; leftover ns `k7-test-fe13c2d7` had to be deleted
by hand).
---
## 16. Kata `--docker` vehicle: persist-bind hides CLI; two-PVC fork; shareProcessNamespace (spec 22a)
**Symptom / traps while bringing `--docker` to kfd/kql:**
1. **kql persist-bind overlays `/usr`.** The spec mounts the docker CLI via
emptyDir `subPath` onto `/usr/local/bin/docker` and
`/usr/local/lib/docker/cli-plugins/*`. kql's persist-bind then
`mount --bind`s the PVC's `/usr` over that path, so compose/buildx
vanish. Fix: also mount the CLI emptyDir at `/run/k7/docker-cli`
(persist-bind does not overlay `/run`) and restage the canonical
paths immediately after the `/usr` bind. Vehicle path-sharing
fingerprints that staging file, not `/usr`.
2. **`hostPID` is the node, not the Kata guest.** Path sharing needs the
vehicle to see the sandbox container's rootfs. `shareProcessNamespace:
true` shares the *guest* PID namespace between the two CRI containers.
`hostPID: true` would be the Kubernetes node (and is the wrong trust
domain).
3. **Two Longhorn VolumeSnapshots are not atomic.** kql fork/restore
snapshots the root PVC and the docker-graph Block PVC separately.
Each is crash-consistent (`sync` in sandbox + vehicle first);
containerd's boltdb recovers. Do not claim cross-volume atomicity.
4. **Never fall back to virtio-fs for the graph.** CHALLENGES #10's vfs
tax and virtiofsd wedge are why the graph is `volumeDevices` + ext4 +
overlay2 or fail loud. Keep the relaxed exec-probe timings (3/15/12/4).
5. **Alpine dind `VOLUME /var/lib/docker` is virtio-fs on Kata.** Unmount
it, mkfs only when blkid is not already ext4 (busybox blkid ignores
`-o value`), then `mount -t ext4`.
6. **`mount --bind` from `/proc/PID/root` is EINVAL on Kata.** Reading
through that path works. Path sharing **symlinks** `/tmp` `/home`
`/root` `/opt` `/workspace` in the vehicle onto the sandbox rootfs so
`docker run -v /tmp/x:/x` resolves.
7. **OpenEBS LVM `GetCapacity` reports VG VFree, not thin-pool free.**
`kata-vg` is almost entirely the thin-pool plus a few GiB leftover.
With `storageCapacity: true` (chart default) the scheduler sees ~5Gi
and never binds a 20Gi `k7-docker-lvm` PVC. Disable capacity tracking;
thin LVs come from the pool (`thinProvision: yes`).
8. **Kata FC virtio-fs file `subPath` mounts are invisible to Docker
plugin discovery.** `docker compose` is not a docker command on kfd
unless compose/buildx are copied into `~/.docker/cli-plugins` from
the directory-mounted emptyDir. kql persist-bind already restages
them as regular files.
**Reference:** spec 22a-kata-docker-vehicle; CHALLENGES #10, #13.
**Time lost:** kql integration (vehicle CrashLoopBackOff on graph mount
and path-share).
---
## 17. k7-fc `--docker` Ready wait saw OutOfcpu corpses; CRI exec after pause hung
**Symptom (bench wait):** `test_bench_k7d_fc` at 4 CPU Guaranteed left a
Ready pod plus a pile of Failed/`OutOfcpu` siblings (`requested: 4050,
used: 9890, capacity: 12000` — the just-deleted k7d sandbox's 4 CPU
still counted). `_wait_all_containers_ready` only looked at `items[0]`,
which was a Failed pod, so the wait would have timed out at 300s while
one replica was already Ready.
**Fix:** wait until **any Running** pod has all containers Ready (same
shape as `bench_backend_lifecycle._pod_ready`). Forked-child `docker info`
asserts overlay2 on both k7d and k7d-fc.
**Symptom (lifecycle resume):** alpine k7d-fc pause returned in 0.18s
and k7d logged `resumed vm-… (guest_cid=0)`. `_wait_exec` after resume
never returned: kubelet `Readiness probe failed: "/bin/sh -c true"
timed out after 5s`. Create→Ready and exec *before* pause had worked
(2.14s / 0.18s). This is #15 again on the post-pause path, not a
docker-perf failure. Lifecycle was aborted; do not quote k7d-fc
resume→exec from that run.
**Reference:** CHALLENGES #15; k7d #232.
**Time lost:** ~15 min on the OutOfcpu wait; lifecycle resume hung until
the pytest process was killed (~8 min).
+126
View File
@@ -5,6 +5,132 @@ All notable changes to this project are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [0.3.0] — 2026-09-12
HTTPS-by-default for `k7-api`, cluster-wide Cilium isolation, first-class
`--docker` on Kata and k7d, and RuntimeClass `k7-fc`. The playbook now
pins k7d **0.6.0**.
### Changed
- **`k7 install` serves `k7-api` over HTTPS by default**.
`/etc/k7/api_endpoint` is now `https://<first-master-ip>:31007`. A
Caddy sidecar terminates TLS; the API container and its HTTP probes
stay on `:8000`. The default cert is a playbook-minted cluster CA
(`/etc/k7/tls/ca.crt`) — Let's Encrypt cannot issue for a bare IP.
**Breaking for existing laptop clients:** `K7_API_URL=http://...` must
switch to `https://` and, on the default track, install the CA
(`k7 config set api.ca ./ca.crt`, or `--api-ca` / `K7_API_CA`).
`verify=False` on an `https://` URL is an error. Optional
`--api-hostname` (Let's Encrypt via Caddy, not Certbot) and
`--api-tls-cert`/`--api-tls-key`. `--api-insecure-http` restores
today's HTTP NodePort. Keys no longer travel in cleartext; there is
still no rate limiting.
### Added
- **Cluster-wide sandbox platform isolation**. A deny-only
`CiliumClusterwideNetworkPolicy` (`k7-sandbox-platform-deny`) stops every
sandbox — in every egress mode, including `--egress-open` — from reaching
the node it runs on, the other nodes, the Kubernetes API server, cloud
metadata / link-local, and the `kube-system` + `longhorn-system` pods
(CoreDNS excepted). Previously an `--egress-open` sandbox could dial the
node's SSH/kubelet/k3s ports, `10.43.0.1:443`, `k7-api` and Longhorn.
The policy sets `enableDefaultDeny: false`; on Cilium 1.19 an
`egressDeny` section alone would otherwise switch the selected sandboxes
to default-deny and break open egress. Cilium-only: `--cni flannel`
clusters do not get this isolation (documented in `SECURITY.md`).
- **Opt-in sandbox ingress** — `--ingress-port` opens TCP ports on a
sandbox and `--ingress-from` (`sandbox:<name>` / `namespace:<ns>` /
`cidr:<cidr>`) scopes who may connect, as stock v1 `NetworkPolicy` rules
(so it works on Flannel too). Default is unchanged: deny all ingress. An
opened port with no source is reachable only from sandboxes in the same
namespace; the internet needs `cidr:0.0.0.0/0` spelled out. `k7 create`
warns that a `cidr:` source does not scope in-cluster peers on Cilium
(an `ipBlock` peer is not evaluated for pod-to-pod traffic) — use
`sandbox:` / `namespace:` for that.
- **`--expose-port`** — publishes a sandbox port outside the cluster through
a `NodePort` Service with `externalTrafficPolicy: Local`, so the pod sees
the real client IP instead of a SNAT'd one. Requires a matching
`--ingress-port`, prints the resolved `http://<node-ip>:<nodeport>`,
reports NodePorts in `k7 list`, and is removed by `k7 delete`.
- **`k7 install --api-allow-cidr`** (repeatable) — restricts the `k7-api`
NodePort to the operator's source CIDRs via a `CiliumNetworkPolicy` on
the API pod (`k7-api-ingress`) plus `externalTrafficPolicy: Local` so the
pod sees the real client IP. Off by default: without the flag no policy
is created and the Service is untouched. Cilium-only (`--cni flannel`
fails at the CLI — the reserved `host` / `remote-node` / `health` /
`kube-apiserver` peers the kubelet probes need are not expressible in a
v1 `NetworkPolicy`); invalid CIDRs fail before Ansible runs, and
`0.0.0.0/0` warns. This is defence in depth for operators who know their
client CIDRs, **not** a secure API: the NodePort is HTTPS by default (see
Changed above) and there is still no rate limiting. The policy selects a
pod endpoint, never the host, so a wrong CIDR never affects SSH or the
k3s API port.
- **`k7 install --hubble`** — opt-in Cilium Hubble flow observability
(relay + pinned `hubble` CLI). Off by default; not a security control.
Combined with `--cni flannel` it fails loudly. Hubble UI is not
installed (no auth, public node IP).
- **`--docker`** — first-class Docker as a guest service. On `k7d` and
`k7d-fc` the guest agent supervises a pinned `dockerd` with overlay2
on a per-sandbox virtio-blk disk; forks stay overlay2. On Kata (`kfd`
/ `kql`) the same flag injects a privileged docker-vehicle with
overlay2 on a block disk (`kql` persists/forks the graph; `kfd` is
ephemeral). `--sidecar docker` remains a deprecated alias.
- **RuntimeClass `k7-fc`** (`--backend k7d-fc`) — Firecracker under the
jailer next to the k7d daemon, CRI exec, and overlay2 on forked
`--docker` graphs.
- Kata guest seccomp is on (`disable_guest_seccomp = false`).
- Playbook default `k7d_version` is **0.6.0** (was 0.5.0). `--docker` and
`k7-fc` need that tarball; an older k7d fails loudly.
### Fixed
- **API sandbox delete returned 400 on every call** because
`k7 delete` always removes the `{name}-expose` Service and the
`k7-api` ClusterRole had no `services` verbs. Kubernetes answers
403 (not 404) when the verb is missing, so even sandboxes that
never used `--expose-port` failed to delete through the API.
- **`k7 snapshot gc` left orphan VolumeSnapshotContents behind**, so
snapshot-lifecycle tests (and operator deletes) pinned namespaces in
Terminating on `volumesnapshotcontent-bound-protection` /
`pvc-as-source-protection`. GC now reaps contents whose
VolumeSnapshot is already gone or deleting, then drops the PVC
source-protection finalizer (merge-patch — strategic-merge leaves
those CRD/PVC finalizers in place). Integration teardown calls that
same sweep instead of a kubectl jsonpath that silently missed items.
- **`k7 install` treated a busy apiserver as "Cilium is missing"**. The
DaemonSet check used `failed_when: false` and `rc != 0` as absent, so a
brief `unable to handle the request` started `cilium install` on a
cluster that already had Cilium. Presence is now present / absent /
unknown: unknown retries, then fails loud, and never installs. The
Hubble relay check got the same split (`cilium hubble enable`
Helm-upgrades the agent — do not run it because kubectl blipped). A
reinstall no longer fails `cilium status --wait` on warnings when the
DaemonSet is Ready.
- **`GET /api/v1/nodes/storage` skipped namespace authorization**
(CWE-862 / CWE-285). A leftover from the 0.2.1 scoping fix: this
cluster-scoped route was guarded only by `verify_api_key`, so a
namespace-scoped tenant key could read cluster-wide per-node storage
topology (node names, pool utilization, per-agent error strings). The
route now calls `authorize_namespace(..., all_namespaces=True)`;
scoped keys get 403, unscoped keys are unchanged. Reported privately
by **Ahmed Ibrahim** ([@skeletonsec](https://github.com/skeletonsec)),
who held disclosure — thank you.
- `k7 install --cni flannel` failed at "Apply K7 API manifests" because the
directory contained a `CiliumNetworkPolicy` and no `cilium.io` CRD
exists on Flannel. Cilium-only manifests now live in
`manifests/k7-api/cilium/` and are applied by a separate task gated on
the CNI.
- **`k7 fork` produced a sandbox with no network policy at all** — neither
the deny-ingress `NetworkPolicy` nor the egress policy was created, so a
fork of a locked-down sandbox came up with unrestricted egress. Forks now
inherit the source's egress configuration and are rolled back if their
policy cannot be created.
## [0.2.2] — 2026-08-18
Docs and install-path release for a two-node `apt` install that ships
+123 -28
View File
@@ -35,7 +35,7 @@ whose fork carries live memory. kfd has no fork by design. Docker-in-VM
sidecar (`--sidecar docker`) works on all three; k7d's `hello-world` pull
is the slow path (tmpfs + virtio + NAT), matching the 2026-08-10 note.
## Backend lifecycle: kql vs k7d — 2026-08-10 (spec 9a M11)
## Backend lifecycle: kql vs k7d — 2026-08-10
Hetzner AX41 dedicated node (Ryzen 5 3600, 64 GiB, NVMe), Ubuntu 24.04,
kernel 6.8.0-137, k3s v1.36.3, flannel CNI, Longhorn 1.10 (r=1), k7d 0.1.0.
@@ -90,7 +90,7 @@ Fork is ~3x slower than a cold create. The snapshot and clone PVC steps add ~6s,
Measured via integration tests (`tests/integration/test_qemu.py`) on 2026-04-10.
## Docker workloads inside sandboxes — 2026-05-29 (spec 10b)
## Docker workloads inside sandboxes — 2026-05-29
Hetzner AX52 dedicated node (Ryzen 7 7700, 64 GiB, NVMe), Ubuntu 24.04,
kernel 6.8.0-100, k3s v1.35.5, docker host 29.5.2 / sandbox 27.5.1,
@@ -238,14 +238,14 @@ Raw per-leg logs and the aggregated CSV are under `bench/docker-perf/results/`
(gitignored — keep them in agent or local scratch space, paste into the
table above when adding a new run).
## Docker-in-VM under real Longhorn replica counts (2026-08, spec 18e/18h)
## Docker-in-VM under real Longhorn replica counts (2026-08)
The "r=2 vs r=1 indistinguishable" result above is **invalid**: the old
bench patched a Longhorn *setting* that only affects newly-created volumes,
so both legs actually ran r=1 (CHALLENGES.md #2). `bench_docker_perf.py` now
sets real per-volume replica counts (`k7-ql-r3` leg). Host / kfd / r1 / r2
medians from the 18e HA run (5 reps); **k7-ql-r3 column re-filled in
spec 18h** after the virtiofsd wedge fix (5 reps, zero VM restarts):
medians from the HA run (5 reps); **k7-ql-r3 column re-filled later** after
the virtiofsd wedge fix (5 reps, zero VM restarts):
| op | host | k7-fd | k7-ql-r1 | k7-ql-r2 | k7-ql-r3 |
|---|---|---|---|---|---|
@@ -254,37 +254,132 @@ spec 18h** after the virtiofsd wedge fix (5 reps, zero VM restarts):
| run read (venv tree cat) | — | — | — | — | 47.7s |
| run cpu (10s budget) | 10.5s | 10.4s | 41.5s | 47.8s | 56.4s |
Two takeaways (spec 18f issue 8 / 18h):
Two takeaways:
- **The r1→r2 jump dominates the redundancy cost** (build 108→271s; r2→r3
adds only ~8% on build): the second replica forces synchronous
cross-node writes, the third mostly parallelizes with them. Post-wedge
r3 `run io` (88.3s) is higher than the old single-rep 64.5s sample —
that sample was the lucky survivor of a ~50% kill rate, not a median.
- **kql-r3 IO wedge — root-caused and FIXED (spec 18g):** the "VM exec
path dies after a run_io rep" wedge was not guest memory, not dockerd,
and not Longhorn faulting — the guest was healthy (load 0.4, 1.5 GB
free, zero dirty pages, clean dmesg) at the moment of death. The killer
was the **kata shim**: kata's default virtiofsd runs with
- **kql-r3 IO wedge — root-caused and FIXED:** the "VM exec path dies
after a run_io rep" wedge was not guest memory, not dockerd, and not
Longhorn faulting — the guest was healthy (load 0.4, 1.5 GB free, zero
dirty pages, clean dmesg) at the moment of death. The killer was the
**kata shim**: kata's default virtiofsd runs with
`--thread-pool-size=1`, so all virtio-fs IO (container rootfs + the
Longhorn-PVC-backed `/var/lib/docker`) serializes through one thread.
A `docker run` of the ~790 MB bench image makes vfs copy the whole
rootfs and then fsync 512 MB through that single thread against an
r=3 volume (~60–80 s saturated); any agent RPC touching virtio-fs
blocks behind it, the shim's agent health ping (`CheckRequest`) times
out, and the shim declares "Dead agent" and kills the healthy VM
(`sandbox stopped unexpectedly`, pod sandbox recreated). Repro rate was
~50% per run_io rep. Fix: `k7 install` now sets
`virtio_fs_extra_args = ["--thread-pool-size=16", ...]` in the
kata-qemu config — 18h re-ran 5/5 `run_io` + 5/5 `run_read` with zero
VM restarts on the same r=3 volume (`run_read` median 47.7s).
Exec-probe pressure was reduced too (kubelet's default 1 s exec-probe
timeout sprayed cancelled ttrpc execs — `docker info` legitimately
takes >1 s while dockerd copies vfs layers), which cuts the
`ttrpc: received message on inactive stream` noise but was NOT
sufficient on its own.
Longhorn-PVC-backed `/var/lib/docker`) serializes through one thread. A
`docker run` of the ~790 MB bench image makes vfs copy the whole rootfs
and then fsync 512 MB through that single thread against an r=3 volume
(~60–80 s saturated); any agent RPC touching virtio-fs blocks behind
it, the shim's agent health ping (`CheckRequest`) times out, and the
shim declares "Dead agent" and kills the healthy VM (`sandbox stopped
unexpectedly`, pod sandbox recreated). Repro rate was ~50% per run_io
rep. Fix: `k7 install` now sets `virtio_fs_extra_args =
["--thread-pool-size=16", ...]` in the kata-qemu config — we re-ran 5/5
`run_io` + 5/5 `run_read` with zero VM restarts on the same r=3 volume
(`run_read` median 47.7s). Exec-probe pressure was reduced too
(kubelet's default 1 s exec-probe timeout sprayed cancelled ttrpc execs
— `docker info` legitimately takes >1 s while dockerd copies vfs
layers), which cuts the `ttrpc: received message on inactive stream`
noise but was NOT sufficient on its own.
## k7d docker-perf leg (2026-08-11, partial)
## k7d vs k7d-fc `--docker` overlay2 — 2026-09-10
Same node as the 08-10 lifecycle cut (k7-node-01 / Hetzner AX41, Ryzen 5
3600, 64 GiB, NVMe), Ubuntu 24.04, kernel 6.8.0-139, k3s v1.36.3, k7d
0.5.0 (locally built artifact, not the public GitHub tarball of the same
version string). Guests 3Gi / 4 CPU, `DOCKER_BUILDKIT=0`, graph on
virtio-blk scratch, `Storage Driver: overlay2` on the source **and** on
the forked child. 3 runs per cell after one warm-up; median (range).
⚠ marks `(max−min)/median > 0.30`.
```bash
K7_BENCH_ENVS=k7d,k7d-fc K7_BENCH_REPS=3 K7_BENCH_WARMUP=1 \
K7_BENCH_OUT=/tmp/bench-out \
uv run pytest -m bench tests/integration/bench_docker_perf.py::test_bench_k7d \
tests/integration/bench_docker_perf.py::test_bench_k7d_fc -v -s
uv run python bench/docker-perf/render.py -i /tmp/bench-out/bench-results-*.csv \
--title "k7d vs k7d-fc --docker overlay2" --hardware "<one-line note>"
```
| Operation | k7d | k7d-fc |
|---------------------------|-----|--------|
| pull debian:12-slim | 11.3 s (9.06 s–11.5 s) | ⚠ 8.16 s (7.24 s–11.2 s) |
| build (no-cache) | 52.0 s (49.8 s–53.2 s) | 51.6 s (50.4 s–52.5 s) |
| build (cached) | ⚠ 469 ms (269 ms–477 ms) | 470 ms (369 ms–471 ms) |
| run cpu (10s budget) | 10.8 s (10.7 s–10.8 s) | 10.7 s (10.6 s–10.7 s) |
| run io (2k small + 512 MB)| ⚠ 1.18 s (1.11 s–1.48 s) | 1.28 s (1.28 s–1.38 s) |
| run read (venv tree cat) | 774 ms (674 ms–789 ms) | 674 ms (670 ms–777 ms) |
| fork warm engine (API) | 6.23 s (n=1) | 8.35 s (n=1) |
| fork → Ready + overlay2 | 6.79 s (n=1) | 9.13 s (n=1) |
Forked-child log line: `# storage_driver_child=Storage Driver: overlay2` on
both backends (`TestDockerK7d` / `TestDockerK7dFc` also assert this).
k7d-fc is host-parity with k7d on the storage-bound ops (no-cache build
~52 s, `run_io` ~1.3 s, `run_read` ~0.7 s). That is the overlay2 /
virtio-blk number, not the vfs-shaped ~37 s `run_io` in the earlier
sidecar rows or the 4.24 s kql Longhorn-block overlay2 row. Warm-engine
fork is the k8s adopt path (not the ~5 ms VMM CoW floor); alpine
create→Ready / fork→exec on the same node this run were **2.15 s /
2.54 s** (k7d) and **2.14 s / 2.45 s** (k7d-fc), n=1.
k7-fc CRI exec after pause/resume hung (readiness probe timeout,
`guest_cid=0` retained) — see CHALLENGES #17. Do not quote a
resume→exec number for k7d-fc from this cut.
## k7d `--docker` (first-class guest service)
`k7 create --docker --backend k7d ubuntu:24.04` stamps
`k7d.katakate.org/docker=true` (no sidecar, no privileged). Graph is a
per-sandbox virtio-blk scratch disk (overlay2), so `run io` is no longer
vfs/tmpfs-shaped. `k7 fork` of a warm engine is in-scope: the child keeps
dockerd and running inner containers.
Reproduce:
```bash
K7_BENCH_ENVS=k7d K7_BENCH_REPS=1 K7_BENCH_WARMUP=0 \
uv run pytest -m bench tests/integration/bench_docker_perf.py::test_bench_k7d -v -s
```
1-rep log from k7-node-01 (2026-09-07, k7d + guest docker payload). **Superseded**
by the 2026-09-10 k7d vs k7d-fc table above (full pull/build/cpu/io/read
column, child overlay2). Kept as the first overlay2 `run_io` datapoint:
| op | k7d `--docker` | notes |
|---|---|---|
| `run_io` | **1.24 s** | overlay2 on virtio-blk; 2k files + 512 MB fsync |
| `fork_warm_engine` | **6.26 s** | CoW of live dockerd (not the 100–300 ms memory-fork floor; includes k7 Deployment adopt) |
Historical sidecar/tmpfs numbers from 2026-08-11 are in the section below.
## kql `--docker` (overlay2 on Longhorn block)
`k7 create --docker --backend kql ubuntu:24.04` injects a privileged
`docker-vehicle` and a second Longhorn **Block** PVC (`<name>-docker-lh`).
The graph is ext4 on that device; dockerd is forced to overlay2. This is
the number that used to be vfs-shaped (~37 s `run_io` in the earlier
sidecar/subPath rows above). Do not treat a vfs `run_io` as the kql
`--docker` baseline.
Reproduce:
```bash
K7_BENCH_ENVS=k7-ql-r1 K7_BENCH_REPS=1 K7_BENCH_WARMUP=0 \
uv run pytest -m bench tests/integration/bench_docker_perf.py::test_bench_k7_ql_r1 -v -s
```
Numbers from this increment (<node-ip> / k7-node-01, 2026-09-10,
`K7_BENCH_ENVS=k7-ql-r1 K7_BENCH_REPS=1 K7_BENCH_WARMUP=0`,
`# docker_data_path=longhorn-block-overlay2`,
`Storage Driver: overlay2`, Docker 27.5.1, Longhorn r=1 on one node):
| op | kql `--docker` overlay2 / Longhorn block | notes |
|---|---|---|
| `run_io` | **4.24 s** | 2k files + 512 MB `dd conv=fsync`; overlay2 on Block PVC (not the ~37 s vfs row) |
## k7d docker-perf leg (2026-08-11, partial, **sidecar / tmpfs**, superseded)
`test_bench_k7d` is wired (`K7_BENCH_ENVS=k7d`). Guest sized at **3Gi /
4 vCPU** — largest size that reliably reaches Ready on this k7d build;
+62 -25
View File
@@ -76,33 +76,36 @@ The Tech Stack
Sandbox backends
</h3>
`k7 install --backend <kfd|kql|k7d>` provisions one or more backends per node; `k7 create --backend …` picks one per sandbox. See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians).
`k7 install --backend <kfd|kql|k7d|k7d-fc>` provisions one or more backends per node; `k7 create --backend …` picks one per sandbox. See [docs/BACKENDS.md](docs/BACKENDS.md) for the architecture and [PERFORMANCE.md](PERFORMANCE.md) for the full measurements (Hetzner AX41 node, medians).
| | `kfd` (kata-firecracker-devmapper) | `kql` (kata-qemu-longhorn) | `k7d` |
|---|---|---|---|
| VMM | Firecracker (Kata) | QEMU (Kata) | k7d (custom KVM VMM) |
| RuntimeClass | `kata` | `kata-qemu` | `k7` |
| Sandbox storage | devmapper thin-pool (needs a spare raw disk) | Longhorn PVC (replicated, persistent) | erofs images + reflink XFS + guest tmpfs |
| Create → Ready* | not re-measured† | 17.1s | **2.1s** |
| Named snapshot* | — | 6.5s (Longhorn, disk-only) | — (VM snapshot trees via the k7d API) |
| Fork → usable* | — | 46.7s (disk clone + cold boot) | **~5 ms VM CoW fork**; **~2.4 s** end-to-end via k7/k8s (pod Ready + exec) |
| Pause / resume* | scale to 0 / 1 | 1.3s / 4.1s (disk survives) | **0.2s / 0.3s (VM frozen in place, memory survives)** |
| Docker-in-VM sidecar | ✅ (ephemeral docker data) | ✅ (persistent docker data; fastest `docker pull`) | ✅ (VM-lifetime docker data) |
| Cross-pod persistence | ✗ | ✅ snapshots/restore | ✗ (fork carries state instead) |
| | `kfd` (kata-firecracker-devmapper) | `kql` (kata-qemu-longhorn) | `k7d` | `k7d-fc` |
|---|---|---|---|---|
| VMM | Firecracker (Kata) | QEMU (Kata) | k7d (custom KVM VMM) | k7d + Firecracker jailer |
| RuntimeClass | `kata` | `kata-qemu` | `k7` | `k7-fc` |
| Sandbox storage | devmapper thin-pool (needs a spare raw disk) | Longhorn PVC (replicated, persistent) | erofs images + reflink XFS + guest tmpfs | same as k7d (no virtiofs / hostPath) |
| Create → Ready* | not re-measured† | 17.1s | **2.1s** | **2.1s** |
| Named snapshot* | — | 6.5s (Longhorn, disk-only) | — (VM snapshot trees via the k7d API) | — (same as k7d) |
| Fork → usable* | n/a (rejected) | 46.7s (disk clone + cold boot) | **~5 ms VM CoW fork**; **~2.4 s** end-to-end via k7/k8s (pod Ready + exec) | **~2.5 s** end-to-end (same CoW; child `--docker` stays overlay2) |
| Pause / resume* | scale to 0 / 1 | 1.3s / 4.1s (disk survives) | **0.2s / 0.3s (VM frozen in place, memory survives)** | **0.2s / —‡** |
| Docker in the VM | **`--docker`**: vehicle + overlay2 on ephemeral LVM block | **`--docker`**: vehicle + overlay2 on Longhorn block (fork/restore) | **`--docker`**: in-guest dockerd, overlay2 on virtio-blk, **forkable** | **same guest dockerd**, overlay2, **forkable** |
| Cross-pod persistence | ✗ | ✅ snapshots/restore | ✗ (fork carries state instead) | ✗ |
\* medians of 3 on one Hetzner AX41 node — methodology, ranges, and the docker-sidecar
\* medians of 3 on one Hetzner AX41 node — methodology, ranges, and docker-in-VM
numbers are in [PERFORMANCE.md](PERFORMANCE.md).
† kfd needs a spare raw disk the benchmark node didn't have; its docker-workload numbers
are in the [PERFORMANCE.md](PERFORMANCE.md) spec-10b section.
† kfd needs a spare raw disk the lifecycle-bench node didn't have; Show HN
measured kfd create→exec 3.74s and **fork n/a (rejected)**. Docker-workload
numbers for kfd are in the [PERFORMANCE.md](PERFORMANCE.md) Docker benchmark section.
‡ k7-fc VMM pause/resume returns immediately; CRI exec after resume hung on this
run (`guest_cid=0` retained — CHALLENGES #17). k7d resume→exec is **0.3s**.
<h3 align="left">
Also available today
</h3>
- 🛠️ Docker <code>build</code> / <code>run</code> inside VM sandboxes (docker sidecar on <b>kfd</b>, <b>kql</b>, and <b>k7d</b>; see [PERFORMANCE.md](PERFORMANCE.md))
- 🛠️ Docker <code>build</code> / <code>run</code> inside VM sandboxes: <code>k7 create --docker --backend k7d ubuntu:24.04</code> (or <code>--backend k7d-fc</code>). In-guest dockerd, overlay2, forkable. The same <code>--docker</code> flag on Kata (kfd/kql) injects a privileged docker-vehicle with overlay2 on a block disk — kql persists/forks the graph, kfd is ephemeral. <code>--sidecar docker</code> is a deprecated alias. See [PERFORMANCE.md](PERFORMANCE.md)
- ⚡ <b>Warm VM fork</b> on the k7d backend: <code>k7 fork</code> CoW-clones a running sandbox's disk <i>and memory</i> in ~5&nbsp;ms at the VMM; end-to-end through k7/Kubernetes is ~2&nbsp;s to a Ready pod
- 🌐 Multi-node clusters (Ansible + Longhorn)
- 🔍 Cilium CNI with FQDN egress policies
- 🔍 Cilium CNI with FQDN egress policies (optional Hubble flow observability via <code>k7 install --hubble</code>; off by default, observability only)
- 📸 Pause / resume / fork / restore and <code>k7 snapshot</code> lifecycle
- 🐍 Python SDK: <code>pip install k7-sdk</code> (<code>katakate</code> package deprecated)
@@ -190,7 +193,7 @@ sudo apt install k7
Then let `k7` get your node ready with everything:
```console
$ k7 install --backend kfd,kql,k7d --k7d-version 0.2.1
$ k7 install --backend kfd,kql,k7d
Current task: Reminder about logging out and back in for group changes
Installing K7 on 1 host(s)... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:01:41
✅ Installation completed successfully!
@@ -206,18 +209,48 @@ Optionally pass `-v` for a verbose output.
> [tutorials/k7_hetzner_node_setup.md](tutorials/k7_hetzner_node_setup.md)
> (this file is also in public [Katakate/k7](https://github.com/Katakate/k7)).
>
> `k7` 0.2.1 from the PPA still defaults the k7d artifact to 0.1.0. Pass
> `--k7d-version 0.2.1` so install pulls the current
> [Katakate/k7d](https://github.com/Katakate/k7d/releases/tag/v0.2.1) release.
> Multi-node inventory shapes (2-node server+agent, 3-node `--ha`) are in
> The playbook pins k7d **0.6.0**. `--docker` needs the guest docker
> service (payload on the node); an older k7d fails loudly with
> `this k7d has no docker service; upgrade`. Multi-node inventory shapes
> (2-node server+agent, 3-node `--ha`) are in
> `src/k7/deploy/inventory.ini.example`.
`k7 install` serves `k7-api` on NodePort `31007` over HTTPS. The default
is a playbook-minted cluster CA (Let's Encrypt cannot issue for a bare
IP). Copy `/etc/k7/tls/ca.crt` off the node and point the CLI at it:
```bash
scp root@<node>:/etc/k7/tls/ca.crt ./k7-ca.crt
k7 config set api.url https://<node-ip>:31007
k7 config set api.ca ./k7-ca.crt
k7 config set api.key <key>
```
`--api-hostname <name>` uses Let's Encrypt via a Caddy sidecar (the DNS
A record must point at the first master). `--api-tls-cert` +
`--api-tls-key` installs an operator-supplied pair. `--api-insecure-http`
is today's plain HTTP NodePort and must be called what it is: keys travel
in cleartext. None of this is rate limiting; do not write "the API is
now secure".
Pass `--api-allow-cidr <cidr>` (repeatable, Cilium only) to restrict who
can connect. Off by default. Defence in depth for operators who know
their client CIDRs — it stacks with TLS and does not replace it. See
[docs/BACKENDS.md](docs/BACKENDS.md) "TLS for `k7-api`" and
"Restricting who can reach `k7-api`".
Pass `--hubble` to turn on Cilium Hubble (relay + CLI, no UI) so policy
drops are a `hubble observe` question. Off by default; requires the
Cilium CNI (`--hubble --cni flannel` fails loudly). See
[docs/BACKENDS.md](docs/BACKENDS.md) "Debugging policy drops".
This will install and most importantly connect together the following components (depending on `--backend`):
- Kubernetes (K3s prod-ready distribution)
- Kata (for container virtualization)
- Firecracker + Jailer + devmapper thin-pool (`kfd`)
- QEMU via Kata + Longhorn PVC-backed roots (`kql`)
- k7d daemon + `containerd-shim-k7-v1` + RuntimeClass `k7` (`k7d`)
- Optional: Hubble relay + `hubble` CLI when `--hubble` is passed
Careful design: config updates will not touch your existing Docker or containerd setups. We chose to use K3s' own containerd for minimal disruption. Installation may however overwrite existing installations of K3s, Kata, Firecracker, Jailer, QEMU/Kata config, or Longhorn.
@@ -465,7 +498,10 @@ K7 sandboxes are hardened by default with multiple layers of security:
- **Linux capabilities**: All capabilities are dropped by default (`drop: ALL`) for defense-in-depth
- Only explicitly add back capabilities you need via `cap_add` parameter
- `allow_privilege_escalation` is always set to `false`
- Seccomp profile: `RuntimeDefault`
- Seccomp profile: `RuntimeDefault` is applied on the sandbox container
and enforced inside the guest: on k7d, and on Kata (`kfd` / `kql`)
with `disable_guest_seccomp = false` in the playbook's Kata config
(the sandbox container's OCI seccomp reaches guest runc).
- **Non-root execution**: Optionally run containers and pods as non-root user (UID 65532):
- `container_non_root`: Run the main container as non-root and disable privilege escalation
@@ -477,9 +513,10 @@ K7 sandboxes are hardened by default with multiple layers of security:
- File-based storage with 600 permissions (`/etc/k7/api_keys.json` by default)
- **Network policies**: Complete network isolation for VM sandboxes
- **Ingress isolation**: All inter-VM communication is blocked by default to prevent sandbox-to-sandbox access
- **Ingress isolation**: All inter-VM communication is blocked by default to prevent sandbox-to-sandbox access; opt in per sandbox with `--ingress-port` / `--ingress-from` (`sandbox:<name>`, `namespace:<ns>`, `cidr:<cidr>`)
- **Egress lockdown**: per-sandbox allowlists — CIDRs via Kubernetes NetworkPolicy, or **FQDN / domain** allowlists via Cilium (`CiliumNetworkPolicy`; default CNI)
- **DNS is blocked** when egress is locked down; only entries in `egress_whitelist` (CIDR or domain) are reachable
- **Platform isolation** (Cilium only): a cluster-wide deny policy stops sandboxes in every egress mode — including `--egress-open` — from reaching the node, other nodes, the Kubernetes API, cloud metadata and the `kube-system`/`longhorn-system` pods (CoreDNS excepted). Not applied on `--cni flannel` clusters.
- Administrative access via `kubectl exec` and `k7 shell` is preserved (uses Kubernetes API, not pod networking)
More security features are on the roadmap (e.g. AppArmor).
@@ -492,4 +529,4 @@ More security features are on the roadmap (e.g. AppArmor).
- Root `setup.py` publishes the SDK; assets under `src/k7/` belong to the Debian CLI / API image, not the PyPI wheel.
- User docs: `~/docs/k7/` (Mintlify). See `docs/README.md` in this repo.
- The CLI Debian package is built via `src/k7/cli/build.sh` and produces `dist/k7_<version>_amd64.deb` and `dist/k7_<version>_arm64.deb`.
- CI (tags `v*`) can publish the PyPI SDK and upload the `.deb` artifact.
- CI (tags `v*`) can publish the PyPI SDK and upload the `.deb` artifact.
+13 -7
View File
@@ -6,9 +6,9 @@ Where **K7** is headed — for contributors and operators.
## Current focus
Release engineering: apt/PPA, GHCR `k7-api`, and PyPI `k7-sdk`, after the
sibling [`Katakate/k7d`](https://github.com/Katakate/k7d) v0.1.0 artifact
exists (default install URL depends on it).
Release engineering: keep the apt/PPA, GHCR `k7-api`, and PyPI `k7-sdk`
pipelines current with `main` (the PPA is still 0.2.2; HTTPS, `--docker`,
and `k7d-fc` are unreleased).
---
@@ -20,16 +20,22 @@ exists (default install URL depends on it).
- [x] Snapshot lifecycle + GC; restore from VolumeSnapshot
- [x] CLI → API by default; `k7 api` / `k7 dev api rebuild`
- [x] Docker-in-VM sidecar + bench harness
- [x] First-class `--docker` (guest dockerd on k7d / k7d-fc; docker-vehicle on Kata)
- [x] Firecracker jailer
- [x] `k7d-fc` backend (k7d driving stock Firecracker + jailer)
- [x] Python SDK as **`k7-sdk`** / `k7_sdk` (`katakate` deprecated)
- [x] `k7d` backend install path (artifact URL / local override)
- [x] `k7d` backend install path (public `Katakate/k7d` GitHub Releases; playbook pin 0.6.0)
- [x] HTTPS-by-default for `k7-api` (Caddy sidecar, cluster CA)
- [x] Network security hardening: cluster-wide sandbox→platform
isolation, opt-in sandbox ingress / `--expose-port`, Hubble, and a
source-CIDR allowlist for the API NodePort. Tetragon was evaluated and
rejected
---
## Next goals
- [ ] PPA (`apt install k7`), GHCR `k7-api`, PyPI `k7-sdk`
- [ ] Default `k7d` install from public `Katakate/k7d` GitHub Releases
- [ ] PPA / GHCR / PyPI cut of the unreleased work (HTTPS, `--docker`, `k7d-fc`)
- [ ] Optional macOS CLI artifacts (tarball / Homebrew) — after the above
---
@@ -54,4 +60,4 @@ exists (default install URL depends on it).
---
*Last updated: August 2026*
*Last updated: September 2026*
+61 -3
View File
@@ -42,6 +42,10 @@ Do **not** open a public issue for security-sensitive reports.
- **`k7d`** — [Katakate/k7d](https://github.com/Katakate/k7d)
(`runtimeClassName: k7`); CoW sibling-fork isolation differs — see
k7d's `SECURITY.md`.
- **`k7d-fc`** (`runtimeClassName: k7-fc`) — same daemon and shim;
stock Firecracker under the stock jailer. What that buys (and does
not) is k7d's `SECURITY.md` **Firecracker profile** table. hostPath
volumes are refused before the pod is created.
- Sandbox containers run as non-root with restricted capabilities on top
of the VM boundary.
- The control plane API uses API keys with hashed storage and expiry
@@ -49,8 +53,31 @@ Do **not** open a public issue for security-sensitive reports.
that file). Keys may optionally be scoped to one or more namespaces
(`k7 generate-api-key -n <ns>`); absent/empty scope keeps the historical
unrestricted behaviour (backward compatible). Scoped keys are enforced
on every namespace-bearing endpoint — they cannot list across all
namespaces or touch namespaces outside their list.
on every namespace-bearing and cluster-scoped control-plane route —
they cannot list across all namespaces, touch namespaces outside their
list, or read cluster-wide topology (e.g. ``GET /api/v1/nodes/storage``).
- **The control-plane API is on a public NodePort (`31007`) on a public
node, over HTTPS by default**. A Caddy sidecar terminates TLS; the API
container and its probes stay HTTP on `:8000`. The default cert is a
playbook-minted cluster CA at `/etc/k7/tls/ca.crt` (Let's Encrypt cannot
issue for a bare IP). Clients trust it with `k7 config set api.ca`,
`--api-ca`, or `K7_API_CA`. `--api-hostname` (Let's Encrypt via Caddy)
and `--api-tls-cert`/`--api-tls-key` are the other tracks;
`--api-insecure-http` restores a plain HTTP NodePort and must be treated
as such (`verify=False` on an `https://` URL is an error). `k7 install
--api-allow-cidr <cidr>` (repeatable, Cilium only) restricts *who can
reach* it: a `CiliumNetworkPolicy` on the `k7-api` pod allows the
operator's CIDRs plus the reserved `host` / `remote-node` / `health` /
`kube-apiserver` peers (the kubelet probe path), and the Service switches
to `externalTrafficPolicy: Local` so the pod sees the real client IP.
This is **defence in depth for operators who know their client CIDRs, not
a secure API**: there is still no rate limiting. The flag is off by
default and affects nothing when unset. It is a pod-level policy — SSH
and the k3s API port are never touched, so a wrong CIDR costs API access
from your workstation and nothing else (`kubectl -n kube-system delete
ciliumnetworkpolicy k7-api-ingress` undoes it). A `fromCIDR` peer is not
evaluated for Cilium-managed sources, so the allowlist constrains
external clients only; in-cluster access is governed by the pod policies.
- Control-plane OCI registry inspection (used to resolve image
entrypoint/cmd) rejects registry hosts that are not on an allowlist
(default: `registry-1.docker.io`, `ghcr.io`, `quay.io`, `public.ecr.aws`;
@@ -58,10 +85,36 @@ Do **not** open a public issue for security-sensitive reports.
to loopback/private/link-local/metadata/reserved addresses. Redirect
following is disabled. The previous `localhost`→`http` downgrade path
has been removed.
- **Ingress** to sandboxes is denied by default (NetworkPolicy).
- **Ingress** to sandboxes is denied by default (NetworkPolicy) and opt-in
per sandbox: `--ingress-port` opens TCP ports, `--ingress-from`
(`sandbox:` / `namespace:` / `cidr:`) scopes who may connect. With no
source given, an opened port is reachable only from sandboxes in the
same namespace — never from the internet. A `cidr:` source does **not**
scope in-cluster peers on Cilium: an `ipBlock` peer is not evaluated for
pod-to-pod traffic, so such a rule leaves the opened port reachable from
any sandbox in the cluster. `k7 create` warns when one is used; scope
in-cluster access with `sandbox:` / `namespace:` instead. See
`docs/BACKENDS.md` for the measured behaviour. `--expose-port`
additionally publishes a port outside the cluster (NodePort with
`externalTrafficPolicy: Local`, so the pod sees the real client IP); it
requires a matching `--ingress-port`, and on a public node it puts the
sandbox on a public IP. A `cidr:` allowlist there does not constrain
clients on the cluster's own nodes.
**Egress** is per-sandbox: open, blocked, CIDR allowlist, or **FQDN**
allowlist when Cilium is the CNI (default). DNS is blocked by default
when egress is locked down.
- **Sandboxes never reach the platform.** With Cilium (default) a
cluster-wide deny policy (`k7-sandbox-platform-deny`) stops every
sandbox — in **every** egress mode, including `--egress-open` — from
dialing the node it runs on, the other nodes, the Kubernetes API
(node IP or `10.43.0.1`), cloud metadata (`169.254.0.0/16`), and the
pods of `kube-system` / `longhorn-system` (`k7-api`, Longhorn, ...),
CoreDNS excepted. "Open" egress means *the internet*, not *the
cluster*. This is deny-only and composes with the per-sandbox egress
policies; it does not defend against a VM escape. **Caveat:**
`k7 install --cni flannel` clusters do **not** get this isolation
(no Cilium policy engine) — an `--egress-open` sandbox there can reach
the node and the API server.
- **Multi-node** clusters are supported (Ansible inventory; Longhorn for
the QEMU/`kql` path). Cilium FQDN egress applies cluster-wide.
@@ -69,6 +122,11 @@ See also the docs: security model, networking, and backends comparison.
### Known limitations (pre-1.0)
- Default `k7 install` serves `k7-api` over HTTPS with a playbook-minted
cluster CA, so API keys no longer travel in cleartext. There is still
no rate limiting. `--api-insecure-http` is the escape hatch back to a
plain HTTP NodePort and must be treated as such. `--api-allow-cidr`
is defence in depth on who can connect; it is not a secure-API claim.
- No rate limiting or abuse protection at the API layer yet.
- API key storage is local file-backed; treat the API host as trusted.
Namespace scoping is an opt-in tenancy boundary on top of that model —
+5 -3
View File
@@ -1,4 +1,4 @@
# Docker workload benchmark — Spec 10b
# Docker workload benchmark
Quantifies the **storage tax** of running Docker inside a k7 sandbox vs
natively on the host. Output lands in [`PERFORMANCE.md`](../../PERFORMANCE.md)
@@ -10,9 +10,11 @@ cited verbatim from any blog post.
| Label | What it is |
|-------------|------------|
| `host` | Native Docker on the Hetzner node (no k7 involved) |
| `k7-fd` | k7 sandbox, `kata-firecracker-devmapper` (kfd) backend, `--sidecar docker` (docker daemon's `/var/lib/docker` is an emptyDir) |
| `k7-ql-r1` | k7 sandbox, `kata-qemu-longhorn` (kql) backend, `--sidecar docker`, Longhorn `replicas=1` |
| `k7-fd` | k7 sandbox, `kata-firecracker-devmapper` (kfd) backend, `--docker` vehicle (overlay2 on ephemeral LVM) |
| `k7-ql-r1` | k7 sandbox, `kata-qemu-longhorn` (kql) backend, `--docker` vehicle, Longhorn `replicas=1` |
| `k7-ql-r2` | Same as `k7-ql-r1` but Longhorn `replicas=2` (requires ≥ 2-node cluster) |
| `k7d` | k7 sandbox, RuntimeClass `k7`, first-class `--docker` (guest dockerd, overlay2 on virtio-blk, forkable) |
| `k7d-fc` | Same guest docker service as `k7d`, RuntimeClass `k7-fc` (Firecracker + jailer) |
The four environments stack the storage path cleanly: `host` → no VM,
no Longhorn. `k7-fd` → VM but no Longhorn. `k7-ql-r1` → VM + one local
+1 -1
View File
@@ -1,4 +1,4 @@
# Spec 10b workload: small but realistic. apt + pip generate thousands of
# Benchmark workload: small but realistic. apt + pip generate thousands of
# small files (metadata pressure), git clone is inode/dir-creation heavy,
# ``dd ... conv=fsync`` measures large-block sync write throughput. Each
# layer is bounded so the whole no-cache build completes in a few minutes
+11 -4
View File
@@ -1,4 +1,4 @@
"""CSV → Markdown renderer for the Spec 10b Docker benchmark.
"""CSV → Markdown renderer for the Docker benchmark.
Input CSV (produced by ``run_all.sh`` from ``/tmp/bench-<label>-*.log``):
@@ -16,7 +16,7 @@ Output: two Markdown tables.
``run_io``, ``run_cpu``), each sandbox env's median divided by host
median. The number that lands on Hacker News.
The renderer gates publishability on the spec-10b rule "(max - min) /
The renderer gates publishability on the rule "(max - min) /
median > 0.30 → investigate". Failing cells are tagged with ``⚠``.
Deterministic: ``--input`` and stable column ordering mean the same CSV
@@ -35,7 +35,7 @@ from pathlib import Path
from statistics import median
# Stable column ordering for the rendered tables.
ENVS: list[str] = ["host", "k7-fd", "k7-ql-r1", "k7-ql-r2", "k7-ql-r3", "k7d"]
ENVS: list[str] = ["host", "k7-fd", "k7-ql-r1", "k7-ql-r2", "k7-ql-r3", "k7d", "k7d-fc"]
# Stable row ordering, plus the human-readable column label for each op.
OP_DISPLAY: list[tuple[str, str]] = [
@@ -45,6 +45,8 @@ OP_DISPLAY: list[tuple[str, str]] = [
("run_cpu", "run cpu (10s budget)"),
("run_io", "run io (2k small + 512 MB)"),
("run_read", "run read (venv tree cat)"),
("fork_warm_engine", "fork warm engine (API)"),
("fork_to_ready", "fork → Ready + overlay2"),
]
# Operations included in the ratio-vs-host table — the ones that genuinely
@@ -57,7 +59,7 @@ RATIO_OPS: list[tuple[str, str]] = [
("run_cpu", "run cpu ratio"),
]
RANGE_GATE = 0.30 # spec 10b: (max - min) / median > 0.30 → ⚠
RANGE_GATE = 0.30 # (max - min) / median > 0.30 → ⚠
def _load_rows(csv_path: Path) -> list[dict[str, str]]:
@@ -157,7 +159,12 @@ def render(
sep = "|".join("-" * (len(e) + 2) for e in envs)
lines.append(f"| Operation |{header_envs}|")
lines.append(f"|---------------------------|{sep}|")
ops_with_data = {op for _, op in agg}
for op_key, op_display in OP_DISPLAY:
# k7d-family-only ops: omit the rows unless at least one env measured them
# so host/kfd/kql-only CSVs keep the historical six-row layout.
if op_key in ("fork_warm_engine", "fork_to_ready") and op_key not in ops_with_data:
continue
cells = "|".join(f" {_fmt_cell(agg.get((env, op_key)))} " for env in envs)
lines.append(f"| {op_display:<26}|{cells}|")
lines.append("")
+1 -1
View File
@@ -1,4 +1,4 @@
# Spec 10b: keep the directory but ignore generated artifacts. Logs / CSVs /
# Keep the directory but ignore generated artifacts. Logs / CSVs /
# rendered sections live here; the published numbers go in PERFORMANCE.md.
*.log
*.csv
+6
View File
@@ -1,3 +1,9 @@
k7 (0.3.0) noble; urgency=medium
* Release 0.3.0.
-- Katakate <hi@katakate.org> Sat, 12 Sep 2026 21:58:51 +0200
k7 (0.2.2) noble; urgency=medium
* Release 0.2.2.
+327 -60
View File
@@ -1,27 +1,44 @@
# Sandbox backends: kfd, kql, k7d
# Sandbox backends: kfd, kql, k7d, k7d-fc
k7 turns every sandbox into a Kubernetes Deployment whose pod runs inside a
hardware-isolated microVM. *How* that VM is built, stored, snapshotted, and
forked is the backend's job. Three backends exist today; a node can install
any combination (`k7 install --backend kfd,kql,k7d`) and each sandbox picks
forked is the backend's job. Four backends exist today; a node can install
any combination (`k7 install --backend kfd,kql,k7d,k7d-fc`) and each sandbox picks
one (`k7 create --backend …`, annotation `k7.katakate.org/backend`).
| | `kfd` — kata-firecracker-devmapper | `kql` — kata-qemu-longhorn | `k7d` |
|---|---|---|---|
| VMM | Firecracker via Kata Containers | QEMU via Kata Containers | [k7d](https://github.com/katakate/k7d), Katakate's purpose-built KVM VMM |
| containerd RuntimeClass | `kata` | `kata-qemu` | `k7` |
| Rootfs / storage | devmapper LVM thin-pool (needs a spare raw disk) | overlayfs + Longhorn PVC mounted at `/mnt/state` | cached erofs images on virtio-blk, guest tmpfs upper, reflink-XFS volume images |
| Named snapshots (`k7 snapshot`) | ✗ | ✅ Longhorn VolumeSnapshot (disk-only, crash-consistent) | ✗ by design — see below |
| `k7 fork` | ✗ | disk clone: snapshot → new PVC → cold boot | **warm fork: CoW disk *and* memory of the live VM** |
| `k7 pause` / `resume` | scale to 0/1 (state lost) | scale to 0/1, disk persists on the PVC | **VM frozen in place: vCPUs stop, memory stays** |
| `k7 restore` | ✗ | ✅ boot a new sandbox from any named snapshot | ✗ (fork the live sandbox instead) |
| Docker-in-VM sidecar | ✅ (emptyDir data) | ✅ (PVC-persistent data) | ✅ (VM-lifetime data) |
| | `kfd` — kata-firecracker-devmapper | `kql` — kata-qemu-longhorn | `k7d` | `k7d-fc` |
|---|---|---|---|---|
| VMM | Firecracker via Kata Containers | QEMU via Kata Containers | [k7d](https://github.com/katakate/k7d) in-process rust-vmm | Same k7d daemon; stock Firecracker under the stock jailer |
| containerd RuntimeClass | `kata` | `kata-qemu` | `k7` | `k7-fc` |
| Rootfs / storage | devmapper LVM thin-pool (needs a spare raw disk) | overlayfs + Longhorn PVC mounted at `/mnt/state` | cached erofs images on virtio-blk, guest tmpfs upper, reflink-XFS volume images | same as k7d (erofs/scratch virtio-blk; **no virtiofs / hostPath**) |
| Named snapshots (`k7 snapshot`) | ✗ | ✅ Longhorn VolumeSnapshot (disk-only, crash-consistent) | ✗ by design — see below | ✗ (same as k7d) |
| `k7 fork` | ✗ | disk clone: snapshot → new PVC → cold boot | **warm fork: CoW disk *and* memory of the live VM** | same daemon socket / `fork-source-*` annotations |
| `k7 pause` / `resume` | scale to 0/1 (state lost) | scale to 0/1, disk persists on the PVC | **VM frozen in place: vCPUs stop, memory stays** | same `/run/k7d/k7d.sock` path |
| `k7 restore` | ✗ | ✅ boot a new sandbox from any named snapshot | ✗ (fork the live sandbox instead) | ✗ |
| Docker | **`--docker`**: privileged `docker-vehicle`, overlay2 on ephemeral `k7-docker-lvm` block LV | **`--docker`**: privileged `docker-vehicle`, overlay2 on Longhorn Block PVC (pause/fork/restore) | **`--docker`**: agent-supervised dockerd | **`--docker`** (same guest service) |
| hostPath | Kata virtiofs | Kata virtiofs | k7d virtiofs fallback | **refused up front** (Firecracker has no virtiofs) |
Measured numbers for all of this live in [PERFORMANCE.md](../PERFORMANCE.md):
on the same node, k7d creates in ~2.1s vs kql's ~17.1s, forks to a usable
sandbox in ~2.4s vs ~46.7s (and the k7d fork inherits the source's memory),
and pauses/resumes in ~0.2s/~0.3s vs ~1.3s/~4.1s — while kql keeps named
persistent snapshots and the faster in-sidecar `docker pull`.
persistent snapshots and overlay2 Docker on a Longhorn block volume.
`k7 create --docker` is one user contract on every backend (`docker` /
`compose` / `buildx` on PATH, `overlay2` on a **block** disk, socket shared
as a directory). k7d/k7d-fc run dockerd as a guest agent service (no extra
CRI container). Kata (kfd/kql) injects one privileged **docker-vehicle**
container in the same VM; the sandbox container's security context is
unchanged. The graph is never virtio-fs / emptyDir. kql's graph is a
second Longhorn Block PVC (`<name>-docker-lh`) included in pause / fork /
restore (two VolumeSnapshots are crash-consistent per volume, not
cross-volume atomic). kfd's graph is a generic ephemeral volume from
StorageClass `k7-docker-lvm` (OpenEBS LVM LocalPV over `kata-vg/thin-pool`)
and is deleted with the pod; `k7 fork` of a kfd `--docker` sandbox is
rejected. `--sidecar docker` is a deprecated alias of `--docker`. Inside
one Kata VM the vehicle and the sandbox are a single trust domain; the
isolation boundary is the VM. No hostPath is used for the payload or the
CLI.
## Which backend when?
@@ -36,6 +53,12 @@ persistent snapshots and the faster in-sidecar `docker pull`.
forked in seconds, and pause/resume is instantaneous because nothing is
torn down. State does not survive pod deletion — persistence is "fork it
again", not "write it to a disk".
- **k7d-fc** (`k7-fc`) — pick this when you want k7d's fork/pause daemon
**and** Firecracker's jailer (separate VMM process, per-VM uid/chroot/seccomp).
You lose virtiofs/hostPath and time-warp. See k7d
`SECURITY.md` "Firecracker profile" and `docs/backends.md`. Kata's
Firecracker (`kfd`) is a different binary (`/opt/kata/bin`, older pin);
k7d-fc installs upstream v1.16.1 at `/usr/local/bin`.
## The k7d backend
@@ -51,16 +74,17 @@ The Ansible playbook:
2. installs `erofs-utils`, `xfsprogs`, `virtiofsd`, and provisions a sparse
reflink-capable XFS image loop-mounted at `/var/lib/k7d/disks` (warm forks
clone writable volume images with `FICLONE` reflinks);
3. downloads the k7d release tarball
(`k7d_artifact_url`, default the public `Katakate/k7d` GitHub release
for `k7d_version`, currently **0.2.1**) and runs the bundled
`install.sh`, which installs `k7d` + `containerd-shim-k7-v1` into
`/usr/local/bin`, guest kernel/initramfs into `/usr/local/share/k7d`,
and starts `k7d.service` (control socket `/run/k7d/k7d.sock`).
Override with `k7 install --k7d-version <ver>` (same GitHub URL, other
tag) or `--k7d-artifact <path>` / `k7d_artifact_local_path` (local
tarball). The PPA `k7` 0.2.1 playbook still defaults to k7d 0.1.0 —
pass `--k7d-version 0.2.1` on that package until the next k7 cut.
3. downloads the k7d release tarball (`k7d_artifact_url`, default the
public `Katakate/k7d` GitHub release for `k7d_version`, currently
**0.6.0**) and runs the bundled `install.sh`, which installs `k7d` +
`containerd-shim-k7-v1` into `/usr/local/bin`, guest kernel/initramfs
into `/usr/local/share/k7d`, and starts `k7d.service` (control socket
`/run/k7d/k7d.sock`). Override with `k7 install --k7d-version <ver>`
(same GitHub URL, other tag) or `--k7d-artifact <path>` /
`k7d_artifact_local_path` (local tarball). `--docker` needs a k7d that
ships the guest docker service (payload at
`/usr/local/share/k7d/docker` or `/root/k7d/guest/docker/payload`); an
older k7d fails loudly.
4. registers the `k7` runtime in the k3s containerd template **with**
`pod_annotations = ["k7d.katakate.org/*"]` and **without** a `BinaryName`
option (containerd resolves the shim from
@@ -68,14 +92,34 @@ The Ansible playbook:
RuntimeClass `k7`;
5. labels the node `k7.katakate.org/backend-k7d=true`.
### What k7 installs (`k7 install --backend k7d-fc`)
Same daemon/shim as `k7d`, plus:
1. pinned Firecracker + jailer via `src/k7/deploy/k7d-fc/install-firecracker.sh`
(pins from `pins.env`; sha mismatch fails the playbook);
2. `/etc/k7d/shim-k7-fc.toml` (`backend = "firecracker"`) and containerd
`runtimes.k7-fc` with `ConfigPath` and **no** `BinaryName`;
3. RuntimeClass `k7-fc` with PodOverhead `32Mi` / `50m`;
4. node label `k7.katakate.org/backend-k7d-fc=true` (and `backend-k7d`
so native k7 pods still schedule).
Do not point `--k7d-artifact` at an older GitHub tarball after developing
the shim — 0.5.0 does not know ConfigPath. The playbook
renders the whole containerd template from the selected backends; on a
node that already has other runtimes (kata, nvidia, wasm, …) use
`--tags k7d-fc` (Firecracker install + RuntimeClass + labels) rather
than a full `k7 install` that would drop those blocks.
### How a sandbox maps to a VM
One pod = one k7d microVM. The pod's containers (sandbox + optional docker
sidecar) all run as runc containers *inside* that single VM — multi-container
pods share the guest kernel, network namespace, and tmpfs. VM size follows the
pod's CPU/memory limits (the shim reads the CRI sandbox annotations, so
`k7 create --cpu 2 --memory 1Gi` gives a 2-vCPU/1 GiB VM with a host-side CFS
cap).
One pod = one k7d microVM. The pod's sandbox container runs as a runc
container *inside* that VM. ``k7 create --docker`` does **not** add a
second CRI container: the guest agent (PID 1) supervises a pinned
`dockerd` with its graph on a per-sandbox virtio-blk scratch disk.
Inner containers share the guest kernel, network namespace, and the
exported sandbox paths (`/home`, `/root`, `/tmp`, `/opt`, `/workspace`).
VM size follows the pod's CPU/memory limits.
### The fork story
@@ -99,8 +143,10 @@ target the inherited state.
Properties and limits (fail loudly, never silently):
- Warm fork works for **single-workload sandboxes**; forking a sandbox with a
sidecar is rejected (no reliable container mapping — use kql for that).
- Warm fork works for **single-workload sandboxes**, including
`--docker` (dockerd is not a CRI sidecar). Forking a sandbox with a
**real CRI sidecar** is rejected (no reliable container mapping — use
kql for that).
- The fork is pinned to the source's node (the k7d daemon is node-local;
cross-node fork is k7d's M12 roadmap item).
- If the fork pod is ever restarted by Kubernetes it re-forks from the (then
@@ -117,8 +163,8 @@ Properties and limits (fail loudly, never silently):
k7d pause/resume/fork need three things that only exist **on the node
hosting the sandbox**: the k7d daemon socket (`/run/k7d/k7d.sock`), the k3s
containerd socket, and `crictl`. Since spec 18g this is handled by the
`k7-agent` DaemonSet (kube-system, same `k7-api:local` image running
containerd socket, and `crictl`. This is handled by the `k7-agent`
DaemonSet (kube-system, same `k7-api:local` image running
`k7.api.agent:app`), so VM ops work **through the API for a sandbox on any
node**:
@@ -135,7 +181,7 @@ node**:
local daemon, remote ones are forwarded the same way (root can read the
token).
- A fork still **lands on the source's node** — the agent proxies to the
node-local daemon; the cross-node fork *data path* is k7d spec 9a M12
node-local daemon; the cross-node fork *data path* is future k7d work
(daemon side, not built yet). When it lands, only the forwarding target
changes.
@@ -154,13 +200,20 @@ disk survives) and resume pays a full VM boot.
## Egress modes
A sandbox has exactly one of three egress modes (spec 18f issue 4):
A sandbox has exactly one of three egress modes:
| Mode | CLI | YAML / API | Result |
|---|---|---|---|
| **block-all** (CLI default) | `k7 create …` (no egress flag) | `egress_whitelist: []` | deny-all egress NetworkPolicy |
| **whitelist** | `--egress <entry>` (repeatable) | `egress_whitelist: [<entries>]` | only listed CIDRs/domains |
| **open** | `--egress-open` | omit `egress_whitelist` (or `null`) | no egress policy at all |
| **open** | `--egress-open` | omit `egress_whitelist` (or `null`) | no per-sandbox egress policy; internet reachable, platform is not (see below) |
In every mode, on a Cilium cluster, the cluster-wide
`k7-sandbox-platform-deny` policy additionally denies sandbox egress to
the node, the other nodes, the Kubernetes API server, link-local / cloud
metadata, and the `kube-system` + `longhorn-system` pods except CoreDNS.
It is deny-only, so it never narrows what a whitelist allows beyond those
targets. `--cni flannel` clusters do **not** get this protection.
`--egress-open` and `--egress` are mutually exclusive. Note the asymmetric
defaults, kept for backward compatibility: the CLI without flags is
@@ -172,15 +225,226 @@ depth** (`registry.docker.com` *and* `production.cloudfront.docker.com`) but
not the apex `docker.com` itself — add it as its own entry. Under the hood
k7 translates a leading `*.` into Cilium's multi-label `**.` matchPattern; a
bare Cilium `*` never crosses label boundaries, which used to silently break
CDN-backed registries (spec 18f issue 3). The install also sets Cilium
`dnsProxy.minTtl=3600` so clients that cache a resolved IP longer than the
CDN's 30–60s DNS TTL (dockerd's blob downloader does) keep their learned
FQDN→IP allowance for an hour.
CDN-backed registries. The install also sets Cilium `dnsProxy.minTtl=3600`
so clients that cache a resolved IP longer than the CDN's 30–60s DNS TTL
(dockerd's blob downloader does) keep their learned FQDN→IP allowance for an
hour.
## Ingress
Ingress is denied by default and opt-in per sandbox. A sandbox with no
ingress flags gets the same deny-all `{name}-deny-ingress` NetworkPolicy
k7 has always created; `--ingress-port` adds allow rules to that same
object.
| Setting | CLI | YAML / API | Result |
|---|---|---|---|
| **deny-all** (default) | `k7 create …` (no ingress flag) | omit `ingress_ports` | deny-all ingress NetworkPolicy |
| **open TCP ports** | `--ingress-port <port>` (repeatable) | `ingress_ports: [<ports>]` | listed TCP ports reachable from sandboxes in the same namespace |
| **scope the sources** | `--ingress-from <source>` (repeatable) | `ingress_from: [<sources>]` | only the listed sources reach those ports |
Sources:
| Entry | Matches |
|---|---|
| `sandbox:<name>` | that sandbox, same namespace (`podSelector` on `katakate.org/sandbox`) |
| `namespace:<ns>` | every pod in that namespace (`namespaceSelector` on `kubernetes.io/metadata.name`) |
| `cidr:<cidr>` | clients in that CIDR (`ipBlock`) |
Anything else is a hard error at create time — an unreadable source in a
security rule is never guessed at. `--ingress-from` without
`--ingress-port` is an error too: sources alone open nothing.
`--ingress-port` with no `--ingress-from` means "other sandboxes in this
namespace", never the world. Reaching a sandbox from anywhere requires
`--ingress-from cidr:0.0.0.0/0`, typed out in full.
These are stock v1 `NetworkPolicy` rules, which Cilium enforces natively,
so ingress works on `--cni flannel` as well.
**`cidr:` does not scope in-cluster peers on Cilium.** Cilium does not
evaluate an `ipBlock` peer for traffic where both ends are Cilium-managed
([Layer 3 policy docs][cilium-l3]), so a rule whose peers are all `cidr:`
degrades to a *port-only* allow inside the cluster: the port is reachable
from **any sandbox in the cluster**, even when the CIDR contains no cluster
IP. Measured on this cluster — a sandbox opened with
`--ingress-from cidr:198.51.100.0/24` answered an unrelated sandbox on the
opened port, while an unlisted port stayed closed. Adding a `sandbox:` peer
alongside the `cidr:` does not close it.
`k7 create` warns on stderr whenever `--ingress-from` contains a `cidr:`
entry. Use `cidr:` for **external** clients (that is what `--expose-port`
needs it for) and `sandbox:` / `namespace:` to scope in-cluster access —
those are evaluated by pod identity and were verified to deny precisely.
Two further reasons not to reach for `cidr:` in-cluster: pod IPs are
reassigned on every restart, so a rule written against one silently stops
matching; and the NetworkPolicy spec leaves `ipBlock`-vs-pod-traffic up to
the CNI, so such a rule is not portable between CNIs either.
[cilium-l3]: https://docs.cilium.io/en/stable/security/policy/layer3/
`kubectl exec` and `k7 shell` are unaffected either way: they go through
the Kubernetes API and the CRI, not the pod network.
A sandbox is a bare Deployment with no Service, so ingress rules only cover
in-cluster traffic. To reach a sandbox from outside the cluster, see
"Exposing a sandbox" below.
## Exposing a sandbox outside the cluster
`--expose-port <port>` (repeatable, `expose_ports` in YAML/API) creates a
`NodePort` Service named `{name}-expose` in the sandbox's namespace. `k7
create` prints the resolved `http://<node-ip>:<nodeport>` for each port, and
`k7 list` shows the allocated NodePorts.
- **Every `--expose-port` needs a matching `--ingress-port`.** k7 refuses
otherwise: a public NodePort in front of a deny-all policy is a confusing
no-op, and one in front of an accidentally-open policy is a breach.
- The Service uses `externalTrafficPolicy: Local`, so the client's real
source IP reaches the pod and `--ingress-from cidr:…` rules can mean
something. With the default `Cluster` the source would be SNAT'd to an
address on the pod network and any CIDR allowlist would be decoration.
- **A `cidr:` allowlist does not constrain clients on the cluster's own
nodes.** As in-cluster (above), Cilium does not evaluate an `ipBlock`
peer against a node IP without `policy-cidr-match-mode=nodes`, so a
request from a node answers even when the CIDR excludes it. Verify a
`cidr:` allowlist from a genuinely external client — `curl` from the node
passes no matter how the rule is written.
- Consequence of `Local`: the NodePort **only answers on the node running
the sandbox** (`k7 list` shows which one).
- On a bare-metal host (Hetzner) the node IP is public, so
`--expose-port` with `--ingress-from cidr:0.0.0.0/0` publishes an
untrusted sandbox to the internet. k7 warns on stderr on every such
create.
- `k7 delete` removes the Service; a leaked NodePort is both a resource leak
and an open port.
- A `k7 fork` inherits the source's ingress and egress **policy** but not its
Service — a fork is not reachable from outside until you expose it.
```bash
k7 create web python:3.12-slim \
--ingress-port 8000 --ingress-from cidr:203.0.113.0/24 --expose-port 8000
# 🌐 Port 8000 exposed at http://<node-ip>:31234
```
## TLS for `k7-api`
Default `k7 install` serves the NodePort on `31007` over HTTPS. The
`k7-api` container stays HTTP on `:8000` (kubelet probes unchanged); a
Caddy sidecar in the same pod terminates TLS on `:8443`. API keys no
longer travel in cleartext. This is not rate limiting and not a claim
that the API is safe to expose.
| Flag | Cert |
|---|---|
| (none) | Playbook cluster CA at `/etc/k7/tls/ca.crt`. Laptop: copy that file and `k7 config set api.ca ./ca.crt` plus `api.url` to `https://<first-master-ip>:31007`. |
| `--api-hostname NAME` | Let's Encrypt via Caddy (HTTP-01 on host port 80). `NAME` must be a DNS name whose A record is the first-master IP; a bare IP is refused. The CLI uses the system trust store (no `api.ca`). |
| `--api-tls-cert` + `--api-tls-key` | Operator-supplied PEM pair. Incompatible with `--api-hostname`. |
| `--api-insecure-http` | Today's HTTP NodePort, bit for bit. Keys travel in cleartext. |
`--api-insecure-http` cannot be combined with the hostname or cert
flags. It can be combined with `--api-allow-cidr` (the allowlist still
applies; the warning is that keys are still cleartext).
## Restricting who can reach `k7-api`
`k7-api` is a `NodePort` Service on `31007`, and on a dedicated box the
node IP is public. `k7 install --api-allow-cidr 203.0.113.4/32
--api-allow-cidr 198.51.100.0/24` (repeatable, Cilium only — with
`--cni flannel` it fails at the CLI) renders a `CiliumNetworkPolicy`
`k7-api-ingress` on the API pod and flips the Service to
`externalTrafficPolicy: Local`. Both are skipped entirely when the flag
is unset, which is the default.
This is **defence in depth for operators who know their client CIDRs**.
It stacks with TLS and does not replace it. There is still no rate
limiting. See `SECURITY.md`.
Measured behaviour worth knowing before you debug it:
- The policy's `ingress:` section makes the API pod default-deny for
ingress. The `fromEntities` block (`host`, `remote-node`, `health`,
`kube-apiserver`) is what keeps the kubelet liveness/readiness probes
alive — probes arrive from the node's `cilium_host` address and match
`host`. Remove it and the pod enters `CrashLoopBackOff`.
- `externalTrafficPolicy: Local` is what preserves the client's source
IP; without it every external caller would arrive SNAT'd as a node IP.
The trade-off: the NodePort then answers **only** on the node running
the `k7-api` pod, which is the first master
(`nodeSelector: k7.katakate.org/first-master: "true"`) — exactly the
address the playbook writes into `/etc/k7/api_endpoint`. Operators
hitting other node IPs directly must switch to that one.
- **A curl from the node is not a test of the allowlist.** Node-local
traffic matches the `host` entity, and node-local NodePort traffic is
SNAT'd to the pod network even under `Local` (measured: the pod sees
the node's `cilium_host` address), so it succeeds however wrong the
CIDR list is. Test from off-cluster.
- Nor is a curl from **another Cilium node that runs its own `k7-api`**:
with `kubeProxyReplacement=true` its socket load-balancer translates a
host-originated connection to *any* address on that NodePort to its own
local backend. Measured: such a client got a healthy `200` while
`tcpdump` on the target node saw no packet at all. Use a client that is
not a Cilium node, or confirm with `tcpdump` that the packets arrive.
- A `fromCIDR` peer is not evaluated when the source is Cilium-managed,
so allowlisting a pod's own /32 does not admit that pod. In-cluster
callers are governed by `fromEndpoints` rules
(`agent-networkpolicy.yaml`), not by this allowlist.
- `hubble observe --verdict DROPPED` shows the drops (see below).
Rollback — re-running `k7 install` without the flag removes both pieces,
or by hand:
```bash
k3s kubectl -n kube-system delete ciliumnetworkpolicy k7-api-ingress
k3s kubectl -n kube-system patch svc k7-api \
-p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
```
A wrong CIDR only costs API access: the policy selects the `k7-api` pod
endpoint, never the host, so SSH, `kubectl` and
`src/k7/cli/dev.sh --core` on the node keep working. The Cilium host
firewall would not have that property, which is why it was rejected.
## Debugging policy drops (Hubble)
Hubble is **opt-in observability**, not enforcement. It is off by
default and is not a security control. Enable it with
`k7 install --hubble` (Cilium CNI only — `--hubble --cni flannel` fails
loudly). The Hubble UI is not installed: it has no authentication and
the node IP is public. Rollback: `cilium hubble disable`.
`hubble observe` talks to Hubble Relay. On the node (verified: Relay
is ClusterIP `:80`; this forwards it to `127.0.0.1:4245`, the Hubble
CLI default):
```bash
export KUBECONFIG=/etc/rancher/k3s/k3s.yaml
cilium hubble port-forward & # kube-system/hubble-relay → 127.0.0.1:4245
hubble status # Healthcheck (via localhost:4245): Ok
hubble observe --verdict DROPPED --last 50
```
When a sandbox with an FQDN whitelist cannot reach a host, the symptom
is always a hang or timeout — policy drop, DNS, MTU, and a dead
upstream look the same from inside the VM. The CHALLENGES.md #9 recipe
tells them apart:
```bash
hubble observe --namespace <ns> --verdict DROPPED --last 50
cilium fqdn cache list
cilium ip list
```
A `Policy denied DROPPED` verdict with the destination still labelled
`world` means the FQDN→identity mapping missed (wildcard too narrow, or
the learned IP expired). `cilium fqdn cache list` shows what Cilium
learned; `cilium ip list` shows which IPs carry an `fqdn:` identity.
## Disk pool sizing (kfd + k7d)
Both node-local storage pools have fixed sizes chosen at install time
(spec 18f issue 5) — set them per node in the inventory:
Both node-local storage pools have fixed sizes chosen at install
time — set them per node in the inventory:
| Pool | Backend | Default | Inventory knob | Utilization |
|---|---|---|---|---|
@@ -195,14 +459,17 @@ disk-pressure eviction, the pools are not part of the root filesystem):
sandboxes start failing with I/O errors and new kfd pods fail to create
their devmapper snapshots (`CreateContainerError`). Only the first
`kata_thinpool_pv_size` of the spare disk is used — size it generously.
Docker-heavy kfd nodes share this pool with `--docker` graph LVs
(`k7-docker-lvm`); raise `kata_thinpool_pv_size` rather than silently
resizing.
- **k7d pool full**: writable volume images and fork reflink clones fail;
the k7d daemon rejects new sandboxes/forks loudly (`no space left on
device`). The image is sparse, so `ls -l` shows the virtual size —
use `du`/`df` for actual usage.
Utilization for both pools on every node is surfaced through the API
(spec 18g) and CLI/SDK (spec 18h): `GET /api/v1/nodes/storage`,
`k7 nodes storage` (`--json` for raw), and `Client.nodes_storage()`
Utilization for both pools on every node is surfaced through the API and
CLI/SDK: `GET /api/v1/nodes/storage`, `k7 nodes storage` (`--json` for
raw), and `Client.nodes_storage()`
return a per-node map of `kata_thinpool` (`lvs` size/data%/metadata%)
and `k7d_disks` (`df` size/used/avail), collected from the k7-agent
DaemonSet. A node whose agent is unreachable gets a loud
@@ -215,24 +482,24 @@ like `2Gi` / `4096Mi`):
| Backend | Mechanism |
|---|---|
| **kfd** | Kata stamps `io.katacontainers.config.hypervisor.default_memory` (MiB). containerd forwards `io.katacontainers.*` (`pod_annotations` on `runtimes.kata`) and `configuration-fc.toml` allowlists `default_memory` (spec 18h — before that the annotation was silently ignored and the VM stayed at the 2048 MiB default). |
| **kql** | Same annotation path via `runtimes.kata-qemu` + `configuration-qemu.toml` (spec 18g). |
| **kfd** | Kata stamps `io.katacontainers.config.hypervisor.default_memory` (MiB). containerd forwards `io.katacontainers.*` (`pod_annotations` on `runtimes.kata`) and `configuration-fc.toml` allowlists `default_memory` (before that the annotation was silently ignored and the VM stayed at the 2048 MiB default). |
| **kql** | Same annotation path via `runtimes.kata-qemu` + `configuration-qemu.toml`. |
| **k7d** | No Kata annotation — the k7d shim sizes the VM straight from the pod's CRI CPU/memory limits. |
## Known issues
- **kata-fc VMM leak under churn** (spec 18f issue 1b): kata 3.24.0 with the
jailer records the `--daemonize`d jailer's PID, so its SIGTERM fallback at
pod deletion signals a dead PID. When the graceful in-guest shutdown fails
(dead agent under parallel churn), the firecracker process is orphaned and
spins at 100% CPU. `k7 install` deploys a per-node systemd timer
- **kata-fc VMM leak under churn**: kata 3.24.0 with the jailer records the
`--daemonize`d jailer's PID, so its SIGTERM fallback at pod deletion
signals a dead PID. When the graceful in-guest shutdown fails (dead agent
under parallel churn), the firecracker process is orphaned and spins at
100% CPU. `k7 install` deploys a per-node systemd timer
(`k7-vmm-reaper.timer`, 1-minute cadence) that kills VMM processes whose
kata shim is gone, and the integration suite asserts zero orphans cluster
wide after teardown (`tests/integration/test_zz_leaks.py`).
- **kql docker-in-VM wedge under heavy fsync + replicas ≥ 2 — FIXED**
(spec 18f issue 8 / spec 18g): kata's default single-threaded virtiofsd
serialized all virtio-fs IO; under a sustained fsync burst against an
r≥2 Longhorn volume the kata-agent's health ping starved behind the IO
convoy and the shim killed the healthy VM. `k7 install` now widens the
virtiofsd thread pool (`--thread-pool-size=16`); see PERFORMANCE.md for
the root-cause narrative and post-fix numbers.
- **kql docker-in-VM wedge under heavy fsync + replicas ≥ 2 — FIXED**:
kata's default single-threaded virtiofsd serialized all virtio-fs IO;
under a sustained fsync burst against an r≥2 Longhorn volume the
kata-agent's health ping starved behind the IO convoy and the shim
killed the healthy VM. `k7 install` now widens the virtiofsd thread
pool (`--thread-pool-size=16`); see PERFORMANCE.md for the root-cause
narrative and post-fix numbers.
+2 -2
View File
@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
[project]
name = "k7"
version = "0.2.2"
version = "0.3.0"
description = "Self-hosted VM sandboxes for untrusted and AI code (CLI, API, SDK)"
readme = "README.md"
requires-python = ">=3.10.11"
@@ -63,7 +63,7 @@ markers = [
"qemu: tests specific to kata-qemu-longhorn backend",
"k7d: tests specific to the k7d backend",
"multinode: tests requiring a multi-node k7 cluster (>=2 Ready nodes)",
"bench: spec-10b benchmark module; opt-in only (-m bench), not run by default",
"bench: Docker benchmark module; opt-in only (-m bench), not run by default",
]
addopts = [
"--strict-markers",
+1 -1
View File
@@ -4,7 +4,7 @@ from setuptools import find_packages, setup
setup(
name="k7-sdk",
version="0.2.2",
version="0.3.0",
description="K7 sandbox management Python SDK (HTTP client for the k7 API)",
long_description=Path(__file__).with_name("README.md").read_text(),
long_description_content_type="text/markdown",
+1 -1
View File
@@ -1,3 +1,3 @@
"""K7 Sandbox Management System"""
__version__ = "0.2.2"
__version__ = "0.3.0"
+2 -2
View File
@@ -11,7 +11,7 @@ RUN uv venv /app/.venv && \
# crictl (pinned): k7d VM ops (pause/resume/fork) resolve a pod to its CRI
# sandbox id via `crictl pods` against the node's containerd socket, which
# the k7-api deployment hostPath-mounts (spec 18f issue 2). The node's own
# the k7-api deployment hostPath-mounts. The node's own
# /usr/local/bin/crictl is a k3s symlink and can't be mounted usefully.
ARG CRICTL_VERSION=v1.31.1
ARG CRICTL_SHA256=0a03ba6b1e4c253d63627f8d210b2ea07675a8712587e697657b236d06d7d231
@@ -28,7 +28,7 @@ ENV PYTHONPATH="/app"
WORKDIR /app
# lvm2: the k7-agent DaemonSet (same image, `k7.api.agent:app`) reports the
# kfd thin-pool utilization via `lvs` (spec 18g) — needs the privileged
# kfd thin-pool utilization via `lvs` — needs the privileged
# agent container with /dev hostPath-mounted.
RUN apt-get update && apt-get install -y --no-install-recommends \
ca-certificates \
+2 -2
View File
@@ -1,4 +1,4 @@
"""Per-node k7 agent (spec 18g).
"""Per-node k7 agent.
Runs as a DaemonSet (``k7-agent``, kube-system) on every node, reusing the
``k7-api:local`` image with an overridden command
@@ -120,7 +120,7 @@ async def vm_lookup(body: dict | None = None):
# ---------------------------------------------------------------------------
# Node storage-pool utilization (spec 18g part 2 / 18f issue 5 leftover).
# Node storage-pool utilization.
# ---------------------------------------------------------------------------
+15 -7
View File
@@ -202,6 +202,9 @@ async def create_sandbox(config: dict, key_data: dict = Depends(verify_api_key))
"namespace": sandbox_config.namespace,
"image": sandbox_config.image,
}
# NodePort URLs for --expose-port, when there are any.
if result.data and result.data.get("endpoints"):
resource["endpoints"] = result.data["endpoints"]
location = f"/api/v1/sandboxes/{sandbox_config.name}?namespace={sandbox_config.namespace}"
return success_response(resource, status_code=status.HTTP_201_CREATED, headers={"Location": location})
else:
@@ -340,7 +343,7 @@ async def get_sandbox_logs(
since: int = 0,
key_data: dict = Depends(verify_api_key),
):
"""Read pod logs (snapshot; no streaming yet — see Spec 10g risks)."""
"""Read pod logs (snapshot; no streaming yet)."""
authorize_namespace(key_data, namespace)
core = K7Core()
result = await core.get_logs(
@@ -392,11 +395,16 @@ async def install_node(install_data: dict):
raise HTTPException(status_code=400, detail=result.error)
@app.get("/api/v1/nodes/storage", dependencies=[Depends(verify_api_key)])
async def get_nodes_storage():
@app.get("/api/v1/nodes/storage")
async def get_nodes_storage(key_data: dict = Depends(verify_api_key)):
"""Per-node storage-pool utilization (kfd thin-pool + k7d disks pool),
aggregated from the k7-agent DaemonSet (spec 18g). A node whose agent
is unreachable gets an ``{"error": ...}`` entry — never omitted."""
aggregated from the k7-agent DaemonSet. A node whose agent
is unreachable gets an ``{"error": ...}`` entry — never omitted.
Cluster-scoped (all-namespaces): a namespace-scoped key is 403.
Fail loud — do not silently narrow the per-node map.
"""
authorize_namespace(key_data, None, all_namespaces=True)
core = K7Core()
return success_response(await core.nodes_storage())
@@ -411,7 +419,7 @@ async def get_sandbox_metrics(namespace: str | None = None, key_data: dict = Dep
# ---------------------------------------------------------------------------
# Spec 10e: VolumeSnapshot lifecycle endpoints.
# VolumeSnapshot lifecycle endpoints.
# ---------------------------------------------------------------------------
@@ -497,7 +505,7 @@ async def delete_snapshot(name: str, namespace: str = "default", key_data: dict
@app.post("/api/v1/snapshots/{name}/restore")
async def restore_snapshot(name: str, body: dict, key_data: dict = Depends(verify_api_key)):
"""Boot a brand-new sandbox from a standalone VolumeSnapshot (Spec 10f).
"""Boot a brand-new sandbox from a standalone VolumeSnapshot.
Body keys:
``new_sandbox_name`` (required),
+5 -1
View File
@@ -1,4 +1,4 @@
"""Snapshot garbage-collection entrypoint (Spec 10e, Option C backstop).
"""Snapshot garbage-collection entrypoint (Option C backstop).
Run as ``python -m k7.api.snapshot_gc`` inside the ``k7-api`` container.
The accompanying CronJob (``snapshot-gc-cronjob.yaml``) invokes this
@@ -12,6 +12,10 @@ Behaviour:
- Honours the same ``keep_fork_for`` window as :meth:`K7Core.gc_snapshots`.
- Skips anything that isn't ``kind=fork`` — pause and named snapshots are
never touched.
- Reaps orphan ``VolumeSnapshotContent`` objects (and stuck
``pvc-as-source-protection`` PVC finalizers) whose VolumeSnapshot is
already gone. That is what otherwise pins ``k7-test-*`` namespaces in
Terminating after snapshot-lifecycle tests.
- Environment overrides: ``K7_GC_KEEP_FORK_FOR_MINUTES`` (default ``10``),
``K7_GC_DRY_RUN`` (``true``/``false``, default ``false``).
"""
+41
View File
@@ -0,0 +1,41 @@
#!/bin/sh
# Copy docker CLI + compose/buildx into a per-pod emptyDir.
# Fail loud if a source path is missing — plugin layout differs by image.
set -eu
DEST="${1:-/cli}"
mkdir -p "${DEST}/bin" "${DEST}/cli-plugins"
src_docker=""
for p in /usr/local/bin/docker /usr/bin/docker; do
if [ -x "${p}" ]; then
src_docker=${p}
break
fi
done
[ -n "${src_docker}" ] || {
echo "k7-docker-cli-copy: docker CLI not found in vehicle image" >&2
exit 1
}
cp -a "${src_docker}" "${DEST}/bin/docker"
for name in docker-compose docker-buildx; do
found=""
for p in \
"/usr/local/libexec/docker/cli-plugins/${name}" \
"/usr/libexec/docker/cli-plugins/${name}" \
"/usr/local/lib/docker/cli-plugins/${name}" \
"/usr/lib/docker/cli-plugins/${name}"; do
if [ -e "${p}" ]; then
found=${p}
break
fi
done
[ -n "${found}" ] || {
echo "k7-docker-cli-copy: ${name} not found in dind image (searched libexec/lib plugin dirs)" >&2
ls -la /usr/local/libexec/docker/cli-plugins /usr/libexec/docker/cli-plugins \
/usr/local/lib/docker/cli-plugins /usr/lib/docker/cli-plugins 2>&1 || true
exit 1
}
cp -a "${found}" "${DEST}/cli-plugins/${name}"
done
+23
View File
@@ -0,0 +1,23 @@
#!/bin/sh
# kfd: Kata Firecracker virtio-fs subPath file mounts are not visible to
# Docker CLI plugin discovery. Stage compose/buildx from the directory
# emptyDir at /run/k7/docker-cli (kql persist-bind already does this).
set -eu
STAGING="${K7_DOCKER_CLI_STAGING:-/run/k7/docker-cli}"
mkdir -p /root/.docker/cli-plugins
if [ -d "${STAGING}/cli-plugins" ]; then
for p in "${STAGING}/cli-plugins"/*; do
[ -e "${p}" ] || continue
dest="/root/.docker/cli-plugins/$(basename "${p}")"
cp -a "${p}" "${dest}"
chmod +x "${dest}"
done
fi
if [ ! -f /root/.docker/config.json ]; then
printf '%s\n' "{\"cliPluginsExtraDirs\":[\"${STAGING}/cli-plugins\"]}" >/root/.docker/config.json
fi
if [ "$#" -eq 0 ]; then
exec sleep 365d
fi
exec "$@"
+128
View File
@@ -0,0 +1,128 @@
#!/bin/sh
# Docker vehicle for Kata backends. Overlay2 on a block
# device or fail loud — no driver auto-selection, no virtio-fs graph.
set -eu
DEVICE="/dev/k7docker"
GRAPH="/var/lib/docker"
SOCK_DIR="/run/k7/docker"
SOCK="${SOCK_DIR}/docker.sock"
log() { echo "k7-docker-vehicle: $*" >&2; }
die() {
log "$*"
exit 1
}
i=0
while [ ! -b "${DEVICE}" ]; do
i=$((i + 1))
if [ "${i}" -gt 60 ]; then
ls -l "${DEVICE}" /dev 2>&1 || true
die "${DEVICE} is not a block device"
fi
sleep 0.5
done
# The dind image VOLUME /var/lib/docker is virtio-fs on Kata. Overlay2
# cannot live there — unmount it, then put ext4 on the block device.
i=0
while grep -q " ${GRAPH} " /proc/mounts; do
i=$((i + 1))
[ "${i}" -le 8 ] || die "could not unmount existing filesystem from ${GRAPH}"
log "unmounting existing mount from ${GRAPH}: $(grep " ${GRAPH} " /proc/mounts || true)"
umount "${GRAPH}" || umount -l "${GRAPH}" || die "umount ${GRAPH} failed"
done
mkdir -p "${GRAPH}"
# Try mounting first. A restore/fork clone already has ext4; mkfs would
# wipe it. Longhorn clones can take a moment to present the superblock.
mounted=0
i=0
while [ "${i}" -lt 60 ]; do
if mount -t ext4 "${DEVICE}" "${GRAPH}"; then
mounted=1
break
fi
i=$((i + 1))
sleep 0.5
done
if [ "${mounted}" -eq 0 ]; then
has_fs=0
blkid_out=""
if command -v blkid >/dev/null 2>&1; then
blkid_out=$(blkid "${DEVICE}" 2>/dev/null || true)
case "${blkid_out}" in
*TYPE=\"ext4\"* | *TYPE=ext4*) has_fs=1 ;;
esac
fi
[ "${has_fs}" -eq 0 ] || die "ext4 on ${DEVICE} but mount failed (blkid='${blkid_out}')"
command -v mkfs.ext4 >/dev/null 2>&1 || die "mkfs.ext4 not found in vehicle image"
log "formatting ${DEVICE} as ext4 (blkid='${blkid_out:-empty}')"
mkfs.ext4 -F -m 0 "${DEVICE}"
mount -t ext4 "${DEVICE}" "${GRAPH}" || {
log "mount -t ext4 failed after mkfs; blkid=$(blkid "${DEVICE}" 2>&1 || true)"
die "mount ${DEVICE} on ${GRAPH} failed"
}
fi
assert_ext4() {
fstype=""
src=""
if command -v findmnt >/dev/null 2>&1; then
fstype=$(findmnt -no FSTYPE "${GRAPH}")
src=$(findmnt -no SOURCE "${GRAPH}")
else
# /proc/mounts is the same assertion without findmnt — not a driver fallback.
while read -r dev mnt fs _rest; do
if [ "${mnt}" = "${GRAPH}" ]; then
fstype=${fs}
src=${dev}
break
fi
done </proc/mounts
fi
[ "${fstype}" = "ext4" ] || die "${GRAPH} fstype is '${fstype}', want ext4 (not virtiofs/tmpfs)"
case "${src}" in
"${DEVICE}" | "${DEVICE} "* | *"${DEVICE}") ;;
*)
real=$(readlink -f "${DEVICE}" 2>/dev/null || echo "${DEVICE}")
case "${src}" in
"${real}" | "${real} "*) ;;
*) die "${GRAPH} source is '${src}', want block device ${DEVICE}" ;;
esac
;;
esac
}
assert_ext4
mkdir -p "${SOCK_DIR}"
# Path sharing: /tmp is a per-pod emptyDir mounted in both containers
# (Kata rejects mount --bind from /proc/PID/root). Persist dirs are
# bind-mounted from the kql root PVC when present.
if grep -q " /tmp " /proc/mounts; then
log "/tmp is a shared volume"
else
die "/tmp is not a mount; want path-share-tmp emptyDir in both containers"
fi
SLOT="${K7_PERSIST_SLOT:-main}"
if [ -d "/mnt/state/${SLOT}" ]; then
for d in home root opt workspace; do
persist="/mnt/state/${SLOT}/${d}"
[ -d "${persist}" ] || continue
if grep -q " /${d} " /proc/mounts; then
umount "/${d}" 2>/dev/null || umount -l "/${d}" 2>/dev/null || true
fi
mkdir -p "/${d}"
if mount --bind "${persist}" "/${d}"; then
log "shared /${d} <- ${persist}"
else
log "bind ${persist} -> /${d} failed"
fi
done
fi
command -v dockerd >/dev/null 2>&1 || die "dockerd not found in vehicle image"
exec dockerd --host="unix://${SOCK}" --tls=false --storage-driver=overlay2
Regular → Executable
+49
View File
@@ -73,6 +73,43 @@ bind_mount_dir() {
echo "k7-persist: bound $target -> $persist"
}
install_docker_plugin() {
src="$1"
dst="$2"
[ -x "$src" ] || {
echo "k7-persist: missing docker plugin $src" >&2
exit 1
}
mkdir -p "$(dirname "$dst")"
# Docker CLI plugin discovery skips symlinks; copy a regular file.
cp -a "$src" "$dst"
chmod +x "$dst"
}
restore_k7_docker_cli() {
# persist-bind overlays /usr and would hide the CLI subPath mounts.
# Restage from the emptyDir at /run/k7/docker-cli (not under /usr).
# Call after /usr and /root binds so these writes land on the PVC.
if [ ! -x /run/k7/docker-cli/bin/docker ]; then
return 0
fi
mkdir -p /usr/local/bin /usr/local/lib/docker/cli-plugins /root/.docker/cli-plugins
cp -a /run/k7/docker-cli/bin/docker /usr/local/bin/docker
chmod +x /usr/local/bin/docker
install_docker_plugin /run/k7/docker-cli/cli-plugins/docker-compose \
/usr/local/lib/docker/cli-plugins/docker-compose
install_docker_plugin /run/k7/docker-cli/cli-plugins/docker-buildx \
/usr/local/lib/docker/cli-plugins/docker-buildx
install_docker_plugin /run/k7/docker-cli/cli-plugins/docker-compose \
/root/.docker/cli-plugins/docker-compose
install_docker_plugin /run/k7/docker-cli/cli-plugins/docker-buildx \
/root/.docker/cli-plugins/docker-buildx
if [ ! -f /root/.docker/config.json ]; then
printf '%s\n' '{"cliPluginsExtraDirs":["/run/k7/docker-cli/cli-plugins"]}' >/root/.docker/config.json
fi
echo "k7-persist: docker CLI -> /run/k7/docker-cli" >&2
}
bind_mount_dir /etc etc
bind_mount_dir /var var
bind_mount_dir /usr usr
@@ -83,11 +120,23 @@ bind_mount_dir /bin bin
bind_mount_dir /sbin sbin
bind_mount_dir /lib lib
bind_mount_dir /lib64 lib64
restore_k7_docker_cli
SCRIPT_END=$(date +%s)
SCRIPT_ELAPSED=$((SCRIPT_END - SCRIPT_START))
echo "k7-persist: script completed in ${SCRIPT_ELAPSED}s" >&2
# Directory socket share. Tools that ignore DOCKER_HOST and
# hardcode /var/run/docker.sock get a symlink; never overlay /var/run.
if [ -d /run/k7/docker ]; then
mkdir -p /run
ln -sfn /run/k7/docker/docker.sock /run/docker.sock
if [ -d /var/run ]; then
ln -sfn /run/k7/docker/docker.sock /var/run/docker.sock
fi
echo "k7-persist: docker.sock -> /run/k7/docker/docker.sock" >&2
fi
date > "${MIGRATION_MARK}" 2>/dev/null || true
if [ "$#" -eq 0 ]; then
echo "k7-persist: no command provided; defaulting to keepalive sleep" >&2
+38 -11
View File
@@ -1,4 +1,4 @@
"""CLI ↔ API client plumbing (Spec 10g).
"""CLI ↔ API client plumbing.
This module owns:
@@ -39,6 +39,7 @@ from ._config import get_config_value
_ETC_API_ENDPOINT = Path("/etc/k7/api_endpoint")
_ETC_API_KEYS = Path("/etc/k7/api_keys.json")
_ETC_API_CA = Path("/etc/k7/tls/ca.crt")
def _resolve_api_url(flag: str | None) -> str | None:
@@ -103,6 +104,28 @@ def _resolve_api_key(flag: str | None) -> str | None:
return None
def _resolve_api_ca(flag: str | None) -> str | None:
"""Look up a CA file: ``--api-ca`` > ``K7_API_CA`` > ``api.ca`` > ``/etc/k7/tls/ca.crt``."""
if flag:
return flag
env = os.environ.get("K7_API_CA")
if env:
return env
cfg = get_config_value("api.ca")
if cfg:
return cfg
if _ETC_API_CA.exists():
return str(_ETC_API_CA)
return None
def verify_ssl_for_url(url: str, ca: str | None) -> bool | str:
"""SDK ``verify_ssl`` value. Never ``False`` for ``https://``."""
if url.startswith("https://"):
return ca if ca else True
return False
# ---------------------------------------------------------------------------
# CliContext + adapters.
# ---------------------------------------------------------------------------
@@ -125,12 +148,13 @@ class CliContext:
use_core: bool = False
api_url: str | None = None
api_key: str | None = None
api_ca: str | None = None
_client: Client | None = None
def client(self) -> Client:
"""Resolve and cache the SDK client; exits 1 when the URL / key are unset."""
if self._client is None:
self._client = resolve_client(self.api_url, self.api_key)
self._client = resolve_client(self.api_url, self.api_key, self.api_ca)
return self._client
@@ -138,12 +162,14 @@ class ApiUnreachable(RuntimeError):
"""Raised when the SDK can't reach the API at all (connection / timeout)."""
def resolve_client(api_url: str | None, api_key: str | None) -> Client:
"""Build a ``katakate.Client`` from the resolved URL + key.
def resolve_client(api_url: str | None, api_key: str | None, api_ca: str | None = None) -> Client:
"""Build a ``katakate.Client`` from the resolved URL + key + CA.
Raises ``typer.Exit(1)`` with a pointed message when either is
missing — the CLI should never silently fall through to a broken
Client constructor.
Raises ``typer.Exit(1)`` with a pointed message when either the URL
or the key is missing — the CLI should never silently fall through
to a broken Client constructor. ``verify=False`` is never used for
an ``https://`` URL: a missing CA falls through to the
system trust store (Let's Encrypt); a broken CA is an error.
"""
url = _resolve_api_url(api_url)
if not url:
@@ -162,10 +188,11 @@ def resolve_client(api_url: str | None, api_key: str | None) -> Client:
err=True,
)
raise typer.Exit(1)
# NodePort exposes plain HTTP; off-cluster setups should put the API
# behind ingress + TLS, in which case verify_ssl=True (the default)
# does the right thing. For local/demo HTTP, the SDK skips verify.
return Client(endpoint=url, api_key=key, verify_ssl=url.startswith("https://"))
ca = _resolve_api_ca(api_ca)
if ca and not Path(ca).is_file():
typer.echo(f"❌ API CA file not found: {ca}", err=True)
raise typer.Exit(1)
return Client(endpoint=url, api_key=key, verify_ssl=verify_ssl_for_url(url, ca))
T = TypeVar("T")
+7 -7
View File
@@ -1,11 +1,11 @@
"""TOML config file CRUD + ``k7 config`` sub-app (Spec 10g).
"""TOML config file CRUD + ``k7 config`` sub-app.
Stores per-user CLI config at ``$XDG_CONFIG_HOME/k7/config.toml``
(falling back to ``~/.config/k7/config.toml``). The single supported
section today is ``[api]`` with ``url`` and ``key`` keys — enough to
power the ``_resolve_api_url`` / ``_resolve_api_key`` chain in
``_client.py`` so users don't have to pass ``--api-url`` /
``--api-key`` on every invocation.
section today is ``[api]`` with ``url``, ``key``, and ``ca`` keys —
enough to power the URL / key / CA chain in ``_client.py`` so users
don't have to pass ``--api-url`` / ``--api-key`` / ``--api-ca`` on
every invocation.
The file is the same posture as ``~/.docker/config.json`` and
``~/.kube/config`` — plaintext, chmod 0600 on write. A future spec
@@ -24,7 +24,7 @@ from pathlib import Path
import typer
CONFIG_DIR_ENV = "K7_CONFIG_DIR" # tests override this to a tmp path
_SUPPORTED_KEYS: set[str] = {"api.url", "api.key"}
_SUPPORTED_KEYS: set[str] = {"api.url", "api.key", "api.ca"}
def _config_dir() -> Path:
@@ -53,7 +53,7 @@ def _toml_loads_simple(text: str) -> dict:
"""Tiny TOML reader for the section-of-string-values shape this file uses.
Python 3.10 (k7's minimum) lacks ``tomllib`` and we don't want to add
``tomli`` just for two-key parsing. Supports::
``tomli`` just for this file. Supports::
[api]
url = "https://10.0.0.1:31000"
+314 -26
View File
@@ -3,6 +3,7 @@
import asyncio
import builtins
import hashlib
import ipaddress
import json
import os
import secrets
@@ -33,6 +34,7 @@ from k7 import __version__ as K7_VERSION
from k7.cli._client import CliContext, handle_api_call
from k7.cli._config import config_app
from k7.core.core import K7Core
from k7.core.docker import parse_docker_disk
from k7.core.models import SandboxConfig, SandboxConfigOverrides
from k7.core.sidecar import SIDECAR_REGISTRY
@@ -63,6 +65,12 @@ def _main(
envvar="K7_API_KEY",
help="API key for the K7 API (overrides env / config file).",
),
api_ca: str | None = typer.Option(
None,
"--api-ca",
envvar="K7_API_CA",
help="CA file to trust for https:// API URLs (overrides env / config / /etc/k7/tls/ca.crt).",
),
use_core: bool = typer.Option(
False,
"--core",
@@ -82,7 +90,7 @@ def _main(
raise typer.Exit()
# CliContext is built lazily — handlers that never call ``ctx.obj.client()``
# (install / start-api / config / ...) don't trigger the missing-URL error.
ctx.obj = CliContext(use_core=use_core, api_url=api_url, api_key=api_key)
ctx.obj = CliContext(use_core=use_core, api_url=api_url, api_key=api_key, api_ca=api_ca)
if ctx.invoked_subcommand is None:
# Show top-level help when no command is provided
try:
@@ -93,6 +101,7 @@ def _main(
API_KEYS_FILE = Path(os.getenv("K7_API_KEYS_FILE", "/etc/k7/api_keys.json"))
_ETC_K7_API_ENDPOINT = Path("/etc/k7/api_endpoint")
# The k7-api container runs as this fixed non-root uid (`useradd -u 1000
# k7user` in src/k7/api/Dockerfile.api) and reads the key store through a
# hostPath mount. The store holds sha256 *hashes* (never raw keys), but we
@@ -123,11 +132,12 @@ def _write_api_keys(api_keys: dict) -> None:
)
BACKEND_ALLOWED = {"kata-firecracker-devmapper", "kata-qemu-longhorn", "k7d"}
BACKEND_ALLOWED = {"kata-firecracker-devmapper", "kata-qemu-longhorn", "k7d", "k7d-fc"}
BACKEND_ALIASES = {
"kfd": "kata-firecracker-devmapper",
"kql": "kata-qemu-longhorn",
"k7": "k7d",
"k7-fc": "k7d-fc",
}
# Deprecated short + pre-rename full names. Still accepted; emit a warning.
BACKEND_DEPRECATED_ALIASES = {
@@ -158,7 +168,7 @@ def _normalize_backend(backend: str | None) -> str | None:
return value
allowed = ", ".join(sorted(BACKEND_ALLOWED))
raise typer.BadParameter(
f"Unsupported backend '{backend}'. Use {allowed} (aliases: kfd, kql, k7; deprecated: fd, ql)."
f"Unsupported backend '{backend}'. Use {allowed} (aliases: kfd, kql, k7, k7-fc; deprecated: fd, ql)."
)
@@ -237,16 +247,72 @@ def _build_default_inventory(
return "\n".join(lines)
def _tls_cert_key_match(cert: Path, key: Path) -> None:
"""Refuse a cert/key pair that openssl does not consider matching."""
if not cert.is_file():
raise typer.BadParameter(f"--api-tls-cert '{cert}' is not a file")
if not key.is_file():
raise typer.BadParameter(f"--api-tls-key '{key}' is not a file")
cert_pub = subprocess.run(
["openssl", "x509", "-in", str(cert), "-noout", "-pubkey"],
capture_output=True,
text=True,
)
key_pub = subprocess.run(
["openssl", "pkey", "-in", str(key), "-pubout"],
capture_output=True,
text=True,
)
if cert_pub.returncode != 0:
raise typer.BadParameter(f"--api-tls-cert is not a readable PEM certificate: {cert_pub.stderr.strip()}")
if key_pub.returncode != 0:
raise typer.BadParameter(f"--api-tls-key is not a readable PEM private key: {key_pub.stderr.strip()}")
if cert_pub.stdout != key_pub.stdout:
raise typer.BadParameter("--api-tls-cert and --api-tls-key are not a matching pair")
def _validate_api_hostname(name: str) -> str:
host = name.strip()
if not host:
raise typer.BadParameter("--api-hostname must be a DNS name")
try:
ipaddress.ip_address(host)
except ValueError:
return host
raise typer.BadParameter(
"--api-hostname must be a DNS name, not an IP (Let's Encrypt will not issue for a bare IP)"
)
def _read_api_endpoint(kubectl: list[str]) -> str | None:
"""Read the K7 API NodePort endpoint from the K3s service."""
"""Read the K7 API NodePort endpoint from the file the playbook wrote, or the Service."""
etc = _ETC_K7_API_ENDPOINT
if etc.exists():
try:
written = etc.read_text().strip()
if written:
return written
except OSError:
pass
port_result = subprocess.run(
kubectl + ["get", "svc", "k7-api", "-n", "kube-system", "-o", "jsonpath={.spec.ports[0].nodePort}"],
kubectl
+ [
"get",
"svc",
"k7-api",
"-n",
"kube-system",
"-o",
"jsonpath={.spec.ports[0].nodePort},{.spec.ports[0].targetPort}",
],
capture_output=True,
text=True,
)
if port_result.returncode != 0 or not port_result.stdout.strip():
return None
node_port = port_result.stdout.strip()
node_port, _, target_port = port_result.stdout.strip().partition(",")
if not node_port:
return None
# Get node addresses as JSON and pick the first IPv4 InternalIP
node_result = subprocess.run(
@@ -265,7 +331,8 @@ def _read_api_endpoint(kubectl: list[str]) -> str | None:
except (json.JSONDecodeError, TypeError, KeyError):
pass
return f"http://{node_ip}:{node_port}"
scheme = "https" if target_port.strip() == "8443" else "http"
return f"{scheme}://{node_ip}:{node_port}"
def _get_api_manifests_dir() -> str:
@@ -304,16 +371,16 @@ def install(
"-b",
help=(
"Comma-separated sandbox backends to install on this node. "
"Choices: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d. "
"Default installs the two Kata backends; add k7d explicitly for the "
"warm-fork microVM runtime."
"Choices: kata-firecracker-devmapper (kfd), kata-qemu-longhorn (kql), k7d, k7d-fc (alias k7-fc). "
"Default installs the two Kata backends; add k7d / k7d-fc explicitly for the "
"warm-fork microVM runtime (k7d-fc is Firecracker under the jailer)."
),
show_default=True,
),
k7d_version: str | None = typer.Option(
None,
"--k7d-version",
help="k7d release version to install (k7d backend only; default from playbook: 0.2.1)",
help="k7d release version to install (k7d backend only; default from playbook: 0.6.0)",
),
k7d_artifact: str | None = typer.Option(
None,
@@ -365,6 +432,46 @@ def install(
help="CNI plugin: cilium (default, enables FQDN egress) or flannel (CIDR-only)",
show_default=True,
),
hubble: bool = typer.Option(
False,
"--hubble",
help=("Enable Cilium Hubble flow observability (relay + CLI; no UI). Off by default. Requires --cni cilium."),
),
api_allow_cidr: list[str] | None = typer.Option(
None,
"--api-allow-cidr",
help=(
"Restrict the k7-api NodePort to these source CIDRs (repeatable, e.g. 203.0.113.4/32). "
"Off by default: the API answers any client that has an API key. Requires --cni cilium. "
"Defence in depth only — this is not rate limiting and not a secure-API claim."
),
),
api_insecure_http: bool = typer.Option(
False,
"--api-insecure-http",
help="Serve k7-api as today's plain HTTP NodePort (no Caddy sidecar, keys travel in cleartext).",
),
api_hostname: str | None = typer.Option(
None,
"--api-hostname",
help="DNS name pointing at the first master; Caddy obtains a Let's Encrypt cert (HTTP-01 on :80).",
),
api_tls_cert: str | None = typer.Option(
None,
"--api-tls-cert",
help="Operator-supplied server certificate (PEM). Requires --api-tls-key.",
),
api_tls_key: str | None = typer.Option(
None,
"--api-tls-key",
help="Operator-supplied server private key (PEM). Requires --api-tls-cert.",
),
api_acme_staging: bool = typer.Option(
False,
"--api-acme-staging",
hidden=True,
help="Use the Let's Encrypt staging ACME directory (integration tests).",
),
no_api: bool = typer.Option(
False,
"--no-api",
@@ -377,8 +484,9 @@ def install(
Single-node (default): `k7 install` provisions localhost with **both**
Kata backends (kata-firecracker-devmapper + kata-qemu-longhorn). Pass
`--backend kfd,kql,k7d` to also install the k7d warm-fork runtime
(downloads `Katakate/k7d` v0.2.1 unless `--k7d-version` / `--k7d-artifact`
override it).
(downloads `Katakate/k7d` v0.6.0 unless `--k7d-version` / `--k7d-artifact`
override it). `--backend k7d-fc` (alias `k7-fc`) adds RuntimeClass
`k7-fc` and the pinned Firecracker+jailer next to that daemon.
Multi-node: `k7 install -i inventory.ini` reads roles, backends, and disks
from the Ansible inventory. See `inventory.ini.example` for the layout.
@@ -391,6 +499,56 @@ def install(
cni_value = (cni or "").strip().lower()
if cni_value not in ("cilium", "flannel"):
raise typer.BadParameter(f"--cni must be 'cilium' or 'flannel', got '{cni}'")
if hubble and cni_value != "cilium":
raise typer.BadParameter(f"--hubble requires the Cilium CNI (got --cni {cni_value}).")
# Validate the k7-api allowlist here, before Ansible runs. A bad
# CIDR must never reach a security policy.
api_cidrs = [c.strip() for c in (api_allow_cidr or []) if c.strip()]
if api_cidrs:
if cni_value != "cilium":
raise typer.BadParameter(
f"--api-allow-cidr requires the Cilium CNI (got --cni {cni_value}). The reserved "
"host/remote-node/health/kube-apiserver peers are not expressible in a v1 "
"NetworkPolicy, and a v1 policy without `host` would break the kubelet probes on "
"the k7-api pod."
)
for cidr in api_cidrs:
try:
network = ipaddress.ip_network(cidr, strict=False)
except ValueError as exc:
raise typer.BadParameter(f"--api-allow-cidr '{cidr}' is not a valid CIDR: {exc}") from exc
if network.prefixlen == 0:
typer.echo(
f"⚠️ --api-allow-cidr {cidr} allows every source address — that is today's "
"behaviour with extra steps. Pass the CIDRs your clients actually come from.",
err=True,
)
# TLS flag combinations fail here, before Ansible.
hostname = (api_hostname or "").strip() or None
cert_path = Path(api_tls_cert).expanduser() if api_tls_cert else None
key_path = Path(api_tls_key).expanduser() if api_tls_key else None
if api_insecure_http and (hostname or cert_path or key_path):
raise typer.BadParameter(
"--api-insecure-http cannot be combined with --api-hostname or --api-tls-cert/--api-tls-key"
)
if hostname and (cert_path or key_path):
raise typer.BadParameter("--api-hostname cannot be combined with --api-tls-cert/--api-tls-key")
if (cert_path is None) ^ (key_path is None):
raise typer.BadParameter("--api-tls-cert and --api-tls-key must be passed together")
if hostname:
hostname = _validate_api_hostname(hostname)
if cert_path is not None and key_path is not None:
_tls_cert_key_match(cert_path.resolve(), key_path.resolve())
if api_insecure_http and api_cidrs:
typer.echo(
"⚠️ --api-insecure-http with --api-allow-cidr: the allowlist still applies, "
"but API keys still travel in cleartext.",
err=True,
)
if api_acme_staging and not hostname:
raise typer.BadParameter("--api-acme-staging requires --api-hostname")
if join and not join_token:
raise typer.BadParameter("--join requires --join-token")
@@ -517,12 +675,23 @@ def install(
"longhorn_data_path": longhorn_data_path,
"longhorn_extra_disk": longhorn_extra_disk,
"k7_cni": cni_value,
"k7_hubble_enabled": "true" if hubble else "false",
"k7_repo_root": repo_root,
# Spec 10d: ``--no-api`` flips the playbook's existing
# ``--no-api`` flips the playbook's existing
# ``k7_api_enabled`` gate (default true) so the API
# manifests aren't applied.
"k7_api_enabled": "false" if no_api else "true",
# k7d backend artifact source overrides (spec 9a M11).
# Empty means no CiliumNetworkPolicy and no Service
# change — today's install, bit for bit.
"k7_api_allow_cidrs": ",".join(api_cidrs),
# Default is HTTPS + playbook CA. Extra-vars are
# strings so Ansible's `| bool` filter stays deterministic.
"k7_api_insecure_http": "true" if api_insecure_http else "false",
"k7_api_hostname": hostname or "",
"k7_api_tls_cert_src": str(cert_path.resolve()) if cert_path else "",
"k7_api_tls_key_src": str(key_path.resolve()) if key_path else "",
"k7_api_acme_staging": "true" if api_acme_staging else "false",
# k7d backend artifact source overrides.
"k7d_version": k7d_version,
"k7d_artifact_local_path": k7d_artifact,
}
@@ -531,7 +700,7 @@ def install(
# Ansible extra-vars have the highest precedence and would clobber
# per-host `k7_backends` declared in a user-supplied inventory. Only
# forward the CLI value when the user explicitly passed --backend;
# otherwise the inventory is authoritative (spec 18e).
# otherwise the inventory is authoritative.
backend_explicit = ctx.get_parameter_source("backend") == ParameterSource.COMMANDLINE
if inventory_user_supplied and not backend_explicit:
extra_vars.pop("k7_backends", None)
@@ -596,6 +765,28 @@ def create(
"Without this flag and without --egress, all egress is blocked."
),
),
ingress_port: list[int] | None = typer.Option(
None,
"--ingress-port",
help=("TCP port to open for inbound traffic. Repeatable. Default (no --ingress-port): deny all ingress."),
),
ingress_from: list[str] | None = typer.Option(
None,
"--ingress-from",
help=(
"Who may reach --ingress-port: sandbox:<name> | namespace:<ns> | cidr:<cidr>. Repeatable. "
"Default when --ingress-port is given: sandboxes in the same namespace. "
"Reaching the sandbox from anywhere needs cidr:0.0.0.0/0, spelled out."
),
),
expose_port: list[int] | None = typer.Option(
None,
"--expose-port",
help=(
"Publish a TCP port outside the cluster through a NodePort Service. Repeatable. "
"Requires a matching --ingress-port. On a public node the node IP is public too."
),
),
before_script: str | None = typer.Option(
None,
"--before-script",
@@ -645,21 +836,43 @@ def create(
None,
"--backend",
"-b",
help="Backend: kata-firecracker-devmapper (kfd) or kata-qemu-longhorn (kql) (auto-detected if not specified)",
help="Backend: kfd, kql, k7d, or k7d-fc (alias k7-fc) (auto-detected if not specified)",
),
sidecar: str | None = typer.Option(
None,
"--sidecar",
help="Sidecar daemon type to inject (e.g. 'docker'). See SIDECAR_REGISTRY for available types.",
help=(
"Sidecar daemon type to inject. --sidecar docker is a deprecated alias of "
"--docker. Other types: see SIDECAR_REGISTRY."
),
),
docker: bool = typer.Option(
False,
"--docker",
help=(
"Run a pinned dockerd (overlay2 on a block graph disk). "
"k7d/k7d-fc: guest agent service, forkable. Kata (kfd/kql): privileged "
"docker-vehicle in the VM; kql persists the graph, kfd is ephemeral."
),
),
docker_disk: str | None = typer.Option(
None,
"--docker-disk",
help="Docker graph disk size for --docker (e.g. 20Gi, 40Gi). Default 20Gi on all backends.",
),
):
"""Create a new sandbox from YAML config or CLI arguments."""
# Spec 18f issue 4: three explicit egress modes. --egress-open → open
# Three explicit egress modes. --egress-open → open
# (no policy, egress_whitelist=None); --egress ... → whitelist; neither
# → block-all ([]). Combining both is ambiguous — fail loudly.
if egress_open and egress_whitelist:
raise typer.BadParameter("--egress-open cannot be combined with --egress (pick one egress mode)")
# Sources without ports open nothing — the user believes they
# exposed the sandbox and did not, so fail instead of guessing.
if ingress_from and not ingress_port:
raise typer.BadParameter("--ingress-from requires --ingress-port (sources alone open no port)")
# Auto-detect default config file in current directory when not provided
if not config:
for candidate in ("k7.yaml", "k7.yml"):
@@ -685,6 +898,12 @@ def create(
sandbox_config.egress_whitelist = None
elif egress_whitelist:
sandbox_config.egress_whitelist = egress_whitelist
if ingress_port:
sandbox_config.ingress_ports = ingress_port
if ingress_from:
sandbox_config.ingress_from = ingress_from
if expose_port:
sandbox_config.expose_ports = expose_port
if before_script:
sandbox_config.before_script = before_script
# Security overrides from CLI take precedence when provided
@@ -708,6 +927,10 @@ def create(
sandbox_config.backend = _normalize_backend(backend)
if sidecar is not None:
sandbox_config.sidecar = sidecar
if docker:
sandbox_config.docker = True
if docker_disk is not None:
sandbox_config.docker_disk = docker_disk
if not sandbox_config.image:
raise typer.BadParameter("image is required")
@@ -742,6 +965,9 @@ def create(
namespace=namespace,
env_file=env_file,
egress_whitelist=None if egress_open else (egress_whitelist or []),
ingress_ports=ingress_port or None,
ingress_from=ingress_from or None,
expose_ports=expose_port or None,
limits=limits if limits else None,
before_script=before_script or "",
entrypoint=entrypoint,
@@ -754,15 +980,34 @@ def create(
runtime_class_name=runtime_class,
root_disk_size=root_disk_size,
backend=_normalize_backend(backend) if backend else None,
docker=docker,
docker_disk=docker_disk,
)
resolved_backend = _normalize_backend(sandbox_config.backend) if sandbox_config.backend else None
if root_disk_size and resolved_backend == "kata-firecracker-devmapper":
typer.echo("⚠️ --root-disk-size has no effect with the kata-firecracker-devmapper backend", err=True)
if sandbox_config.sidecar == "docker":
typer.echo("⚠️ --sidecar docker is deprecated; use --docker", err=True)
sandbox_config.docker = True
sandbox_config.sidecar = None
if sandbox_config.docker and sandbox_config.sidecar:
raise typer.BadParameter("--docker and --sidecar are mutually exclusive until multiple guest services exist")
if sandbox_config.docker_disk:
if not sandbox_config.docker:
raise typer.BadParameter("--docker-disk requires --docker")
try:
sandbox_config.docker_disk = parse_docker_disk(sandbox_config.docker_disk)
except ValueError as e:
raise typer.BadParameter(str(e)) from e
if sandbox_config.sidecar is not None and sandbox_config.sidecar not in SIDECAR_REGISTRY:
available = ", ".join(sorted(SIDECAR_REGISTRY.keys()))
raise typer.BadParameter(f"Unknown sidecar type '{sandbox_config.sidecar}'. Available: {available}")
others = ", ".join(sorted(k for k in SIDECAR_REGISTRY if k != "docker"))
hint = others if others else "no non-docker sidecar types (--sidecar docker is --docker)"
raise typer.BadParameter(f"Unknown sidecar type '{sandbox_config.sidecar}'. Available: {hint}")
core = K7Core()
progress = Progress(
@@ -885,10 +1130,42 @@ def create(
except Exception:
cap_drop_display = "unknown"
# A NodePort on a Hetzner dedicated box lands on a public IP: say so on
# every create, not just once in the docs.
if sandbox_config.expose_ports and "cidr:0.0.0.0/0" in (sandbox_config.ingress_from or []):
typer.echo(
f"⚠️ {sandbox_config.name} is exposed on ports {sandbox_config.expose_ports} to cidr:0.0.0.0/0 — "
"on a public node this publishes the sandbox to the internet.",
err=True,
)
if sandbox_config.ingress_ports:
ingress_mode = (
f"ports={sandbox_config.ingress_ports} from="
f"{sandbox_config.ingress_from or ['sandboxes in this namespace']}"
)
else:
ingress_mode = "deny_all"
expose_mode = f"NodePort for {sandbox_config.expose_ports}" if sandbox_config.expose_ports else "none"
# Measured on Cilium: an ipBlock peer is not evaluated for pod-to-pod
# traffic, so a rule carrying one degrades to a port-only allow in-cluster.
# The flag reads like a restriction, so say plainly that it is not one.
if any((s or "").strip().startswith("cidr:") for s in sandbox_config.ingress_from or []):
typer.echo(
f"⚠️ {sandbox_config.name}: a cidr: source does not restrict in-cluster peers on Cilium — "
f"ports {sandbox_config.ingress_ports} are reachable from any sandbox in the cluster. "
"Use sandbox:<name> or namespace:<ns> to scope in-cluster access.",
err=True,
)
params_lines = [
f"Image: {sandbox_config.image}",
f"Namespace: {sandbox_config.namespace}",
f"Egress: {egress_mode}",
f"Ingress: {ingress_mode}",
f"Expose: {expose_mode}",
f"Before script: {'present' if (sandbox_config.before_script or '').strip() else 'none'}",
f"Pod non-root: {getattr(sandbox_config, 'pod_non_root', False)}",
f"Container non-root: {getattr(sandbox_config, 'container_non_root', False)}",
@@ -1136,6 +1413,7 @@ def create(
pass
cli_ctx: CliContext = ctx.obj
endpoints: builtins.list[dict] = []
if cli_ctx.use_core:
group = RichGroup(status_text, details_text, progress)
with RichLive(group, refresh_per_second=8, transient=False) as live:
@@ -1143,6 +1421,7 @@ def create(
if not result.success:
typer.echo(f"❌ Failed to create sandbox: {result.error}", err=True)
raise typer.Exit(1)
endpoints = (result.data or {}).get("endpoints") or []
sandboxes_for_ready = asyncio.run(core.list_sandboxes(sandbox_config.namespace))
target_ready = any(s.name == sandbox_config.name and s.ready == "True" for s in sandboxes_for_ready)
else:
@@ -1154,6 +1433,12 @@ def create(
handle_api_call(lambda: cli_ctx.client().create(sandbox_config.to_dict()))
sandboxes = handle_api_call(lambda: cli_ctx.client().list(namespace=sandbox_config.namespace))
target_ready = any(s.get("name") == sandbox_config.name and s.get("ready") == "True" for s in sandboxes)
if sandbox_config.expose_ports:
# The SDK's create() returns a proxy, so the API path cannot show a
# resolved URL here. Point at the NodePorts instead of guessing one.
typer.echo(f"🌐 Exposed ports: see the NodePorts in k7 list --name {sandbox_config.name}")
for endpoint in endpoints:
typer.echo(f"🌐 Port {endpoint['port']} exposed at {endpoint['url']}")
if target_ready:
typer.echo(f"✅ Sandbox {sandbox_config.name} created.")
typer.echo(
@@ -1221,6 +1506,7 @@ def list(
table.add_column("Image", style="green")
table.add_column("Backend", style="magenta")
table.add_column("Node", style="yellow")
table.add_column("NodePorts", style="yellow")
table.add_column("Error", style="red")
for sandbox in sandboxes:
@@ -1247,6 +1533,7 @@ def list(
sandbox.get("image", ""),
sandbox.get("backend", ""),
sandbox.get("node", ""),
",".join(str(p) for p in sandbox.get("node_ports") or []),
sandbox.get("error_message", ""),
)
@@ -1899,7 +2186,7 @@ def _build_and_import_api_image() -> bool:
# ---------------------------------------------------------------------------
# Spec 10d: ``k7 api ...`` sub-app — feature-toggle framing for the API server.
# ``k7 api ...`` sub-app — feature-toggle framing for the API server.
# The API is deployed by ``k7 install`` (gated on ``k7_api_enabled``) and lives
# on as a normal Kubernetes Deployment from then on. ``api enable/disable``
# scale that Deployment up/down for temporary off/on; ``status``/``endpoint``
@@ -1935,7 +2222,7 @@ def _api_scale_deployment(replicas: int) -> None:
Errors are loud — they exit the CLI with a non-zero status — because
"I asked to disable but nothing happened" is exactly the silent
failure mode 10d is trying to eliminate.
failure mode this is meant to eliminate.
"""
kubectl = _kubectl_cmd()
result = subprocess.run(
@@ -2145,7 +2432,7 @@ def _deprecated_get_api_endpoint():
# ---------------------------------------------------------------------------
# Spec 18h: ``k7 nodes …`` sub-app for cluster-node queries.
# ``k7 nodes …`` sub-app for cluster-node queries.
# ---------------------------------------------------------------------------
@@ -2205,7 +2492,7 @@ def nodes_storage(
_print_nodes_storage_table(data)
# Spec 10e: ``k7 snapshot …`` sub-app for VolumeSnapshot CRUD + GC.
# ``k7 snapshot …`` sub-app for VolumeSnapshot CRUD + GC.
# ---------------------------------------------------------------------------
@@ -2387,7 +2674,8 @@ def snapshot_gc(
):
"""Delete stale ``kind=fork`` snapshots older than ``--keep-fork-for``.
Pause snapshots and named snapshots are never touched.
Pause snapshots and named snapshots are never touched. Also reaps
orphan VolumeSnapshotContents whose VolumeSnapshot is already gone.
"""
cli_ctx: CliContext = ctx.obj
if cli_ctx.use_core:
+1333 -158
View File
File diff suppressed because it is too large Load Diff
+144
View File
@@ -0,0 +1,144 @@
"""User-facing Docker pins and --docker helpers.
k7d's source of truth is ``guest/docker/pins.env`` in the pinned k7d
tree. The constants below are what k7 displays and what the Kata
vehicle consumes. ``tests/unit/test_docker_pins.py`` fails loud if they
drift from the checked-in snapshot of that file.
"""
from __future__ import annotations
import os
import re
from pathlib import Path
# User-facing pins. Bump as a set with k7d's guest/docker/pins.env.
DOCKER_VERSION = "27.5.1"
COMPOSE_VERSION = "2.32.4"
BUILDX_VERSION = "0.20.0"
IPTABLES_VERSION = "1.8.11"
IPTABLES_BACKEND = "nft"
DEFAULT_DOCKER_DISK = "20Gi"
# Daemon args as displayed to users. k7d's agent adds data-root / exec-root
# / pidfile / host path under /run/k7d/docker/; overlay2 is mandatory.
DAEMON_ARGS = "--host=unix:///run/k7d/docker/docker.sock --tls=false --storage-driver=overlay2"
CLI_PATH = "/usr/local/bin/docker"
CLI_PLUGINS_DIR = "/usr/local/lib/docker/cli-plugins"
# Per-pod emptyDir staging; kql persist-bind overlays /usr, so the sandbox
# also mounts this path (not under /usr) and restages the canonical CLI.
CLI_STAGING_DIR = "/run/k7/docker-cli"
# Kata vehicle: directory socket, never /var/run or /run/k7d.
KATA_SOCKET_DIR = "/run/k7/docker"
DOCKER_HOST_URL = "unix:///run/k7/docker/docker.sock"
KATA_DAEMON_ARGS = "--host=unix:///run/k7/docker/docker.sock --tls=false --storage-driver=overlay2"
GRAPH_DEVICE_PATH = "/dev/k7docker"
VEHICLE_CONTAINER_NAME = "docker-vehicle"
KFD_DOCKER_STORAGE_CLASS = "k7-docker-lvm"
# linux/amd64 digest of docker:27.5.1-dind (Docker Hub, 2026-09).
DIND_IMAGE = "docker:27.5.1-dind@sha256:f649ef046008ca7f926a2571c32b0ac22e5c59eb61b959617f9acc2a4c638cf5"
PATH_SHARE_DIRS = ("/home", "/root", "/tmp", "/opt", "/workspace")
# Pod-template annotations the k7d shim reads.
ANN_K7D_DOCKER = "k7d.katakate.org/docker"
ANN_K7D_DOCKER_DISK = "k7d.katakate.org/docker-disk"
# Deployment bookkeeping for k7 (not a sidecar).
ANN_K7_DOCKER = "k7.katakate.org/docker"
ANN_DOCKER_PVC = "k7.katakate.org/docker-pvc-name"
KATA_FORK_GRAPH_REJECT = (
"k7 fork of a kfd --docker sandbox is not supported: the docker graph cannot be cloned (ephemeral LV)"
)
# Playbook records the installed k7d version here. Guest dockerd
# and k7-fc shipped in public k7d 0.6.0; a node whose recorded version
# is older, or that has no docker payload, fails loud.
K7D_VERSION_FILE = "/etc/k7/k7d_version"
K7D_DOCKER_MIN_VERSION = (0, 6, 0)
K7D_DOCKER_PAYLOAD_DOCKERD = (
"/usr/local/share/k7d/docker/bin/dockerd",
"/root/k7d/guest/docker/payload/bin/dockerd",
)
DOCKER_UNSUPPORTED_K7D = "this k7d has no docker service; upgrade"
_SIZE_RE = re.compile(r"^[0-9]+(?:[KMGT]i)?$")
def parse_docker_disk(raw: str) -> str:
"""Validate a ``--docker-disk`` quantity. Loud on garbage."""
value = (raw or "").strip()
if not value or not _SIZE_RE.match(value):
raise ValueError(f"invalid --docker-disk {raw!r}: use a Kubernetes quantity like 20Gi or 40Gi")
return value
def _parse_version(raw: str) -> tuple[int, int, int] | None:
text = raw.strip()
if text.lower().startswith("k7d "):
text = text[4:].strip()
if text.startswith("v"):
text = text[1:]
parts = text.split(".")
if len(parts) < 2:
return None
try:
major = int(parts[0])
minor = int(parts[1])
patch = int(parts[2].split("-")[0].split("+")[0]) if len(parts) > 2 else 0
except ValueError:
return None
return major, minor, patch
def recorded_k7d_version(version_file: str = K7D_VERSION_FILE) -> str | None:
"""Return the k7d version the playbook recorded, or None if missing."""
try:
text = Path(version_file).read_text(encoding="utf-8").strip()
except FileNotFoundError:
return None
except OSError:
return None
return text or None
def k7d_docker_payload_present() -> bool:
return any(os.path.isfile(p) for p in K7D_DOCKER_PAYLOAD_DOCKERD)
def k7d_supports_docker(version_file: str = K7D_VERSION_FILE) -> bool:
"""True when the installed k7d can run ``--docker``.
Detects via the version the playbook records (and the docker payload
the shim execs). Does not create a VM and wait for a timeout.
"""
recorded = recorded_k7d_version(version_file)
if recorded:
parsed = _parse_version(recorded)
if parsed is not None and parsed < K7D_DOCKER_MIN_VERSION:
return False
return k7d_docker_payload_present()
def pins_from_env_text(text: str) -> dict[str, str]:
"""Parse a k7d ``guest/docker/pins.env`` body into name → value."""
out: dict[str, str] = {}
for line in text.splitlines():
stripped = line.strip()
if not stripped or stripped.startswith("#") or "=" not in stripped:
continue
key, value = stripped.split("=", 1)
out[key.strip()] = value.strip()
return out
def user_facing_pins() -> dict[str, str]:
return {
"DOCKER_VERSION": DOCKER_VERSION,
"COMPOSE_VERSION": COMPOSE_VERSION,
"BUILDX_VERSION": BUILDX_VERSION,
"IPTABLES_VERSION": IPTABLES_VERSION,
"IPTABLES_BACKEND": IPTABLES_BACKEND,
}
+26 -6
View File
@@ -11,6 +11,7 @@ class SandboxConfig:
For kata-firecracker-devmapper: image is REQUIRED (the container to run)
For kata-qemu-longhorn: image is REQUIRED (no bare VM mode)
For k7d: image is REQUIRED (runs in a k7d microVM, runtimeClassName k7)
For k7d-fc: image is REQUIRED (same daemon, runtimeClassName k7-fc)
"""
name: str
@@ -18,7 +19,7 @@ class SandboxConfig:
namespace: str = "default"
runtime_class_name: str | None = None
root_disk_size: str | None = "10Gi"
backend: str | None = None # "kata-firecracker-devmapper", "kata-qemu-longhorn", or "k7d"
backend: str | None = None # "kata-firecracker-devmapper", "kata-qemu-longhorn", "k7d", or "k7d-fc"
env_file: str | None = None
egress_whitelist: list[str] | None = None
limits: dict[str, str] | None = None
@@ -26,6 +27,8 @@ class SandboxConfig:
entrypoint: list[str] | None = None
cmd: list[str] | None = None
sidecar: str | None = None # key into SIDECAR_REGISTRY, or None
docker: bool = False # first-class dockerd
docker_disk: str | None = None # graph disk size, e.g. "20Gi"; default 20Gi
# Security toggles (default off) and capabilities configuration
pod_non_root: bool = False
container_non_root: bool = False
@@ -34,7 +37,19 @@ class SandboxConfig:
# Optional explicit node placement (sets pod's node_name). Used by tests
# that need to inspect host-side state for a sandbox they just created.
node_name: str | None = None
# Note: ingress isolation is enforced by core with a hardcoded NetworkPolicy
# Ingress is denied by default and opt-in per sandbox.
# ``ingress_ports`` are TCP ports to open; ``None``/``[]`` denies everything.
# ``ingress_from`` scopes who may connect (``sandbox:<name>`` /
# ``namespace:<ns>`` / ``cidr:<cidr>``); empty with ports set means "other
# sandboxes in this namespace" — opening a port never opens it to the world.
ingress_ports: list[int] | None = None
ingress_from: list[str] | None = None
# TCP ports to publish outside the cluster through a NodePort Service.
# Each must also be in ``ingress_ports``: a public NodePort
# in front of a deny-all policy is a confusing no-op, and one in front of an
# accidentally-open policy is a breach — the user states both.
expose_ports: list[int] | None = None
volumes: list[dict[str, Any]] | None = None # hostPath/PVC entries; refused on k7d-fc
def __post_init__(self):
if self.limits is None:
@@ -68,9 +83,12 @@ class SandboxInfo:
image: str
backend: str = "unknown"
# Kubernetes node hosting the sandbox pod ("" while unscheduled). Needed
# by API/SDK clients to reason about k7d VM-op node locality (spec 18f
# issue 6): k7d pause/resume/fork must run on the sandbox's node.
# by API/SDK clients to reason about k7d VM-op node locality: k7d
# pause/resume/fork must run on the sandbox's node.
node: str = ""
# NodePorts allocated by the sandbox's `{name}-expose` Service, if any.
# Empty when the sandbox is not exposed.
node_ports: list[int] | None = None
error_message: str = ""
def to_dict(self) -> dict:
@@ -99,7 +117,7 @@ class OperationResult:
return asdict(self)
# Spec 10e: VolumeSnapshot kinds, used both for the ``k7.io/kind`` annotation
# VolumeSnapshot kinds, used both for the ``k7.io/kind`` annotation
# we stamp at creation time and for the heuristic fallback that classifies
# pre-existing snapshots by name pattern.
SNAPSHOT_KIND_PAUSE = "pause"
@@ -109,7 +127,7 @@ SNAPSHOT_KIND_NAMED = "named"
@dataclass
class SandboxConfigOverrides:
"""Optional per-call overrides for ``K7Core.restore_sandbox`` (Spec 10f).
"""Optional per-call overrides for ``K7Core.restore_sandbox``.
Every field is optional. When ``None`` the corresponding value is taken
from the snapshot's ``k7.io/source-*`` annotations (stamped at snapshot
@@ -122,6 +140,8 @@ class SandboxConfigOverrides:
backend: str | None = None
root_disk_size: str | None = None
sidecar: str | None = None
docker: bool | None = None
docker_disk: str | None = None
limits: dict[str, str] | None = None
entrypoint: list[str] | None = None
cmd: list[str] | None = None
+4
View File
@@ -22,6 +22,10 @@ class SidecarSpec:
SIDECAR_REGISTRY: dict[str, SidecarSpec] = {
# Deprecated as the user-facing Docker path (``--docker``). Nothing
# routes Docker through this entry on any
# backend. Kept so a real CRI sidecar still injects, and so
# ``k7 fork`` on k7d can still reject it loudly (test_k7d_sidecar_fork_rejected_loudly).
"docker": SidecarSpec(
image="docker:27.5-dind",
socket_mount="/var/run",
+1
View File
@@ -0,0 +1 @@
"""Ansible playbook, manifests, and install-time helpers."""
+1 -1
View File
@@ -75,7 +75,7 @@ longhorn_replicas=2
; ansible_ssh_private_key_file=/root/.ssh/id_ed25519
; longhorn_replicas=2
;
; 3-node HA with ALL backends on every node — spec 18e. ONE command:
; 3-node HA with ALL backends on every node. ONE command:
;
; k7 install -i inventory.ini --ha --k7d-version 0.2.1
;
File diff suppressed because it is too large Load Diff
+222
View File
@@ -0,0 +1,222 @@
"""Render TLS-flavored k7-api manifests and the Caddyfile.
The static ``manifests/k7-api/{deployment,service}.yaml`` stay the HTTP
NodePort (``--api-insecure-http``). When TLS is on, the playbook copies
those files and this module overwrites them with a Caddy sidecar on
``:8443``. The k7-api container and its HTTP probes are unchanged.
"""
from __future__ import annotations
import argparse
from pathlib import Path
# linux/amd64 digest for caddy:2.10.2 (Docker Hub, 2026-02-11). The
# playbook pulls CADDY_PIN; the pod spec uses CADDY_IMAGE (the tag).
# kubelet imagePullPolicy=Never does not resolve tag@digest to a
# locally imported tag (measured: ErrImageNeverPull).
CADDY_IMAGE = "caddy:2.10.2"
CADDY_DIGEST = "sha256:d8c17a862962def15cde69863a3a463f25a2664942eafd7bdbf050e9c3116b83"
CADDY_PIN = f"{CADDY_IMAGE}@{CADDY_DIGEST}"
_CADDY_CONTAINER_FILES = f""" - name: caddy
image: {CADDY_IMAGE}
imagePullPolicy: Never
securityContext:
runAsUser: 0
runAsGroup: 0
allowPrivilegeEscalation: false
ports:
- containerPort: 8443
protocol: TCP
volumeMounts:
- name: caddy-data
mountPath: /data
- name: caddyfile
mountPath: /etc/caddy/Caddyfile
subPath: Caddyfile
- name: tls-certs
mountPath: /certs
readOnly: true
readinessProbe:
tcpSocket:
port: 8443
initialDelaySeconds: 3
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 3
livenessProbe:
tcpSocket:
port: 8443
initialDelaySeconds: 10
periodSeconds: 30
timeoutSeconds: 3
failureThreshold: 3
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
"""
_CADDY_CONTAINER_ACME = f""" - name: caddy
image: {CADDY_IMAGE}
imagePullPolicy: Never
securityContext:
runAsUser: 0
runAsGroup: 0
allowPrivilegeEscalation: false
ports:
- containerPort: 8443
protocol: TCP
- containerPort: 80
hostPort: 80
protocol: TCP
volumeMounts:
- name: caddy-data
mountPath: /data
- name: caddyfile
mountPath: /etc/caddy/Caddyfile
subPath: Caddyfile
readinessProbe:
tcpSocket:
port: 8443
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 3
failureThreshold: 12
livenessProbe:
tcpSocket:
port: 8443
initialDelaySeconds: 15
periodSeconds: 30
timeoutSeconds: 3
failureThreshold: 3
resources:
requests:
cpu: 50m
memory: 64Mi
limits:
cpu: 500m
memory: 128Mi
"""
_API_TLS_HIDE_MOUNT = """ - name: tls-hide
mountPath: /etc/k7/tls
"""
_VOLUMES_FILES = """ - name: caddy-data
hostPath:
path: /etc/k7/caddy
type: DirectoryOrCreate
- name: caddyfile
hostPath:
path: /etc/k7/caddy
type: Directory
- name: tls-certs
secret:
secretName: k7-api-tls
- name: tls-hide
emptyDir: {}
"""
_VOLUMES_ACME = """ - name: caddy-data
hostPath:
path: /etc/k7/caddy
type: DirectoryOrCreate
- name: caddyfile
hostPath:
path: /etc/k7/caddy
type: Directory
- name: tls-hide
emptyDir: {}
"""
def render_deployment(http_yaml: str, *, acme: bool) -> str:
"""Insert the Caddy sidecar and hide ``/etc/k7/tls`` from the API container."""
if "name: caddy" in http_yaml:
raise ValueError("deployment.yaml already has a caddy container")
hide_at = " - name: k3s-containerd\n mountPath: /run/k3s/containerd\n"
if hide_at not in http_yaml:
raise ValueError("deployment.yaml is missing the k3s-containerd mount; cannot hide /etc/k7/tls")
yaml = http_yaml.replace(hide_at, hide_at + _API_TLS_HIDE_MOUNT, 1)
volumes_at = " volumes:\n"
if volumes_at not in yaml:
raise ValueError("deployment.yaml is missing the volumes: block")
sidecar = _CADDY_CONTAINER_ACME if acme else _CADDY_CONTAINER_FILES
yaml = yaml.replace(volumes_at, sidecar + volumes_at, 1)
extra = _VOLUMES_ACME if acme else _VOLUMES_FILES
if not yaml.endswith("\n"):
yaml += "\n"
return yaml + extra
def render_service(http_yaml: str) -> str:
if "targetPort: 8000" not in http_yaml:
raise ValueError("service.yaml is missing targetPort: 8000")
return http_yaml.replace("targetPort: 8000", "targetPort: 8443", 1)
def render_caddyfile(*, acme: bool, hostname: str = "", staging: bool = False) -> str:
if acme:
if not hostname:
raise ValueError("ACME Caddyfile requires a hostname")
global_block = "{\n https_port 8443\n http_port 80\n"
if staging:
global_block += " acme_ca https://acme-staging-v02.api.letsencrypt.org/directory\n"
global_block += "}\n"
# :80 is ACME only — Caddy's automatic HTTPS handles the challenge
# and does not reverse-proxy the API on HTTP.
return f"{global_block}\n{hostname} {{\n reverse_proxy 127.0.0.1:8000\n}}\n"
return (
"{\n"
" auto_https off\n"
"}\n"
"\n"
":8443 {\n"
" tls /certs/tls.crt /certs/tls.key\n"
" reverse_proxy 127.0.0.1:8000\n"
"}\n"
)
def write_tls_manifests(
manifest_dir: Path,
caddy_dir: Path,
*,
acme: bool,
hostname: str = "",
staging: bool = False,
) -> None:
dep = manifest_dir / "deployment.yaml"
svc = manifest_dir / "service.yaml"
dep.write_text(render_deployment(dep.read_text(), acme=acme))
svc.write_text(render_service(svc.read_text()))
caddy_dir.mkdir(parents=True, exist_ok=True)
(caddy_dir / "Caddyfile").write_text(render_caddyfile(acme=acme, hostname=hostname, staging=staging))
def main(argv: list[str] | None = None) -> int:
p = argparse.ArgumentParser(description="Render TLS k7-api manifests")
p.add_argument("command", choices=["write"])
p.add_argument("--manifest-dir", required=True)
p.add_argument("--caddy-dir", required=True)
p.add_argument("--mode", choices=["files", "acme"], required=True)
p.add_argument("--hostname", default="")
p.add_argument("--acme-staging", action="store_true")
args = p.parse_args(argv)
write_tls_manifests(
Path(args.manifest_dir),
Path(args.caddy_dir),
acme=args.mode == "acme",
hostname=args.hostname,
staging=args.acme_staging,
)
return 0
if __name__ == "__main__":
raise SystemExit(main())
+83
View File
@@ -0,0 +1,83 @@
#!/usr/bin/env bash
# Vendored from k7d utils/install-firecracker.sh.
# Pins live next to this script (not k7d's guest/fc/pins.env).
#
# Installs stock Firecracker + jailer into /usr/local/bin (NOT Kata's
# /opt/kata/bin). A sha mismatch is fatal, never a warning.
set -euo pipefail
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
PINS="${HERE}/pins.env"
BIN_DIR="/usr/local/bin"
CACHE="/var/cache/k7d-firecracker"
die() { echo "install-firecracker: $*" >&2; exit 1; }
[[ -f "${PINS}" ]] || die "missing pin file ${PINS}"
# shellcheck source=/dev/null
source "${PINS}"
for var in FIRECRACKER_VERSION FIRECRACKER_RELEASE_URL FIRECRACKER_TGZ_SHA256 \
FIRECRACKER_SHA256 JAILER_SHA256; do
[[ -n "${!var:-}" ]] || die "${PINS} does not set ${var}"
done
[[ "$(id -u)" -eq 0 ]] || die "must run as root (installs into ${BIN_DIR})"
command -v curl >/dev/null || die "curl is required"
command -v sha256sum >/dev/null || die "sha256sum is required"
command -v tar >/dev/null || die "tar is required"
FC_DST="${BIN_DIR}/firecracker-${FIRECRACKER_VERSION}"
JAILER_DST="${BIN_DIR}/jailer-${FIRECRACKER_VERSION}"
digest() { sha256sum "$1" | cut -d' ' -f1; }
if [[ -x "${FC_DST}" && -x "${JAILER_DST}" \
&& "$(digest "${FC_DST}")" == "${FIRECRACKER_SHA256}" \
&& "$(digest "${JAILER_DST}")" == "${JAILER_SHA256}" \
&& "$(readlink -f "${BIN_DIR}/firecracker" 2>/dev/null)" == "${FC_DST}" \
&& "$(readlink -f "${BIN_DIR}/jailer" 2>/dev/null)" == "${JAILER_DST}" ]]; then
echo "install-firecracker: already ${FIRECRACKER_VERSION} at ${FC_DST}"
exit 0
fi
mkdir -p "${CACHE}"
tgz="${CACHE}/firecracker-${FIRECRACKER_VERSION}-x86_64.tgz"
if [[ -f "${tgz}" && "$(digest "${tgz}")" != "${FIRECRACKER_TGZ_SHA256}" ]]; then
echo "install-firecracker: cached tarball digest is stale, refetching"
rm -f "${tgz}"
fi
if [[ ! -f "${tgz}" ]]; then
echo "install-firecracker: downloading ${FIRECRACKER_RELEASE_URL}"
curl -fsSL -o "${tgz}.part" "${FIRECRACKER_RELEASE_URL}"
mv "${tgz}.part" "${tgz}"
fi
got="$(digest "${tgz}")"
[[ "${got}" == "${FIRECRACKER_TGZ_SHA256}" ]] \
|| die "tarball sha256 ${got}, pinned ${FIRECRACKER_TGZ_SHA256}"
work="$(mktemp -d)"
trap 'rm -rf "${work}"' EXIT
rel="release-${FIRECRACKER_VERSION}-x86_64"
tar -xzf "${tgz}" -C "${work}" \
"${rel}/firecracker-${FIRECRACKER_VERSION}-x86_64" \
"${rel}/jailer-${FIRECRACKER_VERSION}-x86_64"
install_one() {
local src="$1" dst="$2" want="$3" link="$4"
local got
got="$(digest "${src}")"
[[ "${got}" == "${want}" ]] || die "$(basename "${dst}") sha256 ${got}, pinned ${want}"
install -m 755 "${src}" "${dst}.new"
mv -f "${dst}.new" "${dst}"
ln -sfn "${dst}" "${link}"
}
install_one "${work}/${rel}/firecracker-${FIRECRACKER_VERSION}-x86_64" \
"${FC_DST}" "${FIRECRACKER_SHA256}" "${BIN_DIR}/firecracker"
install_one "${work}/${rel}/jailer-${FIRECRACKER_VERSION}-x86_64" \
"${JAILER_DST}" "${JAILER_SHA256}" "${BIN_DIR}/jailer"
reported="$("${FC_DST}" --version | head -1)"
[[ "${reported}" == *"${FIRECRACKER_VERSION#v}"* ]] \
|| die "installed binary reports ${reported}, want ${FIRECRACKER_VERSION}"
echo "install-firecracker: ${FIRECRACKER_VERSION} at ${FC_DST} (${reported})"
+7
View File
@@ -0,0 +1,7 @@
# Vendored from k7d guest/fc/pins.env.
# Bump together with k7d's pin; sha mismatch is fatal.
FIRECRACKER_VERSION=v1.16.1
FIRECRACKER_RELEASE_URL=https://github.com/firecracker-microvm/firecracker/releases/download/v1.16.1/firecracker-v1.16.1-x86_64.tgz
FIRECRACKER_TGZ_SHA256=382a02a869e4d6d5cb14c40577f9545e8458021ea8b0b2d3fc10ec14d9c242e6
FIRECRACKER_SHA256=2fd0171309af7e24cf8dafc8a6f921c1434c49b5f9349bb996b7ed0a4deb8aa7
JAILER_SHA256=1f3a0c1fe86212d0001819bfe0819071c01208b3ccc9398c3b3bc1b84cf21edd
+56
View File
@@ -0,0 +1,56 @@
#!/usr/bin/env bash
# Classify a namespaced Kubernetes object's presence.
#
# Three states:
# present — kubectl get succeeded
# absent — kubectl succeeded at talking to the apiserver and the object
# is NotFound
# unknown — any other kubectl failure (timeout, 500, "unable to handle
# the request"). Retried, then exit 2. Never treat as absent.
#
# Usage: k8s-object-presence.sh <namespace> <kind> <name>
# stdout (exit 0): present | absent
# exit 2: unknown after retries
#
# K7_KUBECTL — kubectl argv prefix (default: k3s kubectl)
# K7_PRESENCE_RETRIES — attempts (default: 5)
# K7_PRESENCE_DELAY — seconds between attempts (default: 5)
set -eu
if [ "$#" -ne 3 ]; then
echo "usage: $0 <namespace> <kind> <name>" >&2
exit 2
fi
namespace=$1
kind=$2
name=$3
# Intentionally unquoted: K7_KUBECTL is an argv prefix ("k3s kubectl").
# shellcheck disable=SC2206
kubectl=(${K7_KUBECTL:-k3s kubectl})
retries=${K7_PRESENCE_RETRIES:-5}
delay=${K7_PRESENCE_DELAY:-5}
attempt=1
while [ "$attempt" -le "$retries" ]; do
set +e
out=$("${kubectl[@]}" -n "$namespace" get "$kind" "$name" 2>&1)
rc=$?
set -e
if [ "$rc" -eq 0 ]; then
echo present
exit 0
fi
if printf '%s' "$out" | grep -qiE 'notfound|not found'; then
echo absent
exit 0
fi
echo "kubectl error (attempt ${attempt}/${retries}): $out" >&2
if [ "$attempt" -eq "$retries" ]; then
echo unknown >&2
exit 2
fi
sleep "$delay"
attempt=$((attempt + 1))
done
exit 2
@@ -27,7 +27,7 @@ spec:
image: k7-api:local
imagePullPolicy: Never
# Same image as k7-api, different app: only the node-local VM ops
# and the storage endpoint (spec 18g).
# and the storage endpoint.
command: ["uvicorn", "k7.api.agent:app", "--host", "0.0.0.0", "--port", "8000"]
# Privileged: `lvs` (kfd thin-pool utilization) issues device-mapper
# ioctls and reads raw block devices under the hostPath-mounted
@@ -1,4 +1,4 @@
# Spec 18g: restrict pod-originated ingress to the k7-agent DaemonSet to
# Restrict pod-originated ingress to the k7-agent DaemonSet to
# the k7-api pod only. `host` / `remote-node` entities stay allowed so
# kubelet health probes and root CLI usage on cluster nodes keep working —
# those callers still need the shared token (/etc/k7/agent_token, root
@@ -0,0 +1,56 @@
# Sandboxes are untrusted code. Deny them the platform itself,
# in EVERY egress mode including --egress-open. Deny-only on purpose:
# adding an `egress:` allow section here would flip every sandbox to
# default-deny and break `--egress-open`.
#
# enableDefaultDeny is load-bearing. On Cilium 1.19 an `egressDeny`
# section on its own ALSO switches the selected endpoints to default-deny
# for egress (verified empirically: an --egress-open sandbox lost DNS the
# moment a TEST-NET-1-only egressDeny CCNP was applied). Setting it to
# false makes this a pure "drop these, allow everything else you would
# otherwise allow" policy.
#
# Do NOT add 10.0.0.0/8 or 10.43.0.0/16 here: pod CIDR is 10.42.0.0/16,
# service CIDR is 10.43.0.0/16, CoreDNS is 10.43.0.10 — that denies DNS
# and every ClusterIP for every sandbox. The apiserver is covered by the
# kube-apiserver entity (node IP:6443 and 10.43.0.1:443), not by CIDR.
apiVersion: cilium.io/v2
kind: CiliumClusterwideNetworkPolicy
metadata:
name: k7-sandbox-platform-deny
spec:
description: "Deny sandbox egress to node, control plane and metadata"
enableDefaultDeny:
egress: false
ingress: false
endpointSelector:
matchExpressions:
- key: katakate.org/sandbox
operator: Exists
egressDeny:
# M1: the node itself (SSH, kubelet :10250, k3s :6443, k7d/containerd
# sockets' owner), the other nodes of an HA cluster, and the apiserver
# however it is reached. CoreDNS is a normal pod, not hostNetwork, so
# DNS is unaffected.
- toEntities:
- host
- remote-node
- kube-apiserver
# Cloud metadata (169.254.169.254) and the rest of link-local. Cilium
# here runs VXLAN and does not use 169.254.1.1 as the pod gateway.
- toCIDR:
- 169.254.0.0/16
# M2: platform pods (k7-api, Longhorn managers/engines, metrics-server,
# hubble-relay, ...) by ClusterIP or pod IP. They are pod endpoints,
# not `host`, so M1 does not cover them. The kube-dns exclusion is
# load-bearing: CoreDNS (k3s labels it k8s-app=kube-dns) lives in
# kube-system and denying it silently breaks every sandbox's DNS,
# including the FQDN egress path.
- toEndpoints:
- matchExpressions:
- key: k8s:io.kubernetes.pod.namespace
operator: In
values: [kube-system, longhorn-system]
- key: k8s:k8s-app
operator: NotIn
values: [kube-dns]
@@ -5,8 +5,10 @@ metadata:
labels:
app: k7-api
rules:
# services: the `{name}-expose` NodePort. Delete always attempts this
# cleanup; a missing verb is 403, not 404, so every API delete 400s.
- apiGroups: [""]
resources: ["pods", "pods/exec", "pods/log", "secrets", "configmaps", "persistentvolumeclaims", "namespaces", "nodes"]
resources: ["pods", "pods/exec", "pods/log", "secrets", "configmaps", "persistentvolumeclaims", "namespaces", "nodes", "services"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
resources: ["deployments", "deployments/scale"]
@@ -14,7 +16,7 @@ rules:
- apiGroups: ["networking.k8s.io"]
resources: ["networkpolicies"]
verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
# FQDN egress (Spec 4a) is enforced via CiliumNetworkPolicy, so the API
# FQDN egress is enforced via CiliumNetworkPolicy, so the API
# must be able to manage them to apply/delete sandbox egress rules.
- apiGroups: ["cilium.io"]
resources: ["ciliumnetworkpolicies"]
@@ -31,8 +31,8 @@ spec:
- name: k7-api
image: k7-api:local
imagePullPolicy: Never
# Root is required for the co-located k7d VM-op fallback (spec 18f
# issue 2): `crictl` must reach the k3s containerd socket (0660
# Root is required for the co-located k7d VM-op fallback:
# `crictl` must reach the k3s containerd socket (0660
# root:root) to resolve a pod's CRI sandbox id before talking to
# the k7d daemon socket. No added capabilities / not privileged.
# k7d VM ops only work for sandboxes on THIS node (the first
+1 -1
View File
@@ -8,4 +8,4 @@ __all__ = [
"SandboxProxy",
]
__version__ = "0.2.2"
__version__ = "0.3.0"
+21 -6
View File
@@ -56,9 +56,11 @@ class SandboxProxy:
class Client:
"""K7 Python SDK Client."""
def __init__(self, endpoint: str, api_key: str, verify_ssl: bool = True):
def __init__(self, endpoint: str, api_key: str, verify_ssl: bool | str = True):
self.base_url = endpoint.rstrip("/")
self.api_key = api_key
if self.base_url.startswith("https://") and verify_ssl is False:
raise ValueError("verify_ssl=False is not allowed for https:// endpoints")
self.session = requests.Session()
self.session.headers.update({"X-API-Key": api_key})
self.session.verify = verify_ssl
@@ -70,7 +72,18 @@ class Client:
return data
def create(self, sandbox_config: dict) -> SandboxProxy:
"""Create a new sandbox and return a proxy object."""
"""Create a new sandbox and return a proxy object.
Ingress is denied unless asked for::
client.create({
"name": "web",
"image": "python:3.12-slim",
"ingress_ports": [8000],
"ingress_from": ["sandbox:client-a"], # default: same-namespace sandboxes
# "expose_ports": [8000], # NodePort outside the cluster; needs the same ingress_ports
})
"""
response = self.session.post(f"{self.base_url}/api/v1/sandboxes", json=sandbox_config)
response.raise_for_status()
@@ -180,7 +193,7 @@ class Client:
return SandboxProxy(new_name, namespace, self)
# ------------------------------------------------------------------
# Spec 10e: VolumeSnapshot CRUD + GC.
# VolumeSnapshot CRUD + GC.
# ------------------------------------------------------------------
def list_snapshots(
@@ -249,12 +262,12 @@ class Client:
overrides: dict | None = None,
keep_snapshot: bool = True,
) -> SandboxProxy:
"""Restore a brand-new sandbox from a standalone VolumeSnapshot (Spec 10f).
"""Restore a brand-new sandbox from a standalone VolumeSnapshot.
``overrides`` is a JSON-serialisable dict matching ``SandboxConfigOverrides``
on the server (keys: ``image``, ``backend``, ``root_disk_size``, ``sidecar``,
``limits``, ``entrypoint``, ``cmd``, ``before_script``). Pass at minimum
``{"image": "..."}`` if the snapshot was created before Spec 10f and
``{"image": "..."}`` if the snapshot is an old one and
therefore lacks the ``k7.io/source-image`` annotation.
Returns a ``SandboxProxy`` for the new sandbox. The server-side restore
@@ -326,12 +339,14 @@ class AsyncClient:
self,
endpoint: str,
api_key: str,
verify_ssl: bool = True,
verify_ssl: bool | str = True,
timeout: float = 30.0,
):
if httpx is None:
raise RuntimeError("httpx is required for AsyncClient. Install with `pip install httpx`.")
self.base_url = endpoint.rstrip("/")
if self.base_url.startswith("https://") and verify_ssl is False:
raise ValueError("verify_ssl=False is not allowed for https:// endpoints")
self._client = httpx.AsyncClient(
base_url=self.base_url,
headers={"X-API-Key": api_key},
+2 -1
View File
@@ -41,6 +41,7 @@ def _detect_backends() -> set[str]:
"kata-firecracker-devmapper",
"kata-qemu-longhorn",
"k7d",
"k7d-fc",
"firecracker-devmapper",
"qemu-longhorn",
):
@@ -57,7 +58,7 @@ def _detect_backends() -> set[str]:
"qemu-longhorn": "kata-qemu-longhorn",
}
backend = legacy.get(backend, backend)
if backend in ("kata-firecracker-devmapper", "kata-qemu-longhorn", "k7d"):
if backend in ("kata-firecracker-devmapper", "kata-qemu-longhorn", "k7d", "k7d-fc"):
return {backend}
except FileNotFoundError:
pass
+14 -5
View File
@@ -34,7 +34,7 @@ from pathlib import Path
import pytest
from k7.core.core import K7Core
from k7.core.core import K7Core, _is_k7d_family
from k7.core.models import SandboxConfig
pytestmark = pytest.mark.bench
@@ -165,7 +165,13 @@ async def _bench_lifecycle(core: K7Core, recorder: Recorder, backend: str, names
# --- create → Ready ---
start = time.monotonic()
result = await core.create_sandbox(
SandboxConfig(name=name, image=SANDBOX_IMAGE, namespace=namespace, backend=backend)
SandboxConfig(
name=name,
image=SANDBOX_IMAGE,
namespace=namespace,
backend=backend,
node_name=os.uname().nodename,
)
)
assert result.success, f"create failed: {result.error}"
_wait(lambda: _pod_ready(name, namespace), 300, f"{name} Ready")
@@ -204,10 +210,10 @@ async def _bench_lifecycle(core: K7Core, recorder: Recorder, backend: str, names
await _wait_exec(core, fork_name, namespace)
recorder.add(backend, "fork_to_ready", time.monotonic() - t0)
if backend == "k7d":
if _is_k7d_family(backend):
inherited = await core.exec_command(fork_name, "cat /tmp/bench-marker", namespace=namespace)
assert inherited.exit_code == 0 and "inherited" in inherited.stdout, (
"k7d fork lost the source's in-memory state"
f"{backend} fork lost the source's in-memory state"
)
await core.delete_sandbox(fork_name, namespace=namespace)
@@ -216,7 +222,7 @@ async def _bench_lifecycle(core: K7Core, recorder: Recorder, backend: str, names
t0 = time.monotonic()
pause = await core.pause_sandbox(name, namespace=namespace)
assert pause.success, f"pause failed: {pause.error}"
if backend != "k7d":
if not _is_k7d_family(backend):
_wait(lambda: _no_pods(name, namespace), 180, f"{name} pods gone")
recorder.add(backend, "pause_effective", time.monotonic() - t0)
@@ -276,6 +282,9 @@ async def test_bench_backend_lifecycle(k7_core: K7Core, test_namespace: str):
print(f" rep {rep + 1}/{REPS} backend={backend}", flush=True)
await _bench_lifecycle(k7_core, recorder, backend, test_namespace, rep)
for backend in BACKENDS:
if _is_k7d_family(backend):
print(f" skip sidecar backend={backend} (use bench_docker_perf --docker)", flush=True)
continue
print(f" sidecar backend={backend}", flush=True)
await _bench_sidecar(k7_core, recorder, backend, test_namespace, 0)
+145 -46
View File
@@ -1,34 +1,38 @@
"""Spec 10b/18e/18h: Docker workload benchmark — host vs k7-fd vs k7-ql vs k7d.
"""Docker workload benchmark — host vs k7-fd vs k7-ql vs k7d vs k7d-fc.
Why a pytest module rather than a separate bash harness:
``tests/integration/test_sidecar_docker.py`` already pioneered the pattern
of "spin up a ``docker:27.5-cli`` sandbox with ``--sidecar docker`` and run
docker commands via ``k7_core.exec_command``." This benchmark just times
that same pattern — no new infrastructure, no shelling-out to a separate
script, no fiddly Alpine-vs-bash portability problems.
``tests/integration/test_docker.py`` already pioneered the pattern
of spinning up a sandbox with Docker and running docker commands via
``k7_core.exec_command``. This benchmark times that same pattern.
k7d and k7d-fc use first-class ``--docker`` (guest agent dockerd, overlay2 on a
virtio-blk graph disk, forkable). Kata backends use the same ``--docker``
flag via a privileged docker-vehicle container and a block graph
(overlay2): Longhorn Block PVC on kql, ephemeral LVM thin LV on kfd.
Why these environments are valid:
``test_sidecar_docker.py`` has a note claiming Docker-DinD doesn't work on
the Firecracker backend. That note is wrong — confirmed by a manual probe
on the multi-node cluster: ``k7 create --backend kfd --sidecar docker`` +
``docker run hello-world`` exits 0 in ~3s. The sidecar wiring in
``core.py`` is symmetric across backends; the only per-backend branch is
where the docker daemon's ``/var/lib/docker`` lives (emptyDir on fd/k7d,
Longhorn PVC sub_path on ql), which is exactly the isolation we want:
the Firecracker backend. That note is historical — ``k7 create --docker``
now lands on the vehicle, not ``docker:27.5-dind``. The isolation we want:
- ``k7-fd`` — docker-in-VM without Longhorn in the storage path.
- ``k7-ql-r1`` — adds Longhorn r=1 (one local replica).
- ``k7-ql-r2`` — adds Longhorn r=2 (one local + one cross-node sync).
- ``k7-ql-r3`` — Longhorn r=3 (replicas on all three nodes of an HA
cluster; spec 18e — the full redundancy write-amplification cost).
- ``k7d`` — docker-in-VM on k7d (dind data on guest tmpfs / emptyDir;
no Longhorn). Needs a large ``--memory`` because layers live in RAM.
cluster — the full redundancy write-amplification cost).
- ``k7d`` — first-class ``--docker``: dockerd is an agent child, graph
on a per-sandbox virtio-blk scratch disk (overlay2, not tmpfs).
Needs enough ``--memory`` for the engine; layers live on the graph
disk. Fork of a warm engine is in-scope.
- ``k7d-fc`` — same guest docker service as ``k7d``, RuntimeClass
``k7-fc`` (stock Firecracker + jailer). Overlay2 and warm fork of
a running inner engine must match native k7d.
Marker / invocation:
This file uses ``@pytest.mark.bench`` so it doesn't run with the rest of
``-m integration``. Drive it explicitly:
K7_BENCH_ENVS=host,k7-fd,k7-ql-r1,k7-ql-r2,k7-ql-r3,k7d \\
K7_BENCH_ENVS=host,k7-fd,k7-ql-r1,k7-ql-r2,k7-ql-r3,k7d,k7d-fc \\
K7_BENCH_OUT=/tmp/bench-out \\
uv run pytest -m bench tests/integration/bench_docker_perf.py -v
@@ -49,15 +53,18 @@ import csv
import json
import os
import platform
import re
import shlex
import shutil
import subprocess
import time
import uuid
from datetime import datetime, timezone
from pathlib import Path
import pytest
from k7.core.core import K7Core
from k7.core.core import K7Core, _is_k7d_family
from k7.core.models import SandboxConfig
pytestmark = pytest.mark.bench
@@ -82,19 +89,31 @@ BENCH_DOCKERFILE = (Path(__file__).resolve().parents[2] / "bench" / "docker-perf
Operation = tuple[str, str] # (op_name, shell_command)
def _ops() -> list[Operation]:
def _build_cmd(args: str, classic: bool) -> str:
# k7d guest BuildKit lacks the payload CA bundle (CHALLENGES #13).
prefix = "DOCKER_BUILDKIT=0 " if classic else ""
return f"{prefix}docker build {args}"
def _ops(*, classic_build: bool = False) -> list[Operation]:
return [
("pull", f"docker rmi -f {BENCH_IMAGE} 2>/dev/null; docker pull {BENCH_IMAGE}"),
# The big one: apt + pip + git + 256 MB fsync.
# ``--network=host`` works around the same Docker default-bridge DNS
# flake spec 10g hit when building the API image: with default bridge
# flake seen when building the API image: with default bridge
# networking, ``apt-get update`` inside the build container intermittently
# fails with "Try again" / "failed to lookup address information". Host
# networking shares the daemon's already-working resolver — no measurement
# impact for our purposes (the storage path is the same either way).
("build_nocache", "cd /tmp/bench && docker build --network=host --no-cache -t bench:nocache -f Dockerfile ."),
(
"build_nocache",
f"cd /tmp/bench && {_build_cmd('--network=host --no-cache -t bench:nocache -f Dockerfile .', classic_build)}",
),
# Should be sub-second; non-zero would indicate a broken /var/lib/docker bind.
("build_cached", "cd /tmp/bench && docker build --network=host -t bench:nocache -f Dockerfile ."),
(
"build_cached",
f"cd /tmp/bench && {_build_cmd('--network=host -t bench:nocache -f Dockerfile .', classic_build)}",
),
# 10s busy-loop — measures kata-qemu CPU overhead (mostly noise on host).
(
"run_cpu",
@@ -294,8 +313,10 @@ def _wait_all_containers_ready(name: str, namespace: str, timeout: int = 300) ->
if r.returncode == 0:
try:
items = _json.loads(r.stdout).get("items", [])
if items:
statuses = items[0].get("status", {}).get("containerStatuses", [])
for pod in items:
if pod.get("status", {}).get("phase") != "Running":
continue
statuses = pod.get("status", {}).get("containerStatuses", [])
if statuses and all(s.get("ready") for s in statuses):
return
except (_json.JSONDecodeError, KeyError, IndexError):
@@ -304,13 +325,32 @@ def _wait_all_containers_ready(name: str, namespace: str, timeout: int = 300) ->
raise TimeoutError(f"containers not ready for {name}: {timeout}s")
async def _sandbox_run_once(k7_core: K7Core, name: str, namespace: str, command: str) -> float:
async def _sandbox_run_once(k7_core: K7Core, name: str, namespace: str, command: str, timeout: int = 1800) -> float:
"""Time a guest command. Kata CRI exec returns after the first stdout chunk
(CHALLENGES #13), so we launch with no streaming stdout and poll a done-file.
"""
token = uuid.uuid4().hex[:12]
out = f"/tmp/k7-bench-{token}.out"
ec = f"/tmp/k7-bench-{token}.ec"
launch = f"trap '' HUP; rm -f {ec} {out}; ( sh -c {shlex.quote(command)} >{out} 2>&1; echo $? >{ec} ) &"
start = _now()
r = await k7_core.exec_command(name, command, namespace=namespace)
elapsed = _now() - start
if r.exit_code != 0:
raise AssertionError(f"sandbox op failed (rc={r.exit_code}): {r.stderr[-500:]}")
return elapsed
await k7_core.exec_command(name, launch, namespace=namespace)
deadline = start + timeout
last = ""
while _now() < deadline:
result = await k7_core.exec_command(name, f"[ -f {ec} ] && echo __K7_EC:$(cat {ec})", namespace=namespace)
last = (result.stdout or "") + (f"\n{result.stderr}" if result.stderr else "")
match = re.search(r"__K7_EC:(\d+)", last)
if match:
elapsed = _now() - start
code = int(match.group(1))
body_res = await k7_core.exec_command(name, f"tail -c 800 {out}; rm -f {out} {ec}", namespace=namespace)
if code != 0:
body = (body_res.stdout or "").strip()
raise AssertionError(f"sandbox op failed (rc={code}): {body[-500:]}")
return elapsed
await asyncio.sleep(2)
raise AssertionError(f"sandbox op timed out after {timeout}s: {command!r} last={last!r}")
async def _sandbox_reset(k7_core: K7Core, name: str, namespace: str) -> None:
@@ -335,22 +375,25 @@ async def _bench_sandbox(
out_dir: Path,
longhorn_replicas: int | None = None,
limits: dict[str, str] | None = None,
*,
docker: bool = False,
) -> Path:
name = f"bench-{label.replace('_', '-')}"
cfg = SandboxConfig(
name=name,
image=SANDBOX_IMAGE,
image="ubuntu:24.04" if docker else SANDBOX_IMAGE,
namespace=namespace,
backend=backend,
sidecar="docker",
docker=docker,
sidecar=None if docker else "docker",
# The sandbox needs room for the full bench image + its build cache
# + the 2k-files / 512 MB workloads. Default 10Gi is the minimum;
# keep it explicit for reproducibility — and only meaningful when
# the root disk is a Longhorn PVC (kata-qemu-longhorn). The firecracker
# / k7d backends ignore this and the docker daemon's data lives on
# the sidecar's emptyDir (guest tmpfs on k7d — size via ``limits``).
# the root disk is a Longhorn PVC (kata-qemu-longhorn). On k7d
# ``--docker`` the graph is a virtio-blk scratch disk (default 20Gi).
root_disk_size="20Gi",
limits=limits,
node_name=platform.node(),
)
create = await k7_core.create_sandbox(cfg)
assert create.success, f"create {name}: {create.error}"
@@ -400,7 +443,7 @@ async def _bench_sandbox(
assert actual_lines >= expected_lines - 2, (
f"Dockerfile was truncated on the sandbox side: "
f"expected ~{expected_lines} lines, got {actual_lines}. "
f"This is the spec-10b 'silent build failure' regression — see commit msg."
f"This is the 'silent build failure' regression — see commit msg."
)
# Capture docker version + storage driver from inside the sandbox.
@@ -429,11 +472,11 @@ async def _bench_sandbox(
await k7_core.exec_command(name, f"docker pull {BENCH_IMAGE}", namespace=namespace)
await k7_core.exec_command(
name,
"cd /tmp/bench && docker build --network=host --no-cache -t bench:nocache -f Dockerfile .",
f"cd /tmp/bench && {_build_cmd('--network=host --no-cache -t bench:nocache -f Dockerfile .', docker)}",
namespace=namespace,
)
for op, command in _ops():
for op, command in _ops(classic_build=docker):
for i in range(1, REPS + 1):
# Same reset policy as the host leg — see _bench_host for rationale.
if op in ("pull", "build_nocache"):
@@ -453,6 +496,31 @@ async def _bench_sandbox(
_append_row(log, op, i, float("nan"))
await _sandbox_reset(k7_core, name, namespace)
if docker and _is_k7d_family(backend):
# Fork a warm engine; child must stay overlay2.
fork_name = f"{name}-fork"
t0 = time.monotonic()
fork = await k7_core.fork_sandbox(name, fork_name, namespace=namespace)
elapsed = time.monotonic() - t0
assert fork.success, f"{backend} --docker fork failed: {fork.error}"
_append_row(log, "fork_warm_engine", 1, elapsed)
try:
_wait_all_containers_ready(fork_name, namespace, timeout=180)
await _wait_docker_ready(k7_core, fork_name, namespace)
child_sd = await k7_core.exec_command(
fork_name,
"docker info 2>/dev/null | grep 'Storage Driver' | head -1",
namespace=namespace,
)
driver = child_sd.stdout.strip()
assert "overlay2" in driver.lower(), f"{backend} forked child storage driver: {driver!r}"
assert "vfs" not in driver.lower(), f"{backend} forked child fell back to vfs: {driver!r}"
with log.open("a") as f:
f.write(f"# storage_driver_child={driver}\n")
_append_row(log, "fork_to_ready", 1, time.monotonic() - t0)
finally:
await k7_core.delete_sandbox(fork_name, namespace=namespace)
finally:
await k7_core.delete_sandbox(name, namespace=namespace)
return log
@@ -621,8 +689,9 @@ async def test_bench_k7_fd(
label="k7-fd",
namespace=test_namespace,
backend="kata-firecracker-devmapper",
backend_extra={"docker_data_path": "emptyDir"},
backend_extra={"docker_data_path": "k7-docker-lvm-overlay2"},
out_dir=bench_out_dir,
docker=True,
)
bench_logs.append(log)
print(f"\n[bench] k7-fd log → {log}")
@@ -641,9 +710,10 @@ async def _bench_ql(
label=label,
namespace=namespace,
backend="kata-qemu-longhorn",
backend_extra={"longhorn_replicas": str(replicas)},
backend_extra={"longhorn_replicas": str(replicas), "docker_data_path": "longhorn-block-overlay2"},
out_dir=out_dir,
longhorn_replicas=replicas,
docker=True,
)
bench_logs.append(log)
print(f"\n[bench] {label} log → {log}")
@@ -690,13 +760,10 @@ async def test_bench_k7d(
bench_out_dir: Path,
bench_logs: list[Path],
):
"""k7d docker-in-VM — dind data on guest tmpfs (no Longhorn).
"""k7d first-class --docker — overlay2 on a virtio-blk graph disk.
3Gi guest + 4 vCPU: largest size that reliably reaches Ready today.
≥4Gi currently breaks k7d agent connect on create
(``Connection timed out`` / ``No such device``) — use 3Gi until
that is root-caused. Guest overlay scales with memory (~1.5 Gi at
3Gi), so layers + the 512 MB fsync share that budget.
Graph is no longer guest tmpfs, so the 3Gi memory ceiling that blocked
no-cache builds is gone. Fork of a warm engine is timed after run_io.
"""
_maybe_skip("k7d")
log = await _bench_sandbox(
@@ -704,9 +771,41 @@ async def test_bench_k7d(
label="k7d",
namespace=test_namespace,
backend="k7d",
backend_extra={"docker_data_path": "emptyDir/tmpfs"},
backend_extra={"docker_data_path": "virtio-blk-scratch"},
out_dir=bench_out_dir,
limits={"memory": "3Gi", "cpu": "4"},
docker=True,
)
bench_logs.append(log)
print(f"\n[bench] k7d log → {log}")
@pytest.mark.k7d
async def test_bench_k7d_fc(
k7_core: K7Core,
test_namespace: str,
bench_out_dir: Path,
bench_logs: list[Path],
):
"""k7d-fc --docker — same guest service, RuntimeClass k7-fc."""
_maybe_skip("k7d-fc")
present = subprocess.run(
["k3s", "kubectl", "get", "runtimeclass", "k7-fc"],
capture_output=True,
text=True,
check=False,
)
if present.returncode != 0:
pytest.skip("RuntimeClass k7-fc not registered")
log = await _bench_sandbox(
k7_core,
label="k7d-fc",
namespace=test_namespace,
backend="k7d-fc",
backend_extra={"docker_data_path": "virtio-blk-scratch", "runtime_class": "k7-fc"},
out_dir=bench_out_dir,
limits={"memory": "3Gi", "cpu": "4"},
docker=True,
)
bench_logs.append(log)
print(f"\n[bench] k7d-fc log → {log}")
+69 -15
View File
@@ -1,6 +1,9 @@
"""Integration test fixtures — requires a live k7 node."""
import json
import subprocess
import sys
from pathlib import Path
from uuid import uuid4
import pytest
@@ -8,17 +11,69 @@ import pytest
from k7.core.core import K7Core
_K3S = "/usr/local/bin/k3s"
_DEV_SH = Path(__file__).resolve().parents[2] / "src" / "k7" / "cli" / "dev.sh"
def _strip_finalizers(kind: str, name: str, namespace: str | None = None) -> None:
cmd = [_K3S, "kubectl", "patch", kind, name, "--type=json", "-p", '[{"op":"remove","path":"/metadata/finalizers"}]']
# Merge-patch is what actually clears CSI/Longhorn finalizers on these CRs.
cmd = [
_K3S,
"kubectl",
"patch",
kind,
name,
"--type=merge",
"-p",
'{"metadata":{"finalizers":[]}}',
]
if namespace:
cmd.extend(["-n", namespace])
subprocess.run(cmd, capture_output=True)
def _vsc_names_for_namespace(ns: str) -> list[str]:
"""List VolumeSnapshotContents that point at ``ns``.
kubectl jsonpath filters on ``volumeSnapshotRef.namespace`` silently
miss items; parse the JSON list instead.
"""
raw = subprocess.run(
[_K3S, "kubectl", "get", "volumesnapshotcontent", "-o", "json"],
capture_output=True,
text=True,
)
if raw.returncode != 0 or not raw.stdout.strip():
return []
try:
items = json.loads(raw.stdout).get("items") or []
except json.JSONDecodeError:
return []
names: list[str] = []
for item in items:
ref = (item.get("spec") or {}).get("volumeSnapshotRef") or {}
if ref.get("namespace") == ns:
name = (item.get("metadata") or {}).get("name")
if name:
names.append(name)
return names
def _force_delete_namespace(ns: str) -> None:
"""Delete a test namespace without blocking on stuck CSI/Longhorn finalizers."""
"""Delete a test namespace without blocking on stuck CSI/Longhorn finalizers.
Prefer the product GC (orphan VolumeSnapshotContents + stuck PVC
source-protection) so tests exercise the same path as the CronJob.
kubectl leftover stripping is a fallback when the API is unhappy.
"""
gc = subprocess.run(
[str(_DEV_SH), "--core", "snapshot", "gc", "-n", ns, "--keep-fork-for=0s"],
capture_output=True,
text=True,
timeout=120,
)
if gc.returncode != 0:
print(f"gc_snapshots teardown failed: {gc.stderr or gc.stdout}", file=sys.stderr)
snaps = subprocess.run(
[_K3S, "kubectl", "get", "volumesnapshot", "-n", ns, "-o", "jsonpath={.items[*].metadata.name}"],
capture_output=True,
@@ -30,24 +85,23 @@ def _force_delete_namespace(ns: str) -> None:
[_K3S, "kubectl", "delete", "volumesnapshot", snap, "-n", ns, "--ignore-not-found", "--wait=false"],
capture_output=True,
)
vscs = subprocess.run(
[
_K3S,
"kubectl",
"get",
"volumesnapshotcontent",
"-o",
f"jsonpath={{.items[?(@.spec.volumeSnapshotRef.namespace=='{ns}')].metadata.name}}",
],
capture_output=True,
text=True,
)
for vsc in vscs.stdout.split():
for vsc in _vsc_names_for_namespace(ns):
_strip_finalizers("volumesnapshotcontent", vsc)
subprocess.run(
[_K3S, "kubectl", "delete", "volumesnapshotcontent", vsc, "--ignore-not-found", "--wait=false"],
capture_output=True,
)
pvcs = subprocess.run(
[_K3S, "kubectl", "get", "pvc", "-n", ns, "-o", "jsonpath={.items[*].metadata.name}"],
capture_output=True,
text=True,
)
for pvc in pvcs.stdout.split():
_strip_finalizers("pvc", pvc, ns)
subprocess.run(
[_K3S, "kubectl", "delete", "pvc", pvc, "-n", ns, "--ignore-not-found", "--wait=false"],
capture_output=True,
)
subprocess.run(
[_K3S, "kubectl", "delete", "namespace", ns, "--ignore-not-found", "--wait=false"],
capture_output=True,
+86 -7
View File
@@ -11,12 +11,54 @@ import subprocess
import time
from pathlib import Path
import httpx
import httpx as _httpx
import pytest
pytestmark = pytest.mark.integration
class _Httpx:
"""httpx wrapper that trusts the playbook CA when the API is HTTPS."""
ConnectError = _httpx.ConnectError
def get(self, *args, **kwargs):
kwargs.setdefault("verify", _tls_verify())
return _httpx.get(*args, **kwargs)
def post(self, *args, **kwargs):
kwargs.setdefault("verify", _tls_verify())
return _httpx.post(*args, **kwargs)
def delete(self, *args, **kwargs):
kwargs.setdefault("verify", _tls_verify())
return _httpx.delete(*args, **kwargs)
def Client(self, *args, **kwargs):
kwargs.setdefault("verify", _tls_verify())
return _httpx.Client(*args, **kwargs)
httpx = _Httpx()
K7_API_KEYS_FILE = Path("/etc/k7/api_keys.json")
_API_ENDPOINT_FILE = Path("/etc/k7/api_endpoint")
_API_CA_FILE = Path("/etc/k7/tls/ca.crt")
def _tls_verify() -> bool | str:
"""httpx/SDK ``verify`` for the current install."""
endpoint = _API_ENDPOINT_FILE.read_text().strip() if _API_ENDPOINT_FILE.exists() else ""
if endpoint.startswith("http://"):
return False
if _API_CA_FILE.exists():
return str(_API_CA_FILE)
return bool(endpoint.startswith("https://"))
def _api_scheme() -> str:
verify = _tls_verify()
return "http" if verify is False else "https"
def _get_api_base_url() -> str:
@@ -29,7 +71,7 @@ def _get_api_base_url() -> str:
if result.returncode != 0 or not result.stdout.strip():
pytest.skip("k7-api service not found — run 'k7 install' or 'k7 api enable' first")
port = result.stdout.strip()
return f"http://127.0.0.1:{port}"
return f"{_api_scheme()}://127.0.0.1:{port}"
def _api_pod_ready() -> bool:
@@ -82,6 +124,19 @@ def _generate_test_api_key(*, name: str = "integration-test", namespaces: list[s
return token
def _revoke_test_api_key(name: str) -> None:
"""Remove a test key by name from the on-disk store (try/finally cleanup)."""
import os
if not K7_API_KEYS_FILE.exists():
return
keys = json.loads(K7_API_KEYS_FILE.read_text())
keys = {h: d for h, d in keys.items() if d.get("name") != name}
K7_API_KEYS_FILE.write_text(json.dumps(keys, indent=2))
os.chmod(K7_API_KEYS_FILE, 0o600)
os.chown(K7_API_KEYS_FILE, 1000, 1000)
@pytest.fixture(scope="module")
def api_base_url() -> str:
if not _api_pod_ready():
@@ -216,7 +271,7 @@ class TestApiSandboxLifecycle:
headers=api_headers,
timeout=30,
)
assert dr.status_code == 200
assert dr.status_code == 200, dr.text
# Confirm gone
gr2 = httpx.get(
@@ -307,7 +362,7 @@ def cleanup_sandbox(api_base_url: str, api_headers: dict):
class TestApiPauseResumeFork:
"""Spec 10a — pause/resume/fork via the HTTP API, multi-node cluster.
"""Pause/resume/fork via the HTTP API, multi-node cluster.
Uses kata-qemu-longhorn (only backend that supports snapshot + fork).
"""
@@ -383,7 +438,7 @@ class TestApiPauseResumeFork:
assert r.status_code == 201, r.text
_wait_for_sandbox_ready(api_base_url, api_headers, name, test_namespace)
# Spec 10c: ``snapshot`` alone is enough — pvc/snapshot_class derived server-side.
# ``snapshot`` alone is enough — pvc/snapshot_class derived server-side.
pr = httpx.post(
f"{api_base_url}/api/v1/sandboxes/{name}/pause",
json={"namespace": test_namespace, "snapshot": snap},
@@ -563,7 +618,7 @@ class TestSdkSandboxProxy:
):
from k7_sdk import Client
c = Client(endpoint=api_base_url, api_key=api_key, verify_ssl=False)
c = Client(endpoint=api_base_url, api_key=api_key, verify_ssl=_tls_verify())
name = "sdk-prf"
child = "sdk-prf-child"
cleanup_sandbox(name, test_namespace)
@@ -606,7 +661,7 @@ class TestSdkSandboxProxy:
# ---------------------------------------------------------------------------
# Spec 10h: SSRF guard + API-key namespace authorization
# SSRF guard + API-key namespace authorization
# ---------------------------------------------------------------------------
@@ -774,3 +829,27 @@ class TestApiNamespaceAuthz:
timeout=30,
)
assert dr.status_code == 200, dr.text
def test_scoped_key_nodes_storage_forbidden(self, api_base_url: str, api_headers: dict):
"""Cluster-scoped GET /nodes/storage is 403 for a scoped key; unscoped still 200."""
scoped_name = "integ-scoped-nodes-storage"
scoped = _generate_test_api_key(name=scoped_name, namespaces=["alpha"])
try:
scoped_r = httpx.get(
f"{api_base_url}/api/v1/nodes/storage",
headers={"X-API-Key": scoped},
timeout=30,
)
assert scoped_r.status_code == 403, scoped_r.text
assert "all-namespaces" in scoped_r.json()["error"]["message"]
unscoped_r = httpx.get(
f"{api_base_url}/api/v1/nodes/storage",
headers=api_headers,
timeout=120,
)
assert unscoped_r.status_code == 200, unscoped_r.text
data = unscoped_r.json()["data"]
assert isinstance(data, dict) and data, f"expected per-node map, got {data}"
finally:
_revoke_test_api_key(scoped_name)
+320
View File
@@ -0,0 +1,320 @@
"""Integration tests for the opt-in k7-api source-CIDR allowlist.
Read this before changing anything here: **the suite runs on the node**, so
its own traffic originates from the `host` entity, which the policy allows
unconditionally (it has to — that is the kubelet probe path). A test that
only curls the NodePort from the node passes no matter how broken the CIDR
list is. So the deny path is exercised from a **pod**, and the pod's
observed source IP is asserted first, so a pass means "the policy decided"
rather than "SNAT hid the client".
The pod is a plain `alpine` pod with no `katakate.org/sandbox` label: a
sandbox cannot reach `k7-api` at all, so a sandbox client would
be blocked by the wrong policy and prove nothing.
Two measured Cilium behaviours shape these tests, both verified on this
cluster:
- A `fromCIDR` peer is not evaluated for traffic whose source is
Cilium-managed. Allowlisting a pod's own /32 therefore does **not** let
it in — `test_pod_cidr_in_the_allowlist_is_still_denied` pins that down
so nobody mistakes the allowlist for an in-cluster access control.
- Node-local traffic to a NodePort is SNAT'd to the pod network even with
`externalTrafficPolicy: Local`, so the node can neither observe a real
client IP nor produce a `cidr:` drop. The genuinely-external half of the
matrix (a real client allowed by its CIDR, a real client denied by it)
can only be driven from an off-cluster host and is not expressible here.
Rollback, if a run dies between apply and teardown::
k3s kubectl -n kube-system delete ciliumnetworkpolicy k7-api-ingress
k3s kubectl -n kube-system patch svc k7-api \
-p '{"spec":{"externalTrafficPolicy":"Cluster"}}'
"""
import json
import subprocess
import time
import uuid
import pytest
from .test_api import _api_scheme, _tls_verify
from .test_cilium import requires_cilium
pytestmark = pytest.mark.integration
_K3S = "/usr/local/bin/k3s"
_POLICY = "k7-api-ingress"
_PROBE_POD = "k7-api-ingress-probe"
_PROBE_NS = "default"
# TEST-NET-2 (RFC 5737). No real client can come from here, so a policy
# carrying only this CIDR allows exactly the reserved entities.
_EXCLUDES_EVERYONE = "198.51.100.0/24"
def _kubectl(*args: str, check: bool = True, stdin: str | None = None) -> str:
result = subprocess.run(
[_K3S, "kubectl", *args],
input=stdin,
capture_output=True,
text=True,
check=check,
)
return result.stdout
def _policy_body(cidrs: list[str]) -> str:
"""The same shape the install playbook renders from --api-allow-cidr.
The `fromEntities` block is mandatory: an `ingress:` section flips the
k7-api endpoint to default-deny, and `host` is the kubelet probe path.
"""
return json.dumps(
{
"apiVersion": "cilium.io/v2",
"kind": "CiliumNetworkPolicy",
"metadata": {"name": _POLICY, "namespace": "kube-system"},
"spec": {
"endpointSelector": {"matchLabels": {"app": "k7-api"}},
"ingress": [
{"fromEntities": ["host", "remote-node", "health", "kube-apiserver"]},
{"fromCIDR": cidrs},
],
},
}
)
def _api_pod() -> dict:
pods = json.loads(_kubectl("-n", "kube-system", "get", "pod", "-l", "app=k7-api", "-o", "json"))["items"]
live = [p for p in pods if not p["metadata"].get("deletionTimestamp")]
assert len(live) == 1, f"expected exactly one live k7-api pod, got {[p['metadata']['name'] for p in live]}"
return live[0]
def _pod_ready(pod: dict) -> bool:
conditions = pod["status"].get("conditions") or []
return any(c["type"] == "Ready" and c["status"] == "True" for c in conditions)
def _restart_count(pod: dict) -> int:
return sum(c.get("restartCount", 0) for c in pod["status"].get("containerStatuses") or [])
def _nodeport() -> str:
return _kubectl("-n", "kube-system", "get", "svc", "k7-api", "-o", "jsonpath={.spec.ports[0].nodePort}").strip()
def _nodeport_health_curl() -> list[str]:
args = ["curl", "-s", "-o", "/dev/null", "-w", "%{http_code}", "--max-time", "6"]
verify = _tls_verify()
if isinstance(verify, str):
args += ["--cacert", verify]
args.append(f"{_api_scheme()}://127.0.0.1:{_nodeport()}/health")
return args
def _external_traffic_policy() -> str:
return _kubectl("-n", "kube-system", "get", "svc", "k7-api", "-o", "jsonpath={.spec.externalTrafficPolicy}").strip()
def _set_external_traffic_policy(value: str) -> None:
_kubectl(
"-n",
"kube-system",
"patch",
"svc",
"k7-api",
"-p",
json.dumps({"spec": {"externalTrafficPolicy": value}}),
)
def _probe_from_pod(url: str) -> str:
""" "OK" / "BLOCKED" for an HTTP GET from the probe pod.
A policy drop is a hang, not a refusal, hence the short timeout and the
sentinel: "connection refused" would mean the packet got through and the
listener is the problem.
"""
out = _kubectl(
"-n",
_PROBE_NS,
"exec",
_PROBE_POD,
"--",
"sh",
"-c",
f"wget -q -T 6 -O /dev/null '{url}' && echo OK || echo BLOCKED",
check=False,
)
lines = out.strip().splitlines()
return lines[-1] if lines else "NO-OUTPUT"
def _observed_source_ip(token: str) -> str | None:
"""The client IP uvicorn logged for the request carrying ``token``.
Access log lines look like::
INFO: 10.42.0.9:47118 - "GET /health?probe=<token> HTTP/1.1" 200 OK
"""
logs = _kubectl("-n", "kube-system", "logs", f"pod/{_api_pod()['metadata']['name']}", "--tail=200")
for line in logs.splitlines():
if token in line:
return line.split()[1].rsplit(":", 1)[0]
return None
@pytest.fixture(scope="module")
def probe_pod() -> str:
"""A plain pod (no sandbox label) with its own IP, and no policy of its own."""
manifest = json.dumps(
{
"apiVersion": "v1",
"kind": "Pod",
"metadata": {"name": _PROBE_POD, "namespace": _PROBE_NS, "labels": {"app": _PROBE_POD}},
"spec": {
"containers": [
{
"name": "probe",
"image": "alpine:3.20",
"command": ["sleep", "1800"],
}
],
"terminationGracePeriodSeconds": 1,
},
}
)
_kubectl("apply", "-f", "-", stdin=manifest)
try:
_kubectl("-n", _PROBE_NS, "wait", f"pod/{_PROBE_POD}", "--for=condition=Ready", "--timeout=120s")
pod_ip = _kubectl("-n", _PROBE_NS, "get", "pod", _PROBE_POD, "-o", "jsonpath={.status.podIP}").strip()
assert pod_ip, "probe pod has no IP"
yield pod_ip
finally:
_kubectl("-n", _PROBE_NS, "delete", "pod", _PROBE_POD, "--ignore-not-found", "--wait=false", check=False)
@pytest.fixture
def hardened():
"""Apply the allowlist policy + `Local`, then restore the default state.
Restoration is the documented rollback; it runs even when the
body of the test explodes.
"""
previous = _external_traffic_policy() or "Cluster"
def apply(cidrs: list[str]) -> None:
_kubectl("apply", "-f", "-", stdin=_policy_body(cidrs))
_set_external_traffic_policy("Local")
try:
yield apply
finally:
_kubectl("-n", "kube-system", "delete", "ciliumnetworkpolicy", _POLICY, "--ignore-not-found", check=False)
_set_external_traffic_policy(previous)
class TestDefaultInstall:
def test_no_policy_and_traffic_policy_is_cluster(self):
"""A default install (no --api-allow-cidr) is today's install, bit for bit."""
out = _kubectl(
"-n", "kube-system", "get", "ciliumnetworkpolicy", _POLICY, "--ignore-not-found", check=False
).strip()
assert out == "", f"{_POLICY} exists on a cluster installed without --api-allow-cidr: {out}"
assert _external_traffic_policy() == "Cluster"
def test_api_answers_the_nodeport(self):
code = subprocess.run(
_nodeport_health_curl(),
capture_output=True,
text=True,
).stdout
assert code == "200", f"k7-api /health returned {code!r} before any policy was applied"
@requires_cilium
class TestAllowlistPolicy:
def test_pod_to_api_source_ip_is_not_snatted(self, probe_pod: str):
"""Prerequisite for every deny assertion below.
If the pod's packets arrived as some other address, a later "BLOCKED"
would say nothing about the CIDR list.
"""
token = uuid.uuid4().hex
verdict = _probe_from_pod(f"http://{_api_pod()['status']['podIP']}:8000/health?probe={token}")
assert verdict == "OK", f"plain pod cannot reach k7-api with no policy applied: {verdict}"
assert _observed_source_ip(token) == probe_pod, (
f"k7-api observed {_observed_source_ip(token)!r}, not the probe pod IP {probe_pod!r} — "
"the deny tests below would be testing SNAT, not the policy"
)
def test_pod_outside_the_allowlist_is_denied(self, probe_pod: str, hardened):
hardened([_EXCLUDES_EVERYONE])
token = uuid.uuid4().hex
url = f"http://{_api_pod()['status']['podIP']}:8000/health?probe={token}"
# The policy takes a moment to reach the endpoint's BPF maps.
deadline = time.time() + 30
while time.time() < deadline:
if _probe_from_pod(url) == "BLOCKED":
break
time.sleep(3)
else:
pytest.fail("pod outside the allowlist still reaches k7-api after 30s of policy")
assert _observed_source_ip(token) is None, "a request that should have been dropped reached k7-api"
def test_pod_cidr_in_the_allowlist_is_still_denied(self, probe_pod: str, hardened):
"""Measured Cilium behaviour, asserted so it cannot be mistaken for a bug.
`fromCIDR` is not evaluated when the source is Cilium-managed, so the
allowlist is a control on *external* clients only. In-cluster callers
need `fromEndpoints` (see `agent-networkpolicy.yaml`).
"""
hardened([f"{probe_pod}/32"])
url = f"http://{_api_pod()['status']['podIP']}:8000/health"
deadline = time.time() + 30
while time.time() < deadline:
if _probe_from_pod(url) == "BLOCKED":
return
time.sleep(3)
pytest.fail(f"{probe_pod}/32 in fromCIDR let a Cilium-managed pod through — behaviour changed, update the docs")
def test_api_pod_stays_ready_under_policy(self, hardened):
"""The regression this feature is most likely to cause: dropping the
`host` entity kills the kubelet probes and crash-loops the pod."""
before = _api_pod()
hardened([_EXCLUDES_EVERYONE])
restarts = _restart_count(before)
deadline = time.time() + 65
while time.time() < deadline:
pod = _api_pod()
assert _pod_ready(pod), f"k7-api pod went NotReady under the ingress policy: {pod['status']}"
assert _restart_count(pod) == restarts, "k7-api container restarted under the ingress policy"
time.sleep(5)
def test_node_still_reaches_the_nodeport_under_policy(self, hardened):
"""The `host` entity path — deliberately allowed, and deliberately
useless as a deny test: this passes however wrong the CIDR list is."""
hardened([_EXCLUDES_EVERYONE])
code = subprocess.run(
_nodeport_health_curl(),
capture_output=True,
text=True,
).stdout
assert code == "200", f"node-local /health returned {code!r} under the ingress policy"
def test_rollback_restores_the_default_state(self, probe_pod: str, hardened):
hardened([_EXCLUDES_EVERYONE])
assert _external_traffic_policy() == "Local"
_kubectl("-n", "kube-system", "delete", "ciliumnetworkpolicy", _POLICY)
_set_external_traffic_policy("Cluster")
url = f"http://{_api_pod()['status']['podIP']}:8000/health"
deadline = time.time() + 30
while time.time() < deadline:
if _probe_from_pod(url) == "OK":
return
time.sleep(3)
pytest.fail("rollback did not restore pod access to k7-api")
+148 -3
View File
@@ -5,6 +5,7 @@ Flannel), so the file is safe to run on any k7 node.
"""
import asyncio
import os
import subprocess
import time
@@ -167,8 +168,8 @@ class TestCiliumFqdnEgress:
await k7_core.delete_sandbox("integ-fqdn-net", namespace=test_namespace)
async def test_docker_pull_through_fqdn_whitelist(self, k7_core: K7Core, test_namespace: str):
"""`docker pull` from Docker Hub through an FQDN egress whitelist
(spec 18f issue 3). Hub blobs are served from CDN hosts with nested
"""`docker pull` from Docker Hub through an FQDN egress whitelist.
Hub blobs are served from CDN hosts with nested
subdomains (production.cloudfront.docker.com) and 30-60s DNS TTLs —
this exercises the `*.` → `**.` matchPattern translation and the
dnsProxy.minTtl tuning end-to-end."""
@@ -201,7 +202,151 @@ class TestCiliumFqdnEgress:
pull = await k7_core.exec_command(name, "docker pull alpine:3.20 2>&1 | tail -3", namespace=test_namespace)
assert "Downloaded newer image" in pull.stdout or "Image is up to date" in pull.stdout, (
f"docker pull through FQDN whitelist failed — CDN egress is broken again "
f"(spec 18f issue 3): stdout={pull.stdout!r} stderr={pull.stderr!r}"
f"stdout={pull.stdout!r} stderr={pull.stderr!r}"
)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
_KUBECONFIG = "/etc/rancher/k3s/k3s.yaml"
_HUBBLE = "/usr/local/bin/hubble"
_CILIUM = "/usr/local/bin/cilium"
def _hubble_env() -> dict[str, str]:
env = os.environ.copy()
env["KUBECONFIG"] = _KUBECONFIG
return env
def _hubble_relay_ready() -> bool:
try:
result = subprocess.run(
[
_K3S,
"kubectl",
"-n",
"kube-system",
"get",
"deployment",
"hubble-relay",
"-o",
"jsonpath={.status.readyReplicas}",
],
capture_output=True,
text=True,
)
return result.returncode == 0 and result.stdout.strip() not in ("", "0")
except FileNotFoundError:
return False
requires_hubble = pytest.mark.skipif(
not _hubble_relay_ready(),
reason="Hubble relay not installed (re-run k7 install --hubble)",
)
def _start_hubble_port_forward() -> subprocess.Popen:
"""Background `cilium hubble port-forward` and wait until observe works.
Verified on the node: hubble-relay is ClusterIP :80; this command
prints "Hubble Relay is available at 127.0.0.1:4245" and `hubble
status` then reports Healthcheck (via localhost:4245): Ok.
"""
proc = subprocess.Popen(
[_CILIUM, "hubble", "port-forward"],
env=_hubble_env(),
stdout=subprocess.DEVNULL,
stderr=subprocess.PIPE,
start_new_session=True,
)
deadline = time.time() + 30
last_err = ""
while time.time() < deadline:
if proc.poll() is not None:
err = proc.stderr.read().decode() if proc.stderr else ""
raise RuntimeError(f"cilium hubble port-forward exited {proc.returncode}: {err}")
probe = subprocess.run(
[_HUBBLE, "status"],
env=_hubble_env(),
capture_output=True,
text=True,
)
if probe.returncode == 0:
return proc
last_err = probe.stderr or probe.stdout
time.sleep(1)
proc.terminate()
raise TimeoutError(f"hubble status never succeeded via port-forward: {last_err}")
def _stop_hubble_port_forward(proc: subprocess.Popen) -> None:
if proc.poll() is not None:
return
proc.terminate()
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
proc.kill()
proc.wait(timeout=5)
@requires_cilium
@requires_hubble
class TestHubbleObserveDropped:
async def test_hubble_observe_surfaces_fqdn_drop(self, k7_core: K7Core, test_namespace: str):
"""A non-whitelisted curl must show up as DROPPED in `hubble observe`."""
name = "integ-hubble-drop"
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=test_namespace,
egress_whitelist=["example.com"],
before_script="apk add --no-cache curl >/dev/null",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
pf = None
try:
await _wait_ready(k7_core, name, test_namespace)
blocked = await k7_core.exec_command(
name,
"curl -sS -o /dev/null --max-time 8 https://www.iana.org/ && echo OK || echo BLOCKED",
namespace=test_namespace,
)
assert "BLOCKED" in blocked.stdout, (
f"Expected non-whitelisted domain to be blocked, got stdout={blocked.stdout!r} "
f"stderr={blocked.stderr!r}"
)
pf = _start_hubble_port_forward()
deadline = time.time() + 45
observe_out = ""
while time.time() < deadline:
observe = subprocess.run(
[
_HUBBLE,
"observe",
"--namespace",
test_namespace,
"--verdict",
"DROPPED",
"--last",
"50",
],
env=_hubble_env(),
capture_output=True,
text=True,
)
observe_out = (observe.stdout or "") + (observe.stderr or "")
if observe.returncode == 0 and "DROPPED" in observe.stdout:
return
time.sleep(2)
raise AssertionError(
f"hubble observe --verdict DROPPED did not surface the FQDN drop in {test_namespace}: {observe_out!r}"
)
finally:
if pf is not None:
_stop_hubble_port_forward(pf)
await k7_core.delete_sandbox(name, namespace=test_namespace)
+156
View File
@@ -0,0 +1,156 @@
"""Integration tests for the cluster-wide sandbox platform deny.
The CCNP ``k7-sandbox-platform-deny`` must make the node, the control plane
and cloud metadata unreachable from a sandbox in EVERY egress mode,
including ``--egress-open``, without flipping open sandboxes to
default-deny. Skipped when the Cilium CRD is absent (Flannel clusters get
no platform isolation — see SECURITY.md).
The repo manifest is applied by the test itself so a missing policy is a
loud failure, never a silently green run.
"""
import json
import subprocess
from pathlib import Path
import pytest
from k7.core.core import K7Core
from k7.core.models import SandboxConfig
from .test_cilium import _wait_ready, requires_cilium
pytestmark = pytest.mark.integration
_K3S = "/usr/local/bin/k3s"
_MANIFEST = Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api/cilium/sandbox-platform-deny.yaml"
_POLICY = "k7-sandbox-platform-deny"
def _kubectl(*args: str) -> str:
return subprocess.run([_K3S, "kubectl", *args], check=True, capture_output=True, text=True).stdout
def _node_ipv4s() -> list[str]:
out = _kubectl("get", "nodes", "-o", "jsonpath={.items[*].status.addresses[?(@.type=='InternalIP')].address}")
ips = [ip for ip in out.split() if ":" not in ip]
assert ips, "no IPv4 node InternalIP found"
return ips
# exec_command merges stdout/stderr and always reports exit_code=0, so every
# probe silences curl and emits exactly one sentinel token.
def _probe(url: str, timeout: int = 6) -> str:
return f"curl -sk -o /dev/null --max-time {timeout} {url} >/dev/null 2>&1 && echo OK || echo BLOCKED"
@pytest.fixture(scope="module")
def platform_deny_applied() -> None:
assert _MANIFEST.is_file(), f"manifest missing: {_MANIFEST}"
_kubectl("apply", "-f", str(_MANIFEST))
spec = json.loads(_kubectl("get", "ciliumclusterwidenetworkpolicy", _POLICY, "-o", "jsonpath={.spec}"))
assert "egressDeny" in spec and "egress" not in spec and "ingress" not in spec, (
f"{_POLICY} must be deny-only: an allow section flips --egress-open sandboxes to default-deny"
)
assert spec.get("enableDefaultDeny") == {"egress": False, "ingress": False}, (
f"{_POLICY} must set enableDefaultDeny false: on Cilium 1.19 egressDeny alone enables default-deny"
)
@requires_cilium
@pytest.mark.usefixtures("platform_deny_applied")
class TestSandboxPlatformDeny:
async def _open_sandbox(self, k7_core: K7Core, namespace: str, name: str) -> None:
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=namespace,
egress_whitelist=None, # --egress-open
before_script="apk add --no-cache curl >/dev/null",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
await _wait_ready(k7_core, name, namespace)
async def _expect(self, k7_core: K7Core, name: str, namespace: str, cmd: str, sentinel: str, what: str) -> None:
result = await k7_core.exec_command(name, cmd, namespace=namespace)
assert sentinel in result.stdout, f"{what}: expected {sentinel}, got stdout={result.stdout!r}"
async def test_open_sandbox_still_reaches_internet(self, k7_core: K7Core, test_namespace: str):
"""egressDeny + enableDefaultDeny=false must not flip an --egress-open
sandbox to default-deny (the semantics check, encoded)."""
name = "integ-pd-open"
await self._open_sandbox(k7_core, test_namespace, name)
try:
await self._expect(
k7_core, name, test_namespace, _probe("https://example.com/", 15), "OK", "--egress-open → example.com"
)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_open_sandbox_cannot_reach_platform(self, k7_core: K7Core, test_namespace: str):
name = "integ-pd-plat"
await self._open_sandbox(k7_core, test_namespace, name)
try:
api_svc = _kubectl("get", "svc", "k7-api", "-n", "kube-system", "-o", "jsonpath={.spec.clusterIP}")
api_pod = _kubectl(
"get", "pod", "-n", "kube-system", "-l", "app=k7-api", "-o", "jsonpath={.items[0].status.podIP}"
)
lh_svc = _kubectl(
"get", "svc", "longhorn-backend", "-n", "longhorn-system", "-o", "jsonpath={.spec.clusterIP}"
)
checks = {
"apiserver ClusterIP 10.43.0.1:443": _probe("https://10.43.0.1/"),
"metadata 169.254.169.254": _probe("http://169.254.169.254/", 4),
# M2: platform pods by ClusterIP and by pod IP
f"k7-api ClusterIP {api_svc}:8000": _probe(f"http://{api_svc}:8000/health"),
f"k7-api pod {api_pod}:8000": _probe(f"http://{api_pod}:8000/health"),
f"longhorn-backend {lh_svc}:9500": _probe(f"http://{lh_svc}:9500/v1"),
}
# Every node: the sandbox's own node is the `host` entity, the
# others are `remote-node` (multi-node clusters).
for ip in _node_ipv4s():
checks[f"node {ip}:6443"] = _probe(f"https://{ip}:6443/")
checks[f"node {ip}:22"] = f"nc -z -w 4 {ip} 22 >/dev/null 2>&1 && echo OK || echo BLOCKED"
for what, cmd in checks.items():
await self._expect(k7_core, name, test_namespace, cmd, "BLOCKED", what)
# Sanity: the deny is surgical — CoreDNS (in kube-system, the M2
# namespace) still answers, both via ClusterIP and pod IP.
dns_pod = _kubectl(
"get", "pod", "-n", "kube-system", "-l", "k8s-app=kube-dns", "-o", "jsonpath={.items[0].status.podIP}"
)
for what, server in (("CoreDNS ClusterIP", "10.43.0.10"), ("CoreDNS pod", dns_pod)):
await self._expect(
k7_core,
name,
test_namespace,
f"nslookup kubernetes.default.svc.cluster.local {server} >/dev/null 2>&1 && echo OK || echo BLOCKED",
"OK",
f"{what} {server}:53",
)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_fqdn_sandbox_unaffected(self, k7_core: K7Core, test_namespace: str):
"""The deny must compose with the per-sandbox FQDN allow: whitelisted
domain reachable, everything else (incl. the node) still blocked."""
name = "integ-pd-fqdn"
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=test_namespace,
egress_whitelist=["example.com"],
before_script="apk add --no-cache curl >/dev/null",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
await _wait_ready(k7_core, name, test_namespace)
await self._expect(k7_core, name, test_namespace, _probe("https://example.com/", 15), "OK", "whitelisted")
await self._expect(
k7_core, name, test_namespace, _probe("https://www.iana.org/"), "BLOCKED", "non-whitelisted"
)
await self._expect(k7_core, name, test_namespace, _probe("https://10.43.0.1/"), "BLOCKED", "apiserver")
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
+2 -2
View File
@@ -1,6 +1,6 @@
"""Cross-node Longhorn replication integration tests (spec 3c).
"""Cross-node Longhorn replication integration tests.
These tests require a multi-node k7 cluster with topology affinity (specs 3a + 3b)
These tests require a multi-node k7 cluster with topology affinity
and Longhorn replicaCount >= 2. They are skipped on single-node setups.
"""
+671
View File
@@ -0,0 +1,671 @@
"""Integration tests for first-class ``--docker``.
k7d runs dockerd as a guest agent service (no sidecar, no privileged).
Kata backends use a privileged docker-vehicle container + block graph.
"""
from __future__ import annotations
import asyncio
import json
import os
import re
import shlex
import subprocess
import time
import uuid
from pathlib import Path
import pytest
from k7.core.core import K7Core
from k7.core.docker import VEHICLE_CONTAINER_NAME
from k7.core.models import SandboxConfig
pytestmark = pytest.mark.integration
_K7D_DISKS = Path("/var/lib/k7d/disks")
def _wait_all_containers_ready(sandbox_name: str, namespace: str, timeout: int = 300) -> None:
deadline = time.time() + timeout
while time.time() < deadline:
result = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pods",
"-n",
namespace,
"-l",
f"app={sandbox_name}",
"-o",
"json",
],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
time.sleep(2)
continue
try:
data = json.loads(result.stdout)
items = data.get("items", [])
for pod in items:
if pod.get("status", {}).get("phase") != "Running":
continue
statuses = pod.get("status", {}).get("containerStatuses", [])
if statuses and all(s.get("ready") for s in statuses):
return
except (json.JSONDecodeError, KeyError, IndexError):
pass
time.sleep(2)
raise TimeoutError(f"Not all containers ready for {sandbox_name} within {timeout}s")
# Classic builder uses dockerd's pull path (CA bundle in the payload).
# Default BuildKit metadata fetch does not (CHALLENGES #13).
_DOCKER_BUILD = "DOCKER_BUILDKIT=0 docker build"
_LOCAL_NODE = os.uname().nodename
async def _wait_docker_ready(k7_core: K7Core, name: str, namespace: str, timeout: int = 120) -> None:
deadline = time.time() + timeout
while time.time() < deadline:
result = await k7_core.exec_command(name, "docker info >/dev/null 2>&1 && echo ok", namespace=namespace)
if "ok" in (result.stdout or ""):
return
await asyncio.sleep(3)
raise TimeoutError(f"Docker daemon not ready in {name} within {timeout}s")
async def _sh(k7_core: K7Core, name: str, namespace: str, command: str, timeout: int = 180) -> str:
"""Run a command in the sandbox and require a zero guest exit code.
Kata CRI exec returns after the first stdout chunk and kills the rest
of the script (CHALLENGES #13). Launch with no streaming stdout, then
poll a done-file over short execs.
"""
token = uuid.uuid4().hex[:12]
out = f"/tmp/k7-{token}.out"
ec = f"/tmp/k7-{token}.ec"
launch = f"trap '' HUP; rm -f {ec} {out}; ( sh -c {shlex.quote(command)} >{out} 2>&1; echo $? >{ec} ) &"
await k7_core.exec_command(name, launch, namespace=namespace)
deadline = time.time() + timeout
last = ""
while time.time() < deadline:
poll = f"[ -f {ec} ] && echo __K7_EC:$(cat {ec})"
result = await k7_core.exec_command(name, poll, namespace=namespace)
last = (result.stdout or "") + (f"\n{result.stderr}" if result.stderr else "")
match = re.search(r"__K7_EC:(\d+)", last)
if match:
code = int(match.group(1))
body_res = await k7_core.exec_command(name, f"tail -c 8000 {out}; rm -f {out} {ec}", namespace=namespace)
body = (body_res.stdout or "").strip("\n")
assert code == 0, f"{command!r} exited {code}:\n{body}"
return body
await asyncio.sleep(2)
raise AssertionError(f"timeout waiting for {command!r}: {last!r}")
async def _wait_nginx_via_docker_exec(k7_core: K7Core, name: str, namespace: str, timeout: int = 90) -> str:
"""Probe the inner nginx through ``docker exec``, not CRI localhost.
``docker run -p 8080:80`` publishes in the guest host netns (k7d
agent's wget). ``k7 exec`` is the CRI container netns.
"""
deadline = time.time() + timeout
last = ""
while time.time() < deadline:
result = await k7_core.exec_command(
name,
"docker exec web wget -qO- http://127.0.0.1:80 2>&1 | head -c 200",
namespace=namespace,
)
last = result.stdout or ""
if "nginx" in last.lower() or "html" in last.lower() or "Welcome" in last:
return last
await asyncio.sleep(2)
raise TimeoutError(f"nginx did not serve in {name} within {timeout}s: {last!r}")
def _assert_overlay2(docker_info: str) -> None:
"""Overlay2 is the graph driver; vfs on that line is a ready failure."""
assert "Storage Driver: overlay2" in docker_info, docker_info
driver_line = docker_info.split("Storage Driver:")[-1].splitlines()[0].lower()
assert "vfs" not in driver_line, f"storage driver line is vfs-shaped: {driver_line!r}"
def _k7d_docker_disk_names() -> set[str]:
if not _K7D_DISKS.is_dir():
return set()
return {p.name for p in _K7D_DISKS.iterdir() if "docker" in p.name}
def _pod_annotations(name: str, namespace: str) -> dict[str, str]:
result = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pods",
"-n",
namespace,
"-l",
f"app={name}",
"-o",
"json",
],
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 0, result.stderr
items = json.loads(result.stdout).get("items", [])
assert items, f"no pod for {name}"
return items[0].get("metadata", {}).get("annotations") or {}
# ===========================================================================
# k7d — first-class --docker
# ===========================================================================
@pytest.mark.k7d
class TestDockerK7d:
backend = "k7d"
def _name(self, stem: str) -> str:
return stem if self.backend == "k7d" else f"{stem}-fc"
async def test_docker_engine_cli_path_sharing_not_privileged(self, k7_core: K7Core, test_namespace: str):
"""Overlay2, hello-world, compose/buildx, -v, CapBnd, graph disk."""
name = self._name("docker-engine-k7d")
before_disks = _k7d_docker_disk_names()
cfg = SandboxConfig(
name=name,
image="ubuntu:24.04",
namespace=test_namespace,
backend=self.backend,
docker=True,
node_name=_LOCAL_NODE,
limits={"memory": "3Gi", "cpu": "2"},
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
ann = _pod_annotations(name, test_namespace)
assert ann.get("k7d.katakate.org/docker") == "true"
assert "k7.katakate.org/sidecar" not in ann
if self.backend == "k7d-fc":
pod = _pod_json(name, test_namespace)
assert pod.get("spec", {}).get("runtimeClassName") == "k7-fc"
info = await k7_core.exec_command(name, "docker info", namespace=test_namespace)
assert info.exit_code == 0, info.stderr
_assert_overlay2(info.stdout)
during_disks = _k7d_docker_disk_names() - before_disks
assert during_disks, "k7d docker graph disk missing from /var/lib/k7d/disks (graph would be tmpfs)"
hello = await k7_core.exec_command(name, "docker run --rm hello-world", namespace=test_namespace)
assert hello.exit_code == 0, hello.stderr
assert "Hello from Docker!" in hello.stdout
compose = await k7_core.exec_command(name, "docker compose version", namespace=test_namespace)
assert compose.exit_code == 0, compose.stderr
buildx = await k7_core.exec_command(name, "docker buildx version", namespace=test_namespace)
assert buildx.exit_code == 0, buildx.stderr
await k7_core.exec_command(name, "mkdir -p /tmp/x && echo f > /tmp/x/f", namespace=test_namespace)
share = await k7_core.exec_command(
name, "docker run --rm -v /tmp/x:/x alpine:3.21 ls /x", namespace=test_namespace
)
assert share.exit_code == 0, share.stderr
assert "f" in share.stdout
cap = await k7_core.exec_command(name, "grep '^CapBnd:' /proc/self/status", namespace=test_namespace)
assert cap.exit_code == 0, cap.stderr
capbnd = cap.stdout.split()[-1].lower()
assert capbnd != "000001ffffffffff", f"sandbox CapBnd is full: {capbnd}"
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
await asyncio.sleep(3)
leftover = _k7d_docker_disk_names() - before_disks
assert not leftover, f"leaked k7d docker volume images: {leftover}"
async def test_docker_build_and_compose(self, k7_core: K7Core, test_namespace: str):
"""docker build + compose of a one-service fixture."""
name = self._name("docker-build-k7d")
cfg = SandboxConfig(
name=name,
image="ubuntu:24.04",
namespace=test_namespace,
backend=self.backend,
docker=True,
node_name=_LOCAL_NODE,
limits={"memory": "3Gi", "cpu": "2"},
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
await _sh(
k7_core,
name,
test_namespace,
"mkdir -p /tmp/app && "
"printf 'FROM alpine:3.21\\nCMD echo built-ok\\n' > /tmp/app/Dockerfile && "
"printf 'services:\\n app:\\n image: myapp\\n command: echo compose-ok\\n' "
"> /tmp/app/compose.yaml",
)
build = await _sh(
k7_core,
name,
test_namespace,
f"cd /tmp/app && {_DOCKER_BUILD} -t myapp .",
)
assert (
"Successfully tagged" in build
or "Successfully built" in build
or "naming to docker.io/library/myapp" in build
), build
run = await _sh(k7_core, name, test_namespace, "docker run --rm myapp")
assert "built-ok" in run
compose = await _sh(
k7_core,
name,
test_namespace,
"cd /tmp/app && DOCKER_BUILDKIT=0 docker compose run --rm app",
)
assert "compose-ok" in compose
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_docker_fork_keeps_running_inner_container(self, k7_core: K7Core, test_namespace: str):
"""k7 fork of --docker keeps a running inner container; not a sidecar reject."""
source = self._name("docker-fork-src")
fork_name = self._name("docker-fork-dst")
cfg = SandboxConfig(
name=source,
image="ubuntu:24.04",
namespace=test_namespace,
backend=self.backend,
docker=True,
node_name=os.uname().nodename,
limits={"memory": "3Gi", "cpu": "2"},
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(source, test_namespace)
await _wait_docker_ready(k7_core, source, test_namespace)
await _sh(
k7_core,
source,
test_namespace,
"docker run -d --name web -p 8080:80 nginx:1.27-alpine",
)
await _wait_nginx_via_docker_exec(k7_core, source, test_namespace)
fork_result = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace)
assert fork_result.success, f"fork failed (must not trip sidecar reject): {fork_result.error}"
assert "sidecar" not in (fork_result.error or "").lower()
_wait_all_containers_ready(fork_name, test_namespace, timeout=180)
await _wait_docker_ready(k7_core, fork_name, test_namespace)
child_info = await k7_core.exec_command(fork_name, "docker info", namespace=test_namespace)
assert child_info.exit_code == 0, child_info.stderr
_assert_overlay2(child_info.stdout)
ps = await _sh(
k7_core,
fork_name,
test_namespace,
"docker ps --format '{{.Names}}'",
)
assert "web" in ps
child = await _wait_nginx_via_docker_exec(k7_core, fork_name, test_namespace)
assert "Welcome to nginx" in child or "nginx" in child.lower() or "html" in child.lower()
parent = await _wait_nginx_via_docker_exec(k7_core, source, test_namespace)
assert "nginx" in parent.lower() or "html" in parent.lower() or "Welcome" in parent
finally:
await k7_core.delete_sandbox(fork_name, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
@pytest.mark.k7d
class TestDockerK7dFc(TestDockerK7d):
"""The full --docker suite on RuntimeClass k7-fc."""
backend = "k7d-fc"
@pytest.fixture(autouse=True)
def _require_k7_fc(self):
result = subprocess.run(
["k3s", "kubectl", "get", "runtimeclass", "k7-fc"],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
pytest.skip("RuntimeClass k7-fc not registered")
def _pod_json(name: str, namespace: str) -> dict:
result = subprocess.run(
["k3s", "kubectl", "get", "pods", "-n", namespace, "-l", f"app={name}", "-o", "json"],
capture_output=True,
text=True,
check=False,
)
assert result.returncode == 0, result.stderr
items = json.loads(result.stdout).get("items", [])
assert items, f"no pod for {name}"
return items[0]
def _sandbox_security_context(name: str, namespace: str) -> dict:
pod = _pod_json(name, namespace)
sandbox = next(c for c in pod["spec"]["containers"] if c["name"] == "sandbox")
return sandbox.get("securityContext") or {}
def _privileged_container_names(name: str, namespace: str) -> list[str]:
pod = _pod_json(name, namespace)
names = []
for c in pod["spec"]["containers"]:
if (c.get("securityContext") or {}).get("privileged"):
names.append(c["name"])
return names
async def _assert_overlay2_engine_cli_share(k7_core: K7Core, name: str, namespace: str) -> None:
info = await _sh(k7_core, name, namespace, "docker info -f '{{.Driver}} {{.DriverStatus}}'")
assert "overlay2" in info, info
assert "vfs" not in info.lower(), f"vfs storage driver: {info}"
hello = await _sh(k7_core, name, namespace, "docker run --rm hello-world")
assert "Hello from Docker!" in hello
compose = await _sh(k7_core, name, namespace, "docker compose version")
assert "v" in compose.lower() or "compose" in compose.lower()
buildx = await _sh(k7_core, name, namespace, "docker buildx version")
assert "buildx" in buildx.lower() or "github.com/docker/buildx" in buildx.lower()
await _sh(k7_core, name, namespace, "mkdir -p /tmp/x && echo f > /tmp/x/f")
share = await _sh(k7_core, name, namespace, "docker run --rm -v /tmp/x:/x alpine:3.21 ls /x")
assert "f" in share
async def _assert_build_and_compose(k7_core: K7Core, name: str, namespace: str) -> None:
await _sh(
k7_core,
name,
namespace,
"mkdir -p /tmp/app && "
"printf 'FROM alpine:3.21\\nCMD echo built-ok\\n' > /tmp/app/Dockerfile && "
"printf 'services:\\n app:\\n image: myapp\\n command: echo compose-ok\\n' "
"> /tmp/app/compose.yaml",
)
build = await _sh(k7_core, name, namespace, f"cd /tmp/app && {_DOCKER_BUILD} -t myapp .")
assert (
"Successfully tagged" in build or "Successfully built" in build or "naming to docker.io/library/myapp" in build
), build
run = await _sh(k7_core, name, namespace, "docker run --rm myapp")
assert "built-ok" in run
compose = await _sh(
k7_core,
name,
namespace,
"cd /tmp/app && DOCKER_BUILDKIT=0 docker compose run --rm app",
)
assert "compose-ok" in compose
def _remaining_pvcs(namespace: str, prefix: str) -> list[str]:
pvc_check = subprocess.run(
["k3s", "kubectl", "get", "pvc", "-n", namespace, "-o", "jsonpath={.items[*].metadata.name}"],
capture_output=True,
text=True,
check=False,
)
return [p for p in pvc_check.stdout.strip().split() if p.startswith(prefix)]
def _remaining_configmaps(namespace: str, prefix: str) -> list[str]:
check = subprocess.run(
["k3s", "kubectl", "get", "cm", "-n", namespace, "-o", "jsonpath={.items[*].metadata.name}"],
capture_output=True,
text=True,
check=False,
)
return [p for p in check.stdout.strip().split() if p.startswith(prefix)]
class _DockerKataMixin:
backend: str
limits = {"memory": "3Gi", "cpu": "2"}
def _cfg(self, name: str, namespace: str, **kwargs) -> SandboxConfig:
return SandboxConfig(
name=name,
image="ubuntu:24.04",
namespace=namespace,
backend=self.backend,
docker=True,
limits=dict(self.limits),
**kwargs,
)
async def test_docker_engine_cli_path_sharing(self, k7_core: K7Core, test_namespace: str):
"""Overlay2, hello-world, compose/buildx, -v."""
name = f"docker-engine-{self.backend.split('-')[1][:2]}"
result = await k7_core.create_sandbox(self._cfg(name, test_namespace))
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
ann = _pod_annotations(name, test_namespace)
assert ann.get("k7.katakate.org/docker") == "true"
assert "k7.katakate.org/sidecar" not in ann
await _assert_overlay2_engine_cli_share(k7_core, name, test_namespace)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_docker_build_and_compose(self, k7_core: K7Core, test_namespace: str):
"""docker build + compose of a one-service fixture."""
name = f"docker-build-{self.backend.split('-')[1][:2]}"
result = await k7_core.create_sandbox(self._cfg(name, test_namespace))
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
await _assert_build_and_compose(k7_core, name, test_namespace)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_sandbox_security_context_unchanged(self, k7_core: K7Core, test_namespace: str):
"""--docker does not change the sandbox security context."""
docker_name = f"docker-sec-{self.backend.split('-')[1][:2]}"
plain_name = f"plain-sec-{self.backend.split('-')[1][:2]}"
docker_cfg = self._cfg(docker_name, test_namespace)
plain = SandboxConfig(
name=plain_name,
image="ubuntu:24.04",
namespace=test_namespace,
backend=self.backend,
docker=False,
limits=dict(self.limits),
)
docker_res = await k7_core.create_sandbox(docker_cfg)
assert docker_res.success, docker_res.error
try:
plain_res = await k7_core.create_sandbox(plain)
assert plain_res.success, plain_res.error
try:
_wait_all_containers_ready(docker_name, test_namespace)
_wait_all_containers_ready(plain_name, test_namespace)
assert _sandbox_security_context(docker_name, test_namespace) == _sandbox_security_context(
plain_name, test_namespace
)
priv = _privileged_container_names(docker_name, test_namespace)
if self.backend == "kata-qemu-longhorn":
assert "sandbox" in priv
assert VEHICLE_CONTAINER_NAME in priv
assert priv.count(VEHICLE_CONTAINER_NAME) == 1
else:
assert priv == [VEHICLE_CONTAINER_NAME]
finally:
await k7_core.delete_sandbox(plain_name, namespace=test_namespace)
finally:
await k7_core.delete_sandbox(docker_name, namespace=test_namespace)
async def test_delete_leaves_no_docker_resources(self, k7_core: K7Core, test_namespace: str):
"""Delete leaves no docker resources behind."""
name = f"docker-clean-{self.backend.split('-')[1][:2]}"
result = await k7_core.create_sandbox(self._cfg(name, test_namespace))
assert result.success, result.error
_wait_all_containers_ready(name, test_namespace)
await k7_core.delete_sandbox(name, namespace=test_namespace)
await asyncio.sleep(5)
leftover_pvc = _remaining_pvcs(test_namespace, name)
leftover_cm = _remaining_configmaps(test_namespace, name)
assert not leftover_pvc, f"leaked PVCs: {leftover_pvc}"
assert not leftover_cm, f"leaked ConfigMaps: {leftover_cm}"
@pytest.mark.qemu
class TestDockerKQL(_DockerKataMixin):
backend = "kata-qemu-longhorn"
async def test_pause_resume_keeps_image(self, k7_core: K7Core, test_namespace: str):
"""Pause/resume keeps the pulled image."""
name = "docker-pr-ql"
result = await k7_core.create_sandbox(self._cfg(name, test_namespace))
assert result.success, result.error
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
await _sh(k7_core, name, test_namespace, "docker pull alpine:3.21")
pause = await k7_core.pause_sandbox(name, namespace=test_namespace)
assert pause.success, pause.error
await asyncio.sleep(5)
resume = await k7_core.resume_sandbox(name, namespace=test_namespace)
assert resume.success, resume.error
_wait_all_containers_ready(name, test_namespace, timeout=180)
await _wait_docker_ready(k7_core, name, test_namespace)
has = await _sh(k7_core, name, test_namespace, "docker image inspect alpine:3.21 >/dev/null && echo yes")
assert "yes" in has
run = await _sh(k7_core, name, test_namespace, "docker run --rm alpine:3.21 echo survived-pause")
assert "survived-pause" in run
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_fork_clones_both_pvcs(self, k7_core: K7Core, test_namespace: str):
"""Fork clones both the root and the docker graph PVC."""
source = "docker-fork-src-ql"
fork_name = "docker-fork-dst-ql"
result = await k7_core.create_sandbox(self._cfg(source, test_namespace))
assert result.success, result.error
try:
_wait_all_containers_ready(source, test_namespace)
await _wait_docker_ready(k7_core, source, test_namespace)
await _sh(k7_core, source, test_namespace, "docker pull alpine:3.21")
fork = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace)
assert fork.success, fork.error
_wait_all_containers_ready(fork_name, test_namespace, timeout=240)
await _wait_docker_ready(k7_core, fork_name, test_namespace)
has = await _sh(
k7_core, fork_name, test_namespace, "docker image inspect alpine:3.21 >/dev/null && echo yes"
)
assert "yes" in has
run = await _sh(k7_core, fork_name, test_namespace, "docker run --rm alpine:3.21 echo fork-works")
assert "fork-works" in run
parent = await _sh(k7_core, source, test_namespace, "docker run --rm alpine:3.21 echo parent-ok")
assert "parent-ok" in parent
pvcs = _remaining_pvcs(test_namespace, "")
assert f"{source}-docker-lh" in pvcs
assert f"{fork_name}-docker-lh" in pvcs
assert f"{source}-root-lh" in pvcs
assert f"{fork_name}-root-lh" in pvcs
finally:
await k7_core.delete_sandbox(fork_name, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_snapshot_restore_keeps_image(self, k7_core: K7Core, test_namespace: str):
"""Snapshot + restore keeps the pulled image."""
source = "docker-snap-src-ql"
restored = "docker-snap-dst-ql"
snap = "docker-snap-ql"
result = await k7_core.create_sandbox(self._cfg(source, test_namespace))
assert result.success, result.error
try:
_wait_all_containers_ready(source, test_namespace)
await _wait_docker_ready(k7_core, source, test_namespace)
await _sh(k7_core, source, test_namespace, "docker pull alpine:3.21")
snap_res = await k7_core.create_snapshot(source, snap, namespace=test_namespace)
assert snap_res.success, snap_res.error
restore = await k7_core.restore_sandbox(snap, restored, namespace=test_namespace)
assert restore.success, restore.error
_wait_all_containers_ready(restored, test_namespace, timeout=240)
await _wait_docker_ready(k7_core, restored, test_namespace)
has = await _sh(
k7_core, restored, test_namespace, "docker image inspect alpine:3.21 >/dev/null && echo yes"
)
assert "yes" in has
finally:
await k7_core.delete_sandbox(restored, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
await k7_core.delete_snapshot(snap, namespace=test_namespace)
@pytest.mark.firecracker
class TestDockerKFD(_DockerKataMixin):
backend = "kata-firecracker-devmapper"
async def test_graph_is_ephemeral_block_fork_rejected(self, k7_core: K7Core, test_namespace: str):
"""The graph is an ephemeral block device, so fork is rejected."""
name = "docker-ephem-kfd"
result = await k7_core.create_sandbox(self._cfg(name, test_namespace))
assert result.success, result.error
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
mnt = await k7_core.exec_command(
name,
"findmnt -no FSTYPE,SOURCE /var/lib/docker || cat /proc/mounts",
namespace=test_namespace,
container="docker-vehicle",
)
text = (mnt.stdout or "") + (mnt.stderr or "")
assert "ext4" in text.lower() or "ext4" in text
assert "virtiofs" not in text.lower()
assert "tmpfs" not in text.split("/var/lib/docker")[-1].splitlines()[0].lower()
fork = await k7_core.fork_sandbox(name, "docker-kfd-fork", namespace=test_namespace)
assert not fork.success
assert "cannot be cloned" in (fork.error or "").lower()
finally:
before = subprocess.run(["lvs", "--noheadings", "-o", "lv_name", "kata-vg"], capture_output=True, text=True)
await k7_core.delete_sandbox(name, namespace=test_namespace)
await asyncio.sleep(8)
leftover_pvc = _remaining_pvcs(test_namespace, name)
leftover_cm = _remaining_configmaps(test_namespace, name)
assert not leftover_pvc, f"leaked PVCs: {leftover_pvc}"
assert not leftover_cm, f"leaked ConfigMaps: {leftover_cm}"
after = subprocess.run(["lvs", "--noheadings", "-o", "lv_name", "kata-vg"], capture_output=True, text=True)
# thin-pool remains; docker graph LVs must be gone.
leftover_lvs = [
ln.strip()
for ln in (after.stdout or "").splitlines()
if ln.strip() and ln.strip() != "thin-pool" and name.replace("_", "-") in ln
]
assert not leftover_lvs, f"leaked LVs: {leftover_lvs} (before={before.stdout!r})"
+30 -1
View File
@@ -108,8 +108,37 @@ class TestFirecrackerBackend:
finally:
await k7_core.delete_sandbox("integ-fc", namespace=test_namespace)
async def test_guest_seccomp_enforced(self, k7_core: K7Core, test_namespace: str):
"""RuntimeDefault must actually be applied inside the Firecracker guest."""
toml_path = "/opt/kata/share/defaults/kata-containers/configuration-fc.toml"
with open(toml_path) as f:
toml = f.read()
assert "disable_guest_seccomp = false" in toml, toml
assert "disable_guest_seccomp = true" not in toml.replace("disable_guest_seccomp = false", "")
cfg = SandboxConfig(
name="integ-fc-seccomp",
image="alpine:3.20",
namespace=test_namespace,
backend="kata-firecracker-devmapper",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed after disable_guest_seccomp=false: {result.error}"
try:
_wait_for_pod_running("integ-fc-seccomp", test_namespace)
r = await k7_core.exec_command(
"integ-fc-seccomp",
"grep '^Seccomp:' /proc/self/status",
namespace=test_namespace,
)
assert r.exit_code == 0, r.stderr
assert r.stdout.split()[-1] == "2", (
f"guest Seccomp is {r.stdout!r} (want 2=filter); RuntimeDefault is not enforced"
)
finally:
await k7_core.delete_sandbox("integ-fc-seccomp", namespace=test_namespace)
async def test_memory_limit_sizes_vm(self, k7_core: K7Core, test_namespace: str):
"""Spec 18h regression: kfd must honour ``default_memory`` (not silently
"""Regression: kfd must honour ``default_memory`` (not silently
ignore it). Before the install forwarded ``pod_annotations`` on
``runtimes.kata`` and allowlisted the annotation in
configuration-fc.toml, ``--memory 3Gi`` still booted a 2048 MiB VM."""
+3 -2
View File
@@ -1,4 +1,4 @@
"""Integration tests specific to the k7d backend (spec 9a M11).
"""Integration tests specific to the k7d backend.
k7d sandboxes are pods with ``runtimeClassName: k7`` — each pod is one
k7d microVM. Fork is a **warm CoW disk+memory fork** of the live source
@@ -21,7 +21,7 @@ from k7.core.models import SandboxConfig
pytestmark = [pytest.mark.integration, pytest.mark.k7d]
# k7d VM operations (pause/resume/fork) go through the node-local
# /run/k7d/k7d.sock — cross-node ops are out of scope (k7d spec 9a M12)
# /run/k7d/k7d.sock — cross-node ops are out of scope
# and core fails loudly on a node mismatch. On a multi-node cluster the
# scheduler may place the sandbox anywhere, so tests that exercise VM
# ops pin their sandboxes to the node this pytest process runs on.
@@ -271,5 +271,6 @@ class TestK7dSidecarDocker:
fork_result = await k7_core.fork_sandbox(name, f"{name}-fork", namespace=test_namespace)
assert not fork_result.success
assert "sidecar" in fork_result.error
assert "real CRI sidecar" in fork_result.error
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
+38 -18
View File
@@ -1,8 +1,8 @@
"""k7d VM ops through the centralized HTTP API (specs 18f issue 2 + 18g).
"""k7d VM ops through the centralized HTTP API.
The k7-api pod handles sandboxes co-located with it directly (hostPath
sockets + crictl). For a sandbox on any OTHER node, core forwards the VM
op to the k7-agent DaemonSet pod on that node (spec 18g) — so
op to the k7-agent DaemonSet pod on that node — so
pause/resume/fork work through the API for any node. The agent requires
the shared token from /etc/k7/agent_token and rejects everything else.
@@ -14,10 +14,9 @@ import json
import subprocess
import time
import httpx
import pytest
from .test_api import _api_pod_ready, _generate_test_api_key, _get_api_base_url
from .test_api import _api_pod_ready, _generate_test_api_key, _get_api_base_url, _tls_verify, httpx
pytestmark = [pytest.mark.integration, pytest.mark.k7d]
@@ -32,6 +31,15 @@ def _kubectl_json(*args: str) -> dict:
return json.loads(result.stdout)
def _node_ready(node: dict) -> bool:
conds = {c["type"]: c["status"] for c in node.get("status", {}).get("conditions", [])}
return conds.get("Ready") == "True"
def _ready_node_names() -> list[str]:
return [n["metadata"]["name"] for n in _kubectl_json("get", "nodes")["items"] if _node_ready(n)]
def _k7d_runtimeclass_present() -> bool:
try:
result = subprocess.run(
@@ -57,7 +65,7 @@ def api_base_url() -> str:
url = _get_api_base_url()
for _ in range(30):
try:
if httpx.get(f"{url}/health", timeout=3).status_code == 200:
if httpx.get(f"{url}/health", timeout=3, verify=_tls_verify()).status_code == 200:
return url
except httpx.ConnectError:
pass
@@ -87,7 +95,13 @@ def api_node() -> str:
def _wait_sandbox_ready(url: str, headers: dict, name: str, namespace: str, timeout: int = 300) -> dict:
deadline = time.time() + timeout
while time.time() < deadline:
r = httpx.get(f"{url}/api/v1/sandboxes/{name}", params={"namespace": namespace}, headers=headers, timeout=10)
r = httpx.get(
f"{url}/api/v1/sandboxes/{name}",
params={"namespace": namespace},
headers=headers,
timeout=10,
verify=_tls_verify(),
)
if r.status_code == 200 and r.json()["data"]["ready"] == "True":
return r.json()["data"]
time.sleep(3)
@@ -95,7 +109,13 @@ def _wait_sandbox_ready(url: str, headers: dict, name: str, namespace: str, time
def _delete_sandbox(url: str, headers: dict, name: str, namespace: str) -> None:
httpx.delete(f"{url}/api/v1/sandboxes/{name}", params={"namespace": namespace}, headers=headers, timeout=120)
httpx.delete(
f"{url}/api/v1/sandboxes/{name}",
params={"namespace": namespace},
headers=headers,
timeout=120,
verify=_tls_verify(),
)
@requires_k7d
@@ -105,7 +125,7 @@ class TestK7dViaApi:
):
"""Full k7d VM-op round-trip over HTTP for a sandbox pinned to the
API pod's node: pause → resume → fork. Also asserts the new
``node`` field (spec 18f issue 6) so clients can see placement."""
``node`` field so clients can see placement."""
name = "api-k7d-vmops"
fork = "api-k7d-vmops-fork"
r = httpx.post(
@@ -159,14 +179,14 @@ class TestK7dViaApi:
def test_k7d_vm_ops_on_remote_node_via_agent(
self, api_base_url: str, api_headers: dict, api_node: str, test_namespace: str
):
"""Spec 18g: a k7d sandbox on a DIFFERENT node than the API pod is
"""a k7d sandbox on a DIFFERENT node than the API pod is
pausable/resumable/forkable through the API — core forwards the op
to the k7-agent on the sandbox's node. The fork must land on the
source's node (cross-node fork is k7d 9a M12, not built)."""
source's node (cross-node fork is future k7d work, not built)."""
nodes = _kubectl_json("get", "nodes")
other = [n["metadata"]["name"] for n in nodes["items"] if n["metadata"]["name"] != api_node]
other = [n["metadata"]["name"] for n in nodes["items"] if n["metadata"]["name"] != api_node and _node_ready(n)]
if not other:
pytest.skip("single-node cluster — no remote node to test against")
pytest.skip("no other Ready node — remote k7-agent path not testable")
name = "api-k7d-remote"
fork = "api-k7d-remote-fork"
@@ -237,7 +257,7 @@ def _local_agent_pod_ip(api_node: str) -> str:
@requires_k7d
class TestK7Agent:
def test_agent_rejects_unauthenticated_requests(self, api_node: str):
"""Spec 18g: the agent must reject requests without the shared token
"""the agent must reject requests without the shared token
(missing header and wrong token both yield 401)."""
base = f"http://{_local_agent_pod_ip(api_node)}:8000"
r = httpx.post(f"{base}/agent/v1/vm/pause", json={"name": "x"}, timeout=10)
@@ -260,8 +280,8 @@ class TestK7Agent:
def test_nodes_storage_aggregation(self, api_base_url: str, api_headers: dict):
"""GET /api/v1/nodes/storage returns both pools for EVERY cluster
node (spec 18g success criterion)."""
nodes = [n["metadata"]["name"] for n in _kubectl_json("get", "nodes")["items"]]
node."""
nodes = _ready_node_names()
r = httpx.get(f"{api_base_url}/api/v1/nodes/storage", headers=api_headers, timeout=120)
assert r.status_code == 200, f"nodes/storage failed: {r.text}"
data = r.json()["data"]
@@ -273,11 +293,11 @@ class TestK7Agent:
assert entry["k7d_disks"]["size_bytes"] > 0, entry
def test_nodes_storage_sdk(self, api_base_url: str, api_key: str):
"""Spec 18h: SDK ``Client.nodes_storage()`` hits the same endpoint."""
"""SDK ``Client.nodes_storage()`` hits the same endpoint."""
from k7_sdk import Client
nodes = [n["metadata"]["name"] for n in _kubectl_json("get", "nodes")["items"]]
data = Client(endpoint=api_base_url, api_key=api_key, verify_ssl=False).nodes_storage()
nodes = _ready_node_names()
data = Client(endpoint=api_base_url, api_key=api_key, verify_ssl=_tls_verify()).nodes_storage()
for node in nodes:
assert node in data, f"node {node} missing from SDK storage report: {data}"
assert "error" not in data[node], data[node]
+200
View File
@@ -0,0 +1,200 @@
"""Integration tests for the k7d-fc backend.
Same daemon socket as ``k7d``; pods use ``runtimeClassName: k7-fc``.
Skipped when RuntimeClass ``k7-fc`` is not registered.
Daemon-socket verbs: create / fork / pause / resume, plus
hostPath refused before the pod exists. CRI exec / kube Ready
work the same as native k7d (CHALLENGES #15).
"""
import asyncio
import json
import os
import subprocess
import time
import pytest
from k7.core.core import K7D_FC_VIRTIOFS_REFUSED, K7Core
from k7.core.models import SandboxConfig
pytestmark = [pytest.mark.integration, pytest.mark.k7d]
_LOCAL_NODE = os.uname().nodename
def _runtimeclass_present(name: str) -> bool:
result = subprocess.run(
["k3s", "kubectl", "get", "runtimeclass", name],
capture_output=True,
text=True,
check=False,
)
return result.returncode == 0
@pytest.fixture(scope="module", autouse=True)
def _require_k7_fc():
if not _runtimeclass_present("k7-fc"):
pytest.skip("RuntimeClass k7-fc not registered")
def _wait_pod_ready(sandbox_name: str, namespace: str, timeout: int = 300) -> None:
"""Block until the sandbox pod is Ready.
Ready is the exec readiness probe. That probe succeeds on k7-fc
the same way it does on native k7.
"""
deadline = time.time() + timeout
while time.time() < deadline:
result = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pods",
"-n",
namespace,
"-l",
f"app={sandbox_name}",
"-o",
"json",
],
capture_output=True,
text=True,
check=False,
)
if result.returncode == 0:
try:
items = json.loads(result.stdout).get("items", [])
for pod in items:
status = pod.get("status", {})
if status.get("phase") != "Running":
continue
statuses = status.get("containerStatuses") or []
if statuses and all(s.get("ready") for s in statuses):
return
except (json.JSONDecodeError, KeyError):
pass
time.sleep(2)
raise TimeoutError(f"Pod for {sandbox_name} not Ready within {timeout}s")
def _pod_runtime_class(sandbox_name: str, namespace: str) -> str:
result = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pods",
"-n",
namespace,
"-l",
f"app={sandbox_name}",
"-o",
"jsonpath={.items[0].spec.runtimeClassName}",
],
capture_output=True,
text=True,
check=True,
)
return result.stdout.strip()
class TestK7dFcLifecycle:
async def test_k7d_fc_create_pause_resume(self, k7_core: K7Core, test_namespace: str):
name = "test-k7d-fc-lifecycle"
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=test_namespace,
backend="k7d-fc",
node_name=_LOCAL_NODE,
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_pod_ready(name, test_namespace)
assert _pod_runtime_class(name, test_namespace) == "k7-fc"
pause_result = await k7_core.pause_sandbox(name, namespace=test_namespace)
assert pause_result.success, f"pause failed: {pause_result.error}"
assert "frozen" in pause_result.message
await asyncio.sleep(3)
resume_result = await k7_core.resume_sandbox(name, namespace=test_namespace)
assert resume_result.success, f"resume failed: {resume_result.error}"
finally:
delete_result = await k7_core.delete_sandbox(name, namespace=test_namespace)
assert delete_result.success, f"delete failed: {delete_result.error}"
async def test_k7d_fc_fork_inherits_state(self, k7_core: K7Core, test_namespace: str):
source = "test-k7d-fc-fork-src"
fork = "test-k7d-fc-fork-dst"
cfg = SandboxConfig(
name=source,
image="alpine:3.20",
namespace=test_namespace,
backend="k7d-fc",
node_name=_LOCAL_NODE,
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_pod_ready(source, test_namespace)
assert _pod_runtime_class(source, test_namespace) == "k7-fc"
fork_result = await k7_core.fork_sandbox(source, fork, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
assert "warm fork" in fork_result.message
_wait_pod_ready(fork, test_namespace, timeout=120)
assert _pod_runtime_class(fork, test_namespace) == "k7-fc"
finally:
await k7_core.delete_sandbox(fork, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_k7d_fc_exec_echo_hello(self, k7_core: K7Core, test_namespace: str):
"""CRI exec into a k7-fc sandbox returns; kube Ready is a valid signal."""
name = "test-k7d-fc-exec"
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=test_namespace,
backend="k7d-fc",
node_name=_LOCAL_NODE,
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_pod_ready(name, test_namespace)
assert _pod_runtime_class(name, test_namespace) == "k7-fc"
exec_result = await k7_core.exec_command(name, "echo hello", namespace=test_namespace)
assert "hello" in (exec_result.stdout or ""), exec_result.stdout
finally:
delete_result = await k7_core.delete_sandbox(name, namespace=test_namespace)
assert delete_result.success, f"delete failed: {delete_result.error}"
async def test_k7d_fc_hostpath_refused_up_front(self, k7_core: K7Core, test_namespace: str):
cfg = SandboxConfig(
name="test-k7d-fc-hostpath",
image="alpine:3.20",
namespace=test_namespace,
backend="k7d-fc",
volumes=[{"name": "hp", "hostPath": {"path": "/var/log"}}],
)
result = await k7_core.create_sandbox(cfg)
assert not result.success
assert result.error == K7D_FC_VIRTIOFS_REFUSED
listed = subprocess.run(
["k3s", "kubectl", "get", "deploy", "test-k7d-fc-hostpath", "-n", test_namespace],
capture_output=True,
text=True,
check=False,
)
assert listed.returncode != 0, "hostPath k7d-fc create must not leave a Deployment"
+7 -7
View File
@@ -1,4 +1,4 @@
"""Integration tests for ``k7 pause`` after Spec 10c.
"""Integration tests for ``k7 pause``.
These tests live alongside ``tests/integration/test_qemu.py`` (which already
covers pause/resume/fork via K7Core directly) and exercise the **CLI surface**:
@@ -31,8 +31,8 @@ def _run_cli(*args: str, check: bool = True, timeout: int = 60) -> subprocess.Co
"""Invoke ``dev.sh <args>``.
Forces the ``--core`` global flag so these CLI-flag-handling tests don't
require ``K7_API_URL`` / ``K7_API_KEY`` (Spec 10g made the default route
the API). Tests that need the API-routed path go through the SDK / HTTP
require ``K7_API_URL`` / ``K7_API_KEY`` (the default route is the
API). Tests that need the API-routed path go through the SDK / HTTP
fixtures in ``test_api.py``.
"""
cmd = [str(_DEV_SH), "--core", *args]
@@ -128,8 +128,8 @@ def ql_sandbox(k7_core: K7Core, test_namespace: str):
class TestPauseSnapshotBugRegression:
"""Spec 10c: ``k7 pause foo --snapshot=bar`` (without ``--pvc``) **must**
create a real ``VolumeSnapshot``. Pre-10c this silently did nothing.
"""``k7 pause foo --snapshot=bar`` (without ``--pvc``) **must**
create a real ``VolumeSnapshot``. This used to silently do nothing.
"""
def test_pause_with_snapshot_actually_creates_one(
@@ -146,7 +146,7 @@ class TestPauseSnapshotBugRegression:
while time.time() < deadline and not _snapshot_exists(snap, test_namespace):
time.sleep(2)
assert _snapshot_exists(snap, test_namespace), (
f"VolumeSnapshot {snap} was never created — the spec 10c bug regressed"
f"VolumeSnapshot {snap} was never created — the pause bug regressed"
)
finally:
subprocess.run(
@@ -226,7 +226,7 @@ class TestPauseSnapshotBugRegression:
class TestRemovedPauseFlags:
"""Spec 10c: ``--pvc`` and ``--snapshot-class`` were removed from ``k7 pause``."""
"""``--pvc`` and ``--snapshot-class`` were removed from ``k7 pause``."""
def test_pvc_flag_rejected(self, test_namespace: str):
result = subprocess.run(
+24 -4
View File
@@ -28,7 +28,7 @@ def _wait_pod_ready(sandbox_name: str, namespace: str, timeout: int = 300) -> No
"""Block until a non-terminating Running pod for the sandbox is Ready.
After pause→resume, a terminating pod can still report Ready=True while
the replacement is Pending; ``.items[0]`` alone races (spec 18h flake).
the replacement is Pending; ``.items[0]`` alone races (observed flake).
"""
deadline = time.time() + timeout
while time.time() < deadline:
@@ -156,8 +156,28 @@ class TestQemuLonghornBackend:
finally:
await k7_core.delete_sandbox("integ-ql-rt", namespace=test_namespace)
async def test_guest_seccomp_config_and_sandbox_starts(self, k7_core: K7Core, test_namespace: str):
"""Guest seccomp bypass must be off; persist-bind must still boot."""
toml_path = "/opt/kata/share/defaults/kata-containers/configuration-qemu.toml"
with open(toml_path) as f:
toml = f.read()
assert "disable_guest_seccomp = false" in toml, toml
assert "disable_guest_seccomp = true" not in toml.replace("disable_guest_seccomp = false", "")
cfg = SandboxConfig(
name="integ-ql-seccomp",
image="alpine:3.20",
namespace=test_namespace,
backend="kata-qemu-longhorn",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed after disable_guest_seccomp=false: {result.error}"
try:
_wait_pod_ready("integ-ql-seccomp", test_namespace)
finally:
await k7_core.delete_sandbox("integ-ql-seccomp", namespace=test_namespace)
async def test_memory_limit_sizes_vm(self, k7_core: K7Core, test_namespace: str):
"""Spec 18g regression: a kql sandbox with a memory limit stamps the
"""Regression: a kql sandbox with a memory limit stamps the
``default_memory`` hypervisor annotation. Before the install allowed
it in configuration-qemu.toml, kata rejected the WHOLE pod with
"annotation ... is not enabled" — it could never start. Assert the
@@ -344,7 +364,7 @@ class TestSnapshotPauseResumeFork:
_wait_pod_ready(fork_name, test_namespace, timeout=600)
fork_elapsed = time.time() - t0
print(f"[timing] fork total in {fork_elapsed:.2f}s")
# 600s bound (spec 18e): on a 3-node Longhorn r=3 cluster the
# 600s bound: on a 3-node Longhorn r=3 cluster the
# fork's cloned volume hydrates replicas on all nodes, and the
# very FIRST fork after a fresh install additionally pulls the
# Longhorn engine image on every node plus the sandbox image on
@@ -419,7 +439,7 @@ class TestSnapshotPauseResumeFork:
async def test_cleanup_no_leak(self, k7_core: K7Core, test_namespace: str):
"""Validates delete removes PVCs once referencing snapshots are gone.
Spec 10e keeps the root PVC while pause/named snapshots still exist, so
The root PVC is kept while pause/named snapshots still exist, so
the snapshot must be deleted before the sandbox for a full cleanup.
"""
name = "integ-leak"
+1 -1
View File
@@ -1,4 +1,4 @@
"""Integration tests for ``k7 restore`` (Spec 10f), multi-node cluster.
"""Integration tests for ``k7 restore``, multi-node cluster.
Coverage:
+541
View File
@@ -0,0 +1,541 @@
"""Integration tests for sandbox network policy.
M0 covers the fork regression: ``k7 fork`` used to produce a sandbox with no
NetworkPolicy and no CiliumNetworkPolicy at all, i.e. wide-open egress and
ingress no matter how locked down its source was.
"""
import asyncio
import ipaddress
import subprocess
import time
import pytest
from kubernetes_asyncio.client.exceptions import ApiException
from k7.core.core import K7Core
from k7.core.models import OperationResult, SandboxConfig
pytestmark = pytest.mark.integration
_K3S = "/usr/local/bin/k3s"
def _cilium_crd_present() -> bool:
try:
result = subprocess.run(
[_K3S, "kubectl", "get", "crd", "ciliumnetworkpolicies.cilium.io"],
capture_output=True,
text=True,
)
return result.returncode == 0
except FileNotFoundError:
return False
requires_cilium = pytest.mark.skipif(
not _cilium_crd_present(),
reason="Cilium CRD not installed (cluster likely running Flannel)",
)
async def _wait_ready(k7_core: K7Core, name: str, namespace: str, timeout: int = 600) -> None:
deadline = time.time() + timeout
while time.time() < deadline:
sandboxes = await k7_core.list_sandboxes(namespace=namespace)
if any(s.name == name and s.ready == "True" for s in sandboxes):
return
await asyncio.sleep(3)
raise AssertionError(f"Sandbox {name} never became Ready within {timeout}s")
async def _read_netpol(k7_core: K7Core, name: str, namespace: str):
net = await k7_core._get_networking_v1_client()
return await net.read_namespaced_network_policy(name=name, namespace=namespace)
async def _read_cnp(k7_core: K7Core, name: str, namespace: str):
custom = await k7_core._get_custom_objects_client()
return await custom.get_namespaced_custom_object(
group="cilium.io",
version="v2",
namespace=namespace,
plural="ciliumnetworkpolicies",
name=name,
)
def _deployment_exists(name: str, namespace: str) -> bool:
result = subprocess.run(
[_K3S, "kubectl", "-n", namespace, "get", "deployment", name],
capture_output=True,
text=True,
)
return result.returncode == 0
async def _serve(k7_core: K7Core, name: str, namespace: str, port: int) -> None:
"""Start a persistent HTTP listener inside a sandbox, serving `/` and a
`/cgi-bin/ip` that reports the client IP the sandbox actually saw.
busybox-extras `httpd` rather than `nc -l` or socat, both of which were tried
and produced false drops: `printf … | nc -l` never holds a listening socket
(nc exits when its stdin pipe hits EOF), and `socat …,fork SYSTEM:` drops
roughly one connection in ten under load. Either way a real allow rule reads
as a policy drop, which is the one mistake these tests must not make. httpd
is a real daemon and measured 20/20 on the same probe loop that flapped
under socat. `httpd` is not in alpine's stock busybox applet set.
"""
root = f"/srv{port}"
setup = (
"apk add --no-cache busybox-extras >/dev/null 2>&1; "
f"mkdir -p {root}/cgi-bin; echo hi > {root}/index.html; "
f'printf "#!/bin/sh\\necho Content-Type: text/plain\\necho\\necho \\$REMOTE_ADDR\\n" > {root}/cgi-bin/ip; '
f"chmod +x {root}/cgi-bin/ip; "
f"httpd -p {port} -h {root}; "
f"for _ in 1 2 3 4 5; do netstat -ltn | grep -q ':{port} ' && break; sleep 1; done; "
f"netstat -ltn | grep ':{port} '"
)
result = await k7_core.exec_command(name, setup, namespace=namespace)
assert f":{port}" in result.stdout, f"listener on {port} never bound: {result.stdout!r} {result.stderr!r}"
async def _probe(k7_core: K7Core, client_sb: str, target_ip: str, port: int, namespace: str) -> str:
"""Return "OK" or "BLOCKED" for an HTTP GET from ``client_sb`` — the
`&& echo OK || echo BLOCKED` sentinel style `test_cilium.py` uses, because a
policy drop shows up as a hang, not a refusal.
That distinction is the debugging tool when one of these tests fails:
"connection refused" in `wget`'s output means the policy passed the packet
and the listener is the problem, a timeout means the policy dropped it.
"""
result = await k7_core.exec_command(
client_sb,
f"wget -q -T 6 -O /dev/null http://{target_ip}:{port}/ && echo OK || echo BLOCKED",
namespace=namespace,
)
lines = result.stdout.strip().splitlines()
return lines[-1] if lines else f"NO-OUTPUT({result.stderr!r})"
async def _pod_ip(k7_core: K7Core, name: str, namespace: str) -> str:
v1 = await k7_core._get_core_v1_client()
pods = await v1.list_namespaced_pod(namespace=namespace, label_selector=f"app={name}")
live = [p for p in pods.items if not p.metadata.deletion_timestamp]
assert live and live[0].status.pod_ip, f"no pod IP for {name}"
return live[0].status.pod_ip
class TestForkInheritsNetworkPolicy:
"""Default-deny regression tests."""
@requires_cilium
async def test_fork_of_fqdn_egress_sandbox_inherits_policies(self, k7_core: K7Core, test_namespace: str):
source, fork = "integ-np-src", "integ-np-fork"
cfg = SandboxConfig(
name=source,
image="alpine:3.20",
namespace=test_namespace,
backend="kata-qemu-longhorn",
egress_whitelist=["example.com"],
)
# No `before_script`: a fork inherits the source's readiness probe
# (`test -f /tmp/k7_before_done_<source>`), and /tmp is not on the
# cloned disk, so such a fork never goes Ready. Use alpine's built-in
# busybox wget instead of installing curl.
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
await _wait_ready(k7_core, source, test_namespace)
fork_result = await k7_core.fork_sandbox(source, fork, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
cnp = await _read_cnp(k7_core, f"{fork}-egress", test_namespace)
egress = cnp["spec"]["egress"]
assert any(
"toFQDNs" in r and any(m.get("matchName") == "example.com" for m in r["toFQDNs"]) for r in egress
), f"fork did not inherit the source's FQDN egress: {egress!r}"
assert cnp["spec"]["endpointSelector"]["matchLabels"]["katakate.org/sandbox"] == fork
deny = await _read_netpol(k7_core, f"{fork}-deny-ingress", test_namespace)
assert deny.spec.policy_types == ["Ingress"]
assert deny.spec.ingress in (None, [])
await _wait_ready(k7_core, fork, test_namespace)
allowed = await k7_core.exec_command(
fork,
"wget -q -T 20 -O /dev/null https://example.com/ && echo OK || echo BLOCKED",
namespace=test_namespace,
)
assert "OK" in allowed.stdout, (
f"whitelisted domain unreachable from fork: stdout={allowed.stdout!r} stderr={allowed.stderr!r}"
)
blocked = await k7_core.exec_command(
fork,
"wget -q -T 8 -O /dev/null https://www.iana.org/ && echo OK || echo BLOCKED",
namespace=test_namespace,
)
assert "BLOCKED" in blocked.stdout, (
f"fork reached a non-whitelisted domain — it inherited no egress policy "
f"stdout={blocked.stdout!r} stderr={blocked.stderr!r}"
)
finally:
await k7_core.delete_sandbox(fork, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_fork_of_open_egress_sandbox_gets_no_egress_policy(self, k7_core: K7Core, test_namespace: str):
"""Correct inheritance, not a bug: an open-egress source forks open."""
source, fork = "integ-np-open-src", "integ-np-open-fork"
cfg = SandboxConfig(
name=source,
image="alpine:3.20",
namespace=test_namespace,
backend="kata-qemu-longhorn",
egress_whitelist=None,
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
await _wait_ready(k7_core, source, test_namespace)
fork_result = await k7_core.fork_sandbox(source, fork, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
with pytest.raises(ApiException) as excinfo:
await _read_netpol(k7_core, f"{fork}-netpol", test_namespace)
assert excinfo.value.status == 404
# Ingress is still denied — that default is not inherited, it is absolute.
deny = await _read_netpol(k7_core, f"{fork}-deny-ingress", test_namespace)
assert deny.spec.ingress in (None, [])
finally:
await k7_core.delete_sandbox(fork, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_fork_policy_failure_rolls_back_deployment(
self, k7_core: K7Core, test_namespace: str, monkeypatch: pytest.MonkeyPatch
):
"""A fork whose policy creation fails must not survive: an unrestricted
sandbox is exactly the failure mode M0 fixes."""
source, fork = "integ-np-rb-src", "integ-np-rb-fork"
cfg = SandboxConfig(
name=source,
image="alpine:3.20",
namespace=test_namespace,
backend="kata-qemu-longhorn",
egress_whitelist=["10.0.0.0/8"],
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
await _wait_ready(k7_core, source, test_namespace)
real_apply = k7_core._apply_sandbox_network_policies
async def _fail_for_fork(name: str, namespace: str, *args, **kwargs):
if name == fork:
return OperationResult(success=False, error="injected policy failure")
return await real_apply(name, namespace, *args, **kwargs)
monkeypatch.setattr(k7_core, "_apply_sandbox_network_policies", _fail_for_fork)
fork_result = await k7_core.fork_sandbox(source, fork, namespace=test_namespace)
assert not fork_result.success
assert "injected policy failure" in fork_result.error
assert not _deployment_exists(fork, test_namespace), (
"fork Deployment survived a policy failure — it would be running unrestricted"
)
finally:
await k7_core.delete_sandbox(fork, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
class TestSandboxIngress:
"""opt-in in-cluster ingress, deny-all default."""
async def _sandbox(self, k7_core: K7Core, namespace: str, name: str, **kwargs) -> None:
cfg = SandboxConfig(name=name, image="alpine:3.20", namespace=namespace, **kwargs)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create {name} failed: {result.error}"
await _wait_ready(k7_core, name, namespace, timeout=300)
async def test_default_denies_all_ingress(self, k7_core: K7Core, test_namespace: str):
"""The default policy is byte-identical to the one k7 always created,
and it actually blocks a same-namespace peer."""
server, client_sb = "integ-ing-deny", "integ-ing-deny-peer"
try:
await self._sandbox(k7_core, test_namespace, server)
await self._sandbox(k7_core, test_namespace, client_sb)
deny = await _read_netpol(k7_core, f"{server}-deny-ingress", test_namespace)
assert deny.spec.policy_types == ["Ingress"]
assert deny.spec.ingress in (None, [])
await _serve(k7_core, server, test_namespace, 8000)
ip = await _pod_ip(k7_core, server, test_namespace)
assert await _probe(k7_core, client_sb, ip, 8000, test_namespace) == "BLOCKED"
finally:
await k7_core.delete_sandbox(client_sb, namespace=test_namespace)
await k7_core.delete_sandbox(server, namespace=test_namespace)
async def test_ingress_port_opens_only_that_port(self, k7_core: K7Core, test_namespace: str):
server, client_sb = "integ-ing-open", "integ-ing-open-peer"
try:
await self._sandbox(k7_core, test_namespace, server, ingress_ports=[8000])
await self._sandbox(k7_core, test_namespace, client_sb)
await _serve(k7_core, server, test_namespace, 8000)
await _serve(k7_core, server, test_namespace, 8001)
ip = await _pod_ip(k7_core, server, test_namespace)
assert await _probe(k7_core, client_sb, ip, 8000, test_namespace) == "OK"
assert await _probe(k7_core, client_sb, ip, 8001, test_namespace) == "BLOCKED"
finally:
await k7_core.delete_sandbox(client_sb, namespace=test_namespace)
await k7_core.delete_sandbox(server, namespace=test_namespace)
async def test_ingress_from_scopes_to_one_sandbox(self, k7_core: K7Core, test_namespace: str):
server, allowed, denied = "integ-ing-scope", "integ-ing-a", "integ-ing-c"
try:
await self._sandbox(k7_core, test_namespace, allowed)
await self._sandbox(k7_core, test_namespace, denied)
await self._sandbox(
k7_core, test_namespace, server, ingress_ports=[8000], ingress_from=[f"sandbox:{allowed}"]
)
await _serve(k7_core, server, test_namespace, 8000)
ip = await _pod_ip(k7_core, server, test_namespace)
assert await _probe(k7_core, allowed, ip, 8000, test_namespace) == "OK"
assert await _probe(k7_core, denied, ip, 8000, test_namespace) == "BLOCKED"
finally:
for name in (denied, allowed, server):
await k7_core.delete_sandbox(name, namespace=test_namespace)
@requires_cilium
async def test_cidr_source_does_not_restrict_in_cluster_peers(self, k7_core: K7Core, test_namespace: str):
"""Pins the `cidr:` caveat that `k7 create` warns about.
Cilium does not evaluate an `ipBlock` peer for traffic where both ends are
Cilium-managed, so a rule whose peers are all `cidr:` degrades to a
port-only allow: an unrelated sandbox reaches the opened port even though
the CIDR here contains no cluster IP at all. The port list is still
enforced, so this is a peer-scoping hole, not a blanket allow.
If this ever starts returning BLOCKED — a Cilium upgrade, or
`policy-cidr-match-mode=pods` — the warning and the docs are what need
updating.
"""
server, peer = "integ-ing-cidr", "integ-ing-cidr-peer"
try:
await self._sandbox(k7_core, test_namespace, peer)
await self._sandbox(
k7_core,
test_namespace,
server,
ingress_ports=[8000],
ingress_from=["cidr:198.51.100.0/24"],
)
await _serve(k7_core, server, test_namespace, 8000)
ip = await _pod_ip(k7_core, server, test_namespace)
assert await _probe(k7_core, peer, ip, 8000, test_namespace) == "OK"
assert await _probe(k7_core, peer, ip, 9999, test_namespace) == "BLOCKED"
finally:
for name in (peer, server):
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_unparseable_source_fails_at_create(self, k7_core: K7Core, test_namespace: str):
cfg = SandboxConfig(
name="integ-ing-bad",
image="alpine:3.20",
namespace=test_namespace,
ingress_ports=[8000],
ingress_from=["nonsense"],
)
result = await k7_core.create_sandbox(cfg)
assert not result.success
assert "nonsense" in result.error
assert not _deployment_exists("integ-ing-bad", test_namespace)
async def test_sources_without_ports_fail_at_create(self, k7_core: K7Core, test_namespace: str):
cfg = SandboxConfig(
name="integ-ing-noport",
image="alpine:3.20",
namespace=test_namespace,
ingress_from=["sandbox:whoever"],
)
result = await k7_core.create_sandbox(cfg)
assert not result.success
assert "ingress_from requires ingress_ports" in result.error
assert not _deployment_exists("integ-ing-noport", test_namespace)
async def test_fork_inherits_ingress_rules(self, k7_core: K7Core, test_namespace: str):
source, fork = "integ-ing-fsrc", "integ-ing-ffork"
try:
await self._sandbox(
k7_core,
test_namespace,
source,
backend="kata-qemu-longhorn",
ingress_ports=[8000],
ingress_from=["sandbox:integ-ing-peer"],
)
fork_result = await k7_core.fork_sandbox(source, fork, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
np = await _read_netpol(k7_core, f"{fork}-deny-ingress", test_namespace)
rule = np.spec.ingress[0]
assert [(p.protocol, p.port) for p in rule.ports] == [("TCP", 8000)]
assert rule._from[0].pod_selector.match_labels == {"katakate.org/sandbox": "integ-ing-peer"}
assert np.spec.pod_selector.match_labels == {"katakate.org/sandbox": fork}
finally:
await k7_core.delete_sandbox(fork, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
def _curl(url: str, path: str = "/") -> subprocess.CompletedProcess:
"""Reach a NodePort from the node the suite runs on."""
return subprocess.run(
["curl", "-sS", "--max-time", "8", f"{url}{path}"],
capture_output=True,
text=True,
)
def _curl_until_answered(url: str, attempts: int = 10) -> subprocess.CompletedProcess:
for _ in range(attempts):
probe = _curl(url)
if probe.returncode == 0 and "hi" in probe.stdout:
return probe
time.sleep(3)
raise AssertionError(f"NodePort {url} never answered: {probe.stdout!r} {probe.stderr!r}")
def _observed_source_ip(url: str) -> str:
"""The source IP the sandbox saw for our request, straight out of `httpd`'s
CGI `REMOTE_ADDR`.
A NodePort test must establish this before
asserting anything about a source-CIDR rule, otherwise it cannot tell a
working policy from SNAT. `httpd` binds dual-stack and reports IPv4 peers
IPv4-mapped, so unwrap `[::ffff:a.b.c.d]`.
"""
probe = _curl(url, "/cgi-bin/ip")
assert probe.returncode == 0, f"could not read back the source IP from {url}: {probe.stderr!r}"
raw = probe.stdout.strip().strip("[]")
return raw.rsplit(":", 1)[-1] if raw.lower().startswith("::ffff:") else raw
async def v1_nodes(k7_core: K7Core):
return await (await k7_core._get_core_v1_client()).list_node()
class TestSandboxExpose:
"""NodePort exposure outside the cluster."""
async def _exposed(self, k7_core: K7Core, namespace: str, name: str, sources: list[str]) -> str:
cfg = SandboxConfig(
name=name,
image="alpine:3.20",
namespace=namespace,
ingress_ports=[8000],
ingress_from=sources,
expose_ports=[8000],
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create {name} failed: {result.error}"
endpoints = (result.data or {}).get("endpoints") or []
assert len(endpoints) == 1 and endpoints[0]["node_port"], f"no resolved endpoint: {result.data!r}"
await _wait_ready(k7_core, name, namespace, timeout=300)
return endpoints[0]["url"]
async def test_expose_is_reachable_and_cleaned_up(self, k7_core: K7Core, test_namespace: str):
name = "integ-exp-open"
try:
url = await self._exposed(k7_core, test_namespace, name, ["cidr:0.0.0.0/0"])
await _serve(k7_core, name, test_namespace, 8000)
v1 = await k7_core._get_core_v1_client()
svc = await v1.read_namespaced_service(name=f"{name}-expose", namespace=test_namespace)
# Without `Local` the client's source IP is SNAT'd before the pod
# sees it and every cidr: rule in front of a NodePort is decoration.
assert svc.spec.external_traffic_policy == "Local"
assert svc.spec.type == "NodePort"
assert [p.node_port for p in svc.spec.ports] == [int(url.rsplit(":", 1)[1])]
_curl_until_answered(url)
listed = [s for s in await k7_core.list_sandboxes(namespace=test_namespace) if s.name == name]
assert listed and listed[0].node_ports == [int(url.rsplit(":", 1)[1])]
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
remaining = subprocess.run(
[_K3S, "kubectl", "-n", test_namespace, "get", "svc", "-o", "name"],
capture_output=True,
text=True,
)
assert remaining.stdout.strip() == "", f"delete leaked Services: {remaining.stdout!r}"
@requires_cilium
async def test_node_local_traffic_is_snat_and_not_cidr_scoped(self, k7_core: K7Core, test_namespace: str):
"""Pins why this suite cannot validate `--expose-port`'s source-IP
behaviour by itself, which is the trap: it runs *on the
node*, and node-local traffic to a NodePort is both SNAT'd and outside
what a `cidr:` peer is evaluated against.
Two things are asserted, and both are caveats rather than features:
* the pod sees an address on the *pod* network, not the node IP — so
`externalTrafficPolicy: Local` does not preserve the source for a
hairpinned request from the node itself;
* the request is answered even though the allowlist is
`cidr:198.51.100.0/24`, because Cilium does not evaluate an `ipBlock`
peer against a node IP without `policy-cidr-match-mode=nodes`.
So `curl` from the node passes no matter how the rule is written. Both
halves were verified from a genuinely external client instead, and both
work there: with `cidr:0.0.0.0/0` the sandbox observed the client's real
public IP, and with `cidr:198.51.100.0/24` the same client timed out
while a control sandbox on the same node answered. That check needs an
off-cluster host, so it stays a manual step — if this test ever starts
failing, node-local behaviour changed and the docs need revisiting.
"""
name = "integ-exp-nodelocal"
try:
url = await self._exposed(k7_core, test_namespace, name, ["cidr:198.51.100.0/24"])
await _serve(k7_core, name, test_namespace, 8000)
_curl_until_answered(url)
observed = ipaddress.ip_address(_observed_source_ip(url))
assert observed not in ipaddress.ip_network("198.51.100.0/24")
node_ips = {
a.address for n in (await v1_nodes(k7_core)).items for a in n.status.addresses or [] if "IP" in a.type
}
assert str(observed) not in node_ips, (
f"the pod saw the node IP {observed} — node-local traffic is no longer SNAT'd, so this test and "
"the BACKENDS.md caveat both need rewriting"
)
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_expose_without_matching_ingress_port_is_refused(self, k7_core: K7Core, test_namespace: str):
cfg = SandboxConfig(
name="integ-exp-noingress",
image="alpine:3.20",
namespace=test_namespace,
expose_ports=[8000],
)
result = await k7_core.create_sandbox(cfg)
assert not result.success
assert "[8000]" in result.error
assert not _deployment_exists("integ-exp-noingress", test_namespace)
-333
View File
@@ -1,333 +0,0 @@
"""Integration tests for --sidecar docker on both backends."""
import asyncio
import json
import subprocess
import time
import pytest
from k7.core.core import K7Core
from k7.core.models import SandboxConfig
pytestmark = pytest.mark.integration
def _wait_all_containers_ready(sandbox_name: str, namespace: str, timeout: int = 300) -> None:
"""Block until all containers in the pod are ready."""
deadline = time.time() + timeout
while time.time() < deadline:
result = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pods",
"-n",
namespace,
"-l",
f"app={sandbox_name}",
"-o",
"json",
],
capture_output=True,
text=True,
check=False,
)
if result.returncode != 0:
time.sleep(2)
continue
try:
data = json.loads(result.stdout)
items = data.get("items", [])
if not items:
time.sleep(2)
continue
pod = items[0]
statuses = pod.get("status", {}).get("containerStatuses", [])
if statuses and all(s.get("ready") for s in statuses):
return
except (json.JSONDecodeError, KeyError, IndexError):
pass
time.sleep(2)
raise TimeoutError(f"Not all containers ready for {sandbox_name} within {timeout}s")
async def _wait_docker_ready(k7_core: K7Core, name: str, namespace: str, timeout: int = 120) -> None:
"""Wait for the docker daemon inside the sidecar to respond to 'docker info'."""
deadline = time.time() + timeout
while time.time() < deadline:
result = await k7_core.exec_command(name, "docker info >/dev/null 2>&1 && echo ok", namespace=namespace)
if result.exit_code == 0 and "ok" in result.stdout:
return
await asyncio.sleep(3)
raise TimeoutError(f"Docker daemon not ready in {name} within {timeout}s")
async def _docker_has_image(k7_core: K7Core, name: str, namespace: str, image: str) -> bool:
"""Check if a specific docker image exists in the sandbox."""
result = await k7_core.exec_command(
name, f"docker image inspect {image} >/dev/null 2>&1 && echo yes", namespace=namespace
)
return result.exit_code == 0 and "yes" in result.stdout
# ===========================================================================
# kata-qemu-longhorn tests
# ===========================================================================
@pytest.mark.qemu
class TestSidecarDockerQL:
async def test_sidecar_docker_run_ql(self, k7_core: K7Core, test_namespace: str):
name = "sc-docker-run-ql"
cfg = SandboxConfig(
name=name,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
exec_result = await k7_core.exec_command(name, "docker run --rm hello-world", namespace=test_namespace)
assert exec_result.exit_code == 0, f"exec failed: {exec_result.stderr}"
assert "Hello from Docker!" in exec_result.stdout
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_sidecar_docker_build_ql(self, k7_core: K7Core, test_namespace: str):
name = "sc-docker-build-ql"
cfg = SandboxConfig(
name=name,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
await k7_core.exec_command(
name,
"mkdir -p /tmp/build && printf 'FROM alpine:3.21\\nCMD echo built-ok' > /tmp/build/Dockerfile",
namespace=test_namespace,
)
build_result = await k7_core.exec_command(
name, "cd /tmp/build && docker build -t myapp .", namespace=test_namespace
)
assert build_result.exit_code == 0, f"build failed: {build_result.stderr}"
run_result = await k7_core.exec_command(name, "docker run --rm myapp", namespace=test_namespace)
assert run_result.exit_code == 0, f"run failed: {run_result.stderr}"
assert "built-ok" in run_result.stdout
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_sidecar_docker_pause_resume_ql(self, k7_core: K7Core, test_namespace: str):
name = "sc-docker-pr-ql"
cfg = SandboxConfig(
name=name,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
pull_result = await k7_core.exec_command(name, "docker pull alpine:3.21", namespace=test_namespace)
assert pull_result.exit_code == 0, f"pull failed: {pull_result.stderr}"
pause_result = await k7_core.pause_sandbox(name, namespace=test_namespace)
assert pause_result.success, f"pause failed: {pause_result.error}"
await asyncio.sleep(5)
resume_result = await k7_core.resume_sandbox(name, namespace=test_namespace)
assert resume_result.success, f"resume failed: {resume_result.error}"
_wait_all_containers_ready(name, test_namespace, timeout=120)
await _wait_docker_ready(k7_core, name, test_namespace)
assert await _docker_has_image(k7_core, name, test_namespace, "alpine:3.21"), (
"alpine:3.21 not found after resume"
)
run_result = await k7_core.exec_command(
name, "docker run --rm alpine:3.21 echo survived-pause", namespace=test_namespace
)
assert run_result.exit_code == 0
assert "survived-pause" in run_result.stdout
finally:
await k7_core.delete_sandbox(name, namespace=test_namespace)
async def test_sidecar_docker_fork_ql(self, k7_core: K7Core, test_namespace: str):
source = "sc-docker-fork-src"
fork_name = "sc-docker-fork-dst"
cfg = SandboxConfig(
name=source,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(source, test_namespace)
await _wait_docker_ready(k7_core, source, test_namespace)
pull_result = await k7_core.exec_command(source, "docker pull alpine:3.21", namespace=test_namespace)
assert pull_result.exit_code == 0, f"pull failed: {pull_result.stderr}"
fork_result = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
_wait_all_containers_ready(fork_name, test_namespace, timeout=180)
await _wait_docker_ready(k7_core, fork_name, test_namespace)
assert await _docker_has_image(k7_core, fork_name, test_namespace, "alpine:3.21"), (
"alpine:3.21 not found in fork"
)
run_result = await k7_core.exec_command(
fork_name, "docker run --rm alpine:3.21 echo fork-works", namespace=test_namespace
)
assert run_result.exit_code == 0
assert "fork-works" in run_result.stdout
finally:
await k7_core.delete_sandbox(fork_name, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_sidecar_docker_fork_independence_ql(self, k7_core: K7Core, test_namespace: str):
"""Verify docker images pulled in the fork don't leak back to the source."""
source = "sc-docker-fi-src"
fork_name = "sc-docker-fi-dst"
cfg = SandboxConfig(
name=source,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
try:
_wait_all_containers_ready(source, test_namespace)
await _wait_docker_ready(k7_core, source, test_namespace)
pull_result = await k7_core.exec_command(source, "docker pull alpine:3.21", namespace=test_namespace)
assert pull_result.exit_code == 0, f"pull failed: {pull_result.stderr}"
fork_result = await k7_core.fork_sandbox(source, fork_name, namespace=test_namespace)
assert fork_result.success, f"fork failed: {fork_result.error}"
_wait_all_containers_ready(fork_name, test_namespace, timeout=180)
await _wait_docker_ready(k7_core, fork_name, test_namespace)
fork_pull = await k7_core.exec_command(fork_name, "docker pull busybox:1.36", namespace=test_namespace)
assert fork_pull.exit_code == 0, f"fork pull failed: {fork_pull.stderr}"
assert await _docker_has_image(k7_core, fork_name, test_namespace, "busybox:1.36"), (
"busybox:1.36 not found in fork"
)
assert await _docker_has_image(k7_core, fork_name, test_namespace, "alpine:3.21"), (
"alpine:3.21 not found in fork (should be inherited)"
)
assert not await _docker_has_image(k7_core, source, test_namespace, "busybox:1.36"), (
"busybox:1.36 leaked from fork to source"
)
assert await _docker_has_image(k7_core, source, test_namespace, "alpine:3.21"), (
"alpine:3.21 missing from source"
)
finally:
await k7_core.delete_sandbox(fork_name, namespace=test_namespace)
await k7_core.delete_sandbox(source, namespace=test_namespace)
async def test_sidecar_docker_cleanup_ql(self, k7_core: K7Core, test_namespace: str):
name = "sc-docker-clean-ql"
cfg = SandboxConfig(
name=name,
image="docker:27.5-cli",
namespace=test_namespace,
backend="kata-qemu-longhorn",
sidecar="docker",
)
result = await k7_core.create_sandbox(cfg)
assert result.success, f"create failed: {result.error}"
_wait_all_containers_ready(name, test_namespace)
await _wait_docker_ready(k7_core, name, test_namespace)
await k7_core.exec_command(name, "docker pull alpine:3.21", namespace=test_namespace)
await k7_core.delete_sandbox(name, namespace=test_namespace)
await asyncio.sleep(5)
pvc_check = subprocess.run(
[
"k3s",
"kubectl",
"get",
"pvc",
"-n",
test_namespace,
"-o",
"jsonpath={.items[*].metadata.name}",
],
capture_output=True,
text=True,
check=False,
)
remaining_pvcs = [p for p in pvc_check.stdout.strip().split() if p.startswith(name)]
assert not remaining_pvcs, f"leaked PVCs: {remaining_pvcs}"
dep_check = subprocess.run(
[
"k3s",
"kubectl",
"get",
"deployment",
"-n",
test_namespace,
"-o",
"jsonpath={.items[*].metadata.name}",
],
capture_output=True,
text=True,
check=False,
)
remaining_deps = [d for d in dep_check.stdout.strip().split() if d.startswith(name)]
assert not remaining_deps, f"leaked deployments: {remaining_deps}"
# ===========================================================================
# kata-firecracker-devmapper tests
# ===========================================================================
# An earlier note here claimed that Docker DinD couldn't run on the
# Firecracker backend because Kata-firecracker "doesn't support privileged
# containers". That note was wrong — verified during spec 10b's bench work
# by spinning up ``--backend kfd --sidecar docker`` and running
# ``docker run hello-world`` successfully in ~3s. The sidecar wiring in
# core.py is symmetric across backends; the only difference is that the
# docker daemon's ``/var/lib/docker`` lives on an emptyDir for FD vs a
# Longhorn PVC sub_path for kata-qemu-longhorn (so FD's docker state is
# ephemeral by design — gone with the sandbox).
#
# FD sidecar integration tests of the same shape as the QL ones above
# would be a worthwhile follow-up (TestSidecarDockerFD class). Skipped
# in this commit only to keep the bench PR scope tight; the
# tests/integration/bench_docker_perf.py module already exercises the
# FD-docker path end-to-end against a real cluster.
# ===========================================================================
+67 -4
View File
@@ -1,4 +1,4 @@
"""Integration tests for the Spec 10e snapshot lifecycle, multi-node cluster.
"""Integration tests for the snapshot lifecycle, multi-node cluster.
Coverage:
@@ -13,8 +13,10 @@ Coverage:
"""
import asyncio
import json
import subprocess
import time
from datetime import timedelta
from pathlib import Path
from uuid import uuid4
@@ -31,7 +33,7 @@ _DEV_SH = _REPO_ROOT / "src" / "k7" / "cli" / "dev.sh"
def _run_cli(*args: str, check: bool = True, timeout: int = 120) -> subprocess.CompletedProcess:
"""Invoke ``dev.sh --core <args>`` — Spec 10g made the default API; these
"""Invoke ``dev.sh --core <args>`` — the default route is the API; these
CLI-flag tests don't want / need that detour."""
cmd = [str(_DEV_SH), "--core", *args]
return subprocess.run(cmd, capture_output=True, text=True, check=check, timeout=timeout, cwd=str(_REPO_ROOT))
@@ -82,6 +84,50 @@ def _snapshot_exists(name: str, namespace: str) -> bool:
return out.returncode == 0 and out.stdout.strip() != ""
def _wait_snapshot_deleting_or_gone(name: str, namespace: str, timeout: int = 60) -> None:
"""GC skips a VolumeSnapshotContent while its VolumeSnapshot is still live."""
deadline = time.time() + timeout
while time.time() < deadline:
out = subprocess.run(
[_K3S, "kubectl", "get", "volumesnapshot", name, "-n", namespace, "-o", "json"],
capture_output=True,
text=True,
)
if out.returncode != 0:
return
try:
meta = (json.loads(out.stdout) or {}).get("metadata") or {}
except json.JSONDecodeError:
time.sleep(1)
continue
if meta.get("deletionTimestamp"):
return
time.sleep(1)
raise TimeoutError(f"VolumeSnapshot {namespace}/{name} still live after delete")
def _vsc_names_for_namespace(namespace: str) -> list[str]:
raw = subprocess.run(
[_K3S, "kubectl", "get", "volumesnapshotcontent", "-o", "json"],
capture_output=True,
text=True,
)
if raw.returncode != 0 or not raw.stdout.strip():
return []
try:
items = json.loads(raw.stdout).get("items") or []
except json.JSONDecodeError:
return []
names = []
for item in items:
ref = (item.get("spec") or {}).get("volumeSnapshotRef") or {}
if ref.get("namespace") == namespace:
name = (item.get("metadata") or {}).get("name")
if name:
names.append(name)
return names
def _force_delete_snapshot(name: str, namespace: str) -> None:
"""Best-effort cleanup: strip finalizers so Longhorn doesn't hang the delete."""
subprocess.run(
@@ -189,7 +235,7 @@ class TestSnapshotCrud:
class TestForkInlineSnapshotDelete:
"""Spec 10e Option A: ``fork`` cleans up its ``-fork-`` snapshot inline."""
"""Option A: ``fork`` cleans up its ``-fork-`` snapshot inline."""
async def test_fork_auto_deletes_temp_snapshot(
self,
@@ -241,8 +287,25 @@ class TestForkInlineSnapshotDelete:
pass
class TestGcOrphanContents:
"""GC must reap leftover VolumeSnapshotContents so namespaces can die."""
def test_gc_reaps_orphan_vsc_after_snapshot_delete(
self,
ql_sandbox: str,
test_namespace: str,
):
snap = f"{ql_sandbox}-exp-baseline"
_run_cli("snapshot", "create", ql_sandbox, snap, "-n", test_namespace)
assert _snapshot_exists(snap, test_namespace)
_run_cli("snapshot", "delete", snap, "-n", test_namespace, "--yes")
_wait_snapshot_deleting_or_gone(snap, test_namespace)
asyncio.run(K7Core().gc_snapshots(namespace=test_namespace, keep_fork_for=timedelta(0)))
assert _vsc_names_for_namespace(test_namespace) == []
class TestGcSafety:
"""Spec 10e: pause and named snapshots are never auto-collected."""
"""Pause and named snapshots are never auto-collected."""
def test_pause_and_named_snapshots_survive_gc(
self,
+11 -5
View File
@@ -1,10 +1,10 @@
"""Cluster-wide VMM process leak detection (spec 18f issue 1b).
"""Cluster-wide VMM process leak detection.
Under parallel kata-fc pod churn the kata shim intermittently fails to
kill the Firecracker microVM on pod deletion (shim logs "Agent did not
stop sandbox: Dead agent" / "getting vm status failed: ...
firecracker.socket: no such file or directory"), leaving orphaned
``firecracker`` processes spinning at 100% CPU. Spec 18e observed 14
``firecracker`` processes spinning at 100% CPU. An HA run observed 14
orphans across the cluster with zero live Kata pods.
This module is named ``test_zz_*`` so pytest runs it LAST: after the
@@ -39,8 +39,14 @@ def _kubectl(*args: str, check: bool = True) -> subprocess.CompletedProcess:
def _node_names() -> list[str]:
out = _kubectl("get", "nodes", "-o", "jsonpath={.items[*].metadata.name}").stdout
return out.split()
"""Ready nodes only — a NotReady peer cannot run the hostPID scan pod."""
data = json.loads(_kubectl("get", "nodes", "-o", "json").stdout)
names: list[str] = []
for node in data.get("items", []):
conds = {c["type"]: c["status"] for c in node.get("status", {}).get("conditions", [])}
if conds.get("Ready") == "True":
names.append(node["metadata"]["name"])
return names
def _live_kata_pods_per_node() -> dict[str, dict[str, int]]:
@@ -135,7 +141,7 @@ class TestVmmProcessLeaks:
"VMM process leak detected (did not converge within "
f"{_CONVERGE_TIMEOUT}s):\n " + "\n ".join(mismatches) + "\n"
"Orphaned firecracker processes spin at 100% CPU and starve the node "
"(spec 18f issue 1b / CHALLENGES.md #6). Inspect with "
"(CHALLENGES.md #6). Inspect with "
"`ps -eo pid,ppid,comm,%cpu | grep -E 'firecracker|qemu'` on the node "
"and check `journalctl -t kata` for 'Agent did not stop sandbox'."
)
@@ -0,0 +1,26 @@
# Pinned Docker guest payload. Bump as a set.
# IPTABLES_BACKEND was measured on the node: `docker run --rm --privileged
# docker:27.5-dind iptables -V` → `iptables v1.8.11 (nf_tables)`.
DOCKER_VERSION=27.5.1
COMPOSE_VERSION=2.32.4
BUILDX_VERSION=0.20.0
IPTABLES_VERSION=1.8.11
IPTABLES_BACKEND=nft
DOCKER_TGZ_SHA256=4f798b3ee1e0140eab5bf30b0edc4e84f4cdb53255a429dc3bbae9524845d640
COMPOSE_SHA256=ed1917fb54db184192ea9d0717bcd59e3662ea79db48bff36d3475516c480a6b
BUILDX_SHA256=8b21d3ce1011c4c072d64d4a7311c591cf1c2eb6b35bfdfe28f8e0b76e51621b
IPTABLES_TARBALL_SHA256=d87303d55ef8c92bcad4dd3f978b26d272013642b029425775f5bad1009fe7b2
# alpine:3.21 image used to statically link iptables (nft) + busybox ip,
# and as the source of the Mozilla CA bundle dockerd needs for pulls
# (busybox initramfs has none; Go defaults to x509 unknown-authority).
ALPINE_IMAGE=alpine:3.21@sha256:48b0309ca019d89d40f670aa1bc06e426dc0931948452e8491e3d65087abc07d
CA_BUNDLE_SHA256=61efbd6d3f829f71039c57b29dd37d15ac7f33c4ece861aaef8c7d7a519cd1d9
# Guest sources this payload must not require changing (asserted by tests).
KERNEL_CONFIG_SHA256=c0412b90c10850d65d40922d10f684bf1f9ef6d28f2e2e46a2891d527d914c23
INIT_SH_SHA256=5f7ede84d09c0d12a80a6fb9190f4f46a0ab3e1048498f701c52d350d9f4e0fb
ROOTFS_DOCKERFILE_SHA256=9c22ce8ff06bed4a0c31388c1341768dada0334ccb391dd4c8cc00a0ec4a8ed1
ROOTFS_BUILD_SHA256=dca95de17690842838bf6f27fd9671f02af72c8e4efdf9ff344b829ab7ffc119
UBUNTU_TREE_SHA256=f646db404135a59559cbc4f3041b29ca1425e3ab97d45339f4239dbd2b49dd2d
+33 -1
View File
@@ -4,7 +4,7 @@ import hashlib
import json
import time
from pathlib import Path
from unittest.mock import patch
from unittest.mock import AsyncMock, patch
import httpx
import pytest
@@ -185,3 +185,35 @@ class TestAuthorizeNamespace:
params={"namespace": "beta"},
)
assert resp.status_code == 403
async def test_scoped_key_nodes_storage_returns_403(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts, namespaces=["alpha"])
keys_file.write_text(json.dumps(data))
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.get(
"/api/v1/nodes/storage",
headers={"X-API-Key": TEST_KEY},
)
assert resp.status_code == 403
assert "all-namespaces" in resp.json()["error"]["message"]
async def test_unscoped_key_nodes_storage_returns_200(self, _patch_keys_file, keys_file: Path):
future_ts = int(time.time()) + 86400
data = _make_keys_data(expires=future_ts)
keys_file.write_text(json.dumps(data))
payload = {"node-a": {"kata_thinpool": {"size_bytes": 1}}}
with patch("k7.api.main.K7Core") as core_cls:
core_cls.return_value.nodes_storage = AsyncMock(return_value=payload)
transport = httpx.ASGITransport(app=app)
async with httpx.AsyncClient(transport=transport, base_url="http://test") as client:
resp = await client.get(
"/api/v1/nodes/storage",
headers={"X-API-Key": TEST_KEY},
)
assert resp.status_code == 200
assert resp.json()["data"] == payload
core_cls.return_value.nodes_storage.assert_awaited_once()
+2 -2
View File
@@ -1,4 +1,4 @@
"""Unit tests for the pause/resume/fork API endpoints (Spec 10a).
"""Unit tests for the pause/resume/fork API endpoints.
Hits the FastAPI ASGI app directly with a mocked K7Core, so these tests
need no Kubernetes cluster. They verify request/response shape, body
@@ -64,7 +64,7 @@ class TestPauseEndpoint:
fake.assert_awaited_once_with(name="demo", namespace="default", snapshot_name=None)
async def test_pause_ignores_legacy_pvc_and_snapshot_class(self, keys_file):
"""Spec 10c: pvc/snapshot_class in the body must NOT be forwarded to Core.
"""pvc/snapshot_class in the body must NOT be forwarded to Core.
We accept them silently for one release rather than 400ing — but the
server-side behaviour is exactly as if they weren't sent.
+1 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for API key generation / namespace scoping (spec 10h)."""
"""Unit tests for API key generation / namespace scoping."""
import json
from pathlib import Path
+45 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for the Spec 10d ``k7 api ...`` sub-app + deprecation aliases.
"""Unit tests for the ``k7 api ...`` sub-app + deprecation aliases.
Coverage:
@@ -11,6 +11,8 @@ Coverage:
forward to the new home.
- ``k7 install --no-api`` passes ``k7_api_enabled=false`` into the
Ansible extra_vars (which is what the playbook actually keys off).
- ``k7 install --hubble`` passes ``k7_hubble_enabled=true``; combined
with ``--cni flannel`` it fails loudly.
"""
from __future__ import annotations
@@ -222,6 +224,48 @@ class TestInstallNoApi:
assert extra_vars.get("k7_api_enabled") == "true"
class TestInstallHubble:
def test_install_hubble_passes_k7_hubble_enabled_true(self):
seen: dict = {}
def _capture(*args, **kwargs):
seen["kwargs"] = kwargs
return SimpleNamespace(success=True, message="ok", error="")
with (
patch("k7.cli.k7.K7Core") as core_cls,
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "--hubble", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_hubble_enabled") == "true"
def test_install_without_hubble_passes_k7_hubble_enabled_false(self):
seen: dict = {}
def _capture(*args, **kwargs):
seen["kwargs"] = kwargs
return SimpleNamespace(success=True, message="ok", error="")
with (
patch("k7.cli.k7.K7Core") as core_cls,
patch("k7.cli.k7._build_default_inventory", return_value="[k7_servers]\nhost ansible_host=1.2.3.4"),
):
core_cls.return_value.install_node = _capture
result = runner.invoke(app, ["install", "host"])
assert result.exit_code == 0, result.output
extra_vars = seen["kwargs"]["extra_vars"]
assert extra_vars.get("k7_hubble_enabled") == "false"
def test_install_hubble_with_flannel_fails(self):
result = runner.invoke(app, ["install", "--hubble", "--cni", "flannel", "host"])
assert result.exit_code != 0
assert "--hubble requires the Cilium CNI" in result.output
assert "flannel" in result.output
# ---------------------------------------------------------------------------
# ``k7 --help`` no longer lists the deprecated commands.
# ---------------------------------------------------------------------------
+79
View File
@@ -0,0 +1,79 @@
"""CLI tests for --docker / --docker-disk and the --sidecar docker alias."""
from unittest.mock import AsyncMock, patch
from typer.testing import CliRunner
from k7.cli.k7 import app
from k7.core.models import OperationResult
runner = CliRunner()
def _mock_create():
fake = AsyncMock(return_value=OperationResult(success=True, message="created"))
patcher = patch("k7.cli.k7.K7Core")
core_cls = patcher.start()
core_cls.return_value.create_sandbox = fake
core_cls.return_value.list_sandboxes = AsyncMock(return_value=[])
return patcher, fake
def _config_for(args: list[str]):
patcher, fake = _mock_create()
try:
result = runner.invoke(app, ["--core", "create", *args])
finally:
patcher.stop()
return result, fake
def test_docker_flag_sets_config():
result, fake = _config_for(["demo", "ubuntu:24.04", "--docker", "--backend", "k7d"])
assert result.exit_code == 0, result.output
cfg = fake.await_args.args[0]
assert cfg.docker is True
assert cfg.sidecar is None
assert cfg.docker_disk is None
def test_docker_disk_requires_docker():
result, _ = _config_for(["demo", "ubuntu:24.04", "--docker-disk", "40Gi", "--backend", "k7d"])
assert result.exit_code != 0
assert "docker-disk" in result.output.lower() or "docker-disk" in (result.exception and str(result.exception) or "")
def test_docker_disk_forwarded():
result, fake = _config_for(["demo", "ubuntu:24.04", "--docker", "--docker-disk", "40Gi", "--backend", "k7d"])
assert result.exit_code == 0, result.output
cfg = fake.await_args.args[0]
assert cfg.docker is True
assert cfg.docker_disk == "40Gi"
def test_sidecar_docker_alias_warns_and_maps():
result, fake = _config_for(["demo", "ubuntu:24.04", "--sidecar", "docker", "--backend", "k7d"])
assert result.exit_code == 0, result.output
assert "deprecated" in result.output.lower()
cfg = fake.await_args.args[0]
assert cfg.docker is True
assert cfg.sidecar is None
def test_sidecar_docker_alias_on_kata_maps_to_docker():
result, fake = _config_for(["demo", "ubuntu:24.04", "--sidecar", "docker", "--backend", "kata-qemu-longhorn"])
assert result.exit_code == 0, result.output
assert "deprecated" in result.output.lower()
cfg = fake.await_args.args[0]
assert cfg.docker is True
assert cfg.sidecar is None
assert cfg.backend == "kata-qemu-longhorn"
def test_unknown_sidecar_lists_non_docker_types():
result, _ = _config_for(["demo", "ubuntu:24.04", "--sidecar", "nosuch"])
assert result.exit_code != 0
text = result.output + (str(result.exception) if result.exception else "")
assert "nosuch" in text
assert "Available: docker" not in text
assert "--docker" in text or "no non-docker" in text
+100
View File
@@ -0,0 +1,100 @@
"""Unit tests for the ``k7 create`` ingress flags."""
from unittest.mock import AsyncMock, patch
from typer.testing import CliRunner
from k7.cli.k7 import app
from k7.core.models import OperationResult
runner = CliRunner()
def _mock_create():
fake = AsyncMock(return_value=OperationResult(success=True, message="Sandbox demo created successfully"))
patcher = patch("k7.cli.k7.K7Core")
core_cls = patcher.start()
core_cls.return_value.create_sandbox = fake
core_cls.return_value.list_sandboxes = AsyncMock(return_value=[])
return patcher, fake
def _config_for(args: list[str]):
patcher, fake = _mock_create()
try:
result = runner.invoke(app, ["--core", "create", *args])
finally:
patcher.stop()
return result, fake
def test_no_ingress_flags_keeps_deny_all():
result, fake = _config_for(["demo", "alpine:3.20"])
assert result.exit_code == 0, result.output
cfg = fake.await_args.args[0]
assert cfg.ingress_ports is None
assert cfg.ingress_from is None
def test_ingress_port_without_sources_defaults_to_same_namespace():
result, fake = _config_for(["demo", "alpine:3.20", "--ingress-port", "8000"])
assert result.exit_code == 0, result.output
cfg = fake.await_args.args[0]
assert cfg.ingress_ports == [8000]
assert cfg.ingress_from is None
def test_ingress_from_is_forwarded():
result, fake = _config_for(["demo", "alpine:3.20", "--ingress-port", "8000", "--ingress-from", "sandbox:alice"])
assert result.exit_code == 0, result.output
cfg = fake.await_args.args[0]
assert cfg.ingress_from == ["sandbox:alice"]
def test_expose_port_is_forwarded():
result, fake = _config_for(["demo", "alpine:3.20", "--ingress-port", "8000", "--expose-port", "8000"])
assert result.exit_code == 0, result.output
assert fake.await_args.args[0].expose_ports == [8000]
def test_expose_to_the_world_warns_on_stderr():
patcher, _ = _mock_create()
try:
result = runner.invoke(
app,
[
"--core",
"create",
"demo",
"alpine:3.20",
"--ingress-port",
"8000",
"--ingress-from",
"cidr:0.0.0.0/0",
"--expose-port",
"8000",
],
)
finally:
patcher.stop()
assert result.exit_code == 0, result.output
assert "publishes the sandbox to the internet" in result.output
def test_ingress_from_without_port_is_an_error():
result, fake = _config_for(["demo", "alpine:3.20", "--ingress-from", "sandbox:alice"])
assert result.exit_code != 0
assert "--ingress-from requires --ingress-port" in result.output
fake.assert_not_awaited()
def test_cidr_source_warns_that_it_does_not_scope_in_cluster_peers():
result, _ = _config_for(["demo", "alpine:3.20", "--ingress-port", "8000", "--ingress-from", "cidr:203.0.113.0/24"])
assert result.exit_code == 0, result.output
assert "does not restrict in-cluster peers" in result.output
def test_sandbox_source_does_not_warn():
result, _ = _config_for(["demo", "alpine:3.20", "--ingress-port", "8000", "--ingress-from", "sandbox:alice"])
assert result.exit_code == 0, result.output
assert "does not restrict in-cluster peers" not in result.output
@@ -0,0 +1,103 @@
"""Unit tests for ``k7 install --api-allow-cidr``.
Everything here is CLI-side: an invalid CIDR, or the flag combined with
``--cni flannel``, must fail *before* Ansible runs.
"""
from pathlib import Path
from unittest.mock import MagicMock, patch
from typer.testing import CliRunner
from k7.cli.k7 import app
from k7.core.models import OperationResult
runner = CliRunner()
def _install(args: list[str]):
with patch("k7.cli.k7.K7Core") as core_cls:
install = MagicMock(return_value=OperationResult(success=True, message="ok"))
core_cls.return_value.install_node = install
result = runner.invoke(app, ["--core", "install", *args])
return result, install
def _extra_vars(install: MagicMock) -> dict:
return install.call_args.kwargs["extra_vars"]
def test_default_install_sends_an_empty_allowlist():
result, install = _install([])
assert result.exit_code == 0, result.output
assert _extra_vars(install)["k7_api_allow_cidrs"] == ""
def test_repeatable_flag_is_forwarded_comma_joined():
result, install = _install(["--api-allow-cidr", "203.0.113.4/32", "--api-allow-cidr", "198.51.100.0/24"])
assert result.exit_code == 0, result.output
assert _extra_vars(install)["k7_api_allow_cidrs"] == "203.0.113.4/32,198.51.100.0/24"
def test_bare_address_is_accepted_as_a_host_route():
result, install = _install(["--api-allow-cidr", "203.0.113.4"])
assert result.exit_code == 0, result.output
assert _extra_vars(install)["k7_api_allow_cidrs"] == "203.0.113.4"
def test_host_bits_set_are_accepted_strict_false():
result, install = _install(["--api-allow-cidr", "203.0.113.4/24"])
assert result.exit_code == 0, result.output
def test_invalid_cidr_fails_before_ansible():
result, install = _install(["--api-allow-cidr", "203.0.113.0/33"])
assert result.exit_code != 0
assert "is not a valid CIDR" in result.output
install.assert_not_called()
def test_garbage_cidr_fails_before_ansible():
result, install = _install(["--api-allow-cidr", "not-a-cidr"])
assert result.exit_code != 0
assert "is not a valid CIDR" in result.output
install.assert_not_called()
def test_flannel_is_rejected():
result, install = _install(["--cni", "flannel", "--api-allow-cidr", "203.0.113.4/32"])
assert result.exit_code != 0
assert "--api-allow-cidr requires the Cilium CNI" in result.output
install.assert_not_called()
def test_flannel_without_the_flag_still_installs():
result, install = _install(["--cni", "flannel"])
assert result.exit_code == 0, result.output
install.assert_called_once()
def test_allow_everything_warns_loudly():
result, install = _install(["--api-allow-cidr", "0.0.0.0/0"])
assert result.exit_code == 0, result.output
assert "allows every source address" in result.output
assert _extra_vars(install)["k7_api_allow_cidrs"] == "0.0.0.0/0"
def test_specific_cidr_does_not_warn():
result, _ = _install(["--api-allow-cidr", "203.0.113.4/32"])
assert result.exit_code == 0, result.output
assert "allows every source address" not in result.output
_PLAYBOOK = (Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml").read_text()
def test_playbook_policy_keeps_the_reserved_entities():
"""Dropping `host` from the rendered policy crash-loops the k7-api pod:
the kubelet liveness/readiness probes come from the node."""
assert '- fromEntities: ["host", "remote-node", "health", "kube-apiserver"]' in _PLAYBOOK
def test_playbook_only_switches_external_traffic_policy_with_an_allowlist():
assert "'Local' if k7_api_allow_cidrs_list | length > 0 else 'Cluster'" in _PLAYBOOK
+239
View File
@@ -0,0 +1,239 @@
"""Unit tests for ``k7 install`` TLS flags and the TLS manifest renderer."""
from __future__ import annotations
import subprocess
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
from typer.testing import CliRunner
from k7.cli.k7 import app
from k7.core.models import OperationResult
from k7.deploy.k7_api_tls import CADDY_IMAGE, CADDY_PIN, render_caddyfile, render_deployment, render_service
runner = CliRunner()
_PLAYBOOK = (Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml").read_text()
_HTTP_DEPLOYMENT = (Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api/deployment.yaml").read_text()
_HTTP_SERVICE = (Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api/service.yaml").read_text()
_CLUSTERROLE = (Path(__file__).resolve().parents[2] / "src/k7/deploy/manifests/k7-api/clusterrole.yaml").read_text()
def _install(args: list[str]):
with patch("k7.cli.k7.K7Core") as core_cls:
install = MagicMock(return_value=OperationResult(success=True, message="ok"))
core_cls.return_value.install_node = install
result = runner.invoke(app, ["--core", "install", *args])
return result, install
def _extra_vars(install: MagicMock) -> dict:
return install.call_args.kwargs["extra_vars"]
def test_default_install_is_https():
result, install = _install([])
assert result.exit_code == 0, result.output
ev = _extra_vars(install)
assert ev["k7_api_insecure_http"] == "false"
assert ev["k7_api_hostname"] == ""
assert ev["k7_api_tls_cert_src"] == ""
assert ev["k7_api_tls_key_src"] == ""
def test_insecure_http_is_forwarded():
result, install = _install(["--api-insecure-http"])
assert result.exit_code == 0, result.output
assert _extra_vars(install)["k7_api_insecure_http"] == "true"
def test_insecure_http_with_allowlist_warns():
result, install = _install(["--api-insecure-http", "--api-allow-cidr", "203.0.113.4/32"])
assert result.exit_code == 0, result.output
assert "keys still travel in cleartext" in result.output
ev = _extra_vars(install)
assert ev["k7_api_insecure_http"] == "true"
assert ev["k7_api_allow_cidrs"] == "203.0.113.4/32"
def test_hostname_and_insecure_http_fail_before_ansible():
result, install = _install(["--api-insecure-http", "--api-hostname", "api.example.com"])
assert result.exit_code != 0
assert "--api-insecure-http cannot be combined" in result.output
install.assert_not_called()
def test_hostname_and_operator_cert_fail_before_ansible(tmp_path: Path):
cert = tmp_path / "server.crt"
key = tmp_path / "server.key"
cert.write_text("x")
key.write_text("y")
result, install = _install(
["--api-hostname", "api.example.com", "--api-tls-cert", str(cert), "--api-tls-key", str(key)]
)
assert result.exit_code != 0
assert "--api-hostname cannot be combined" in result.output
install.assert_not_called()
def test_hostname_must_be_a_dns_name():
result, install = _install(["--api-hostname", "203.0.113.10"])
assert result.exit_code != 0
assert "must be a DNS name" in result.output
install.assert_not_called()
def test_cert_without_key_fails():
result, install = _install(["--api-tls-cert", "/tmp/server.crt"])
assert result.exit_code != 0
assert "must be passed together" in result.output
install.assert_not_called()
def test_mismatched_cert_key_fails_before_ansible(tmp_path: Path):
cert = tmp_path / "a.crt"
key = tmp_path / "b.key"
subprocess.run(
[
"openssl",
"req",
"-x509",
"-newkey",
"rsa:2048",
"-nodes",
"-days",
"1",
"-subj",
"/CN=a",
"-keyout",
str(tmp_path / "a.key"),
"-out",
str(cert),
],
check=True,
capture_output=True,
)
subprocess.run(
[
"openssl",
"req",
"-x509",
"-newkey",
"rsa:2048",
"-nodes",
"-days",
"1",
"-subj",
"/CN=b",
"-keyout",
str(key),
"-out",
str(tmp_path / "b.crt"),
],
check=True,
capture_output=True,
)
result, install = _install(["--api-tls-cert", str(cert), "--api-tls-key", str(key)])
assert result.exit_code != 0
assert "not a matching pair" in result.output
install.assert_not_called()
def test_matching_cert_key_is_forwarded(tmp_path: Path):
cert = tmp_path / "server.crt"
key = tmp_path / "server.key"
subprocess.run(
[
"openssl",
"req",
"-x509",
"-newkey",
"rsa:2048",
"-nodes",
"-days",
"1",
"-subj",
"/CN=k7-api",
"-keyout",
str(key),
"-out",
str(cert),
],
check=True,
capture_output=True,
)
result, install = _install(["--api-tls-cert", str(cert), "--api-tls-key", str(key)])
assert result.exit_code == 0, result.output
ev = _extra_vars(install)
assert ev["k7_api_tls_cert_src"] == str(cert.resolve())
assert ev["k7_api_tls_key_src"] == str(key.resolve())
def test_acme_staging_requires_hostname():
result, install = _install(["--api-acme-staging"])
assert result.exit_code != 0
assert "--api-acme-staging requires --api-hostname" in result.output
install.assert_not_called()
def test_renderer_adds_caddy_and_hides_ca_key():
out = render_deployment(_HTTP_DEPLOYMENT, acme=False)
assert "name: caddy" in out
assert CADDY_IMAGE in out
assert "secretName: k7-api-tls" in out
assert "mountPath: /etc/k7/tls" in out
assert "name: tls-hide" in out
assert "path: /health" in out
assert "port: 8000" in out
assert "ca.key" not in out
assert "hostPort: 80" not in out
def test_renderer_acme_exposes_host_port_80_and_no_tls_secret():
out = render_deployment(_HTTP_DEPLOYMENT, acme=True)
assert "hostPort: 80" in out
assert "secretName: k7-api-tls" not in out
def test_renderer_service_targets_8443():
assert "targetPort: 8443" in render_service(_HTTP_SERVICE)
assert "nodePort: 31007" in render_service(_HTTP_SERVICE)
def test_caddyfile_files_mode_disables_auto_https():
text = render_caddyfile(acme=False)
assert "auto_https off" in text
assert "tls /certs/tls.crt /certs/tls.key" in text
assert "reverse_proxy 127.0.0.1:8000" in text
def test_caddyfile_acme_uses_hostname_and_optional_staging():
text = render_caddyfile(acme=True, hostname="api.example.com", staging=True)
assert "api.example.com {" in text
assert "https_port 8443" in text
assert "http_port 80" in text
assert "acme-staging-v02" in text
with pytest.raises(ValueError, match="hostname"):
render_caddyfile(acme=True, hostname="")
def test_playbook_caddy_pin_matches_the_renderer():
assert f'k7_caddy_image: "{CADDY_PIN}"' in _PLAYBOOK
assert CADDY_IMAGE in render_deployment(_HTTP_DEPLOYMENT, acme=False)
def test_playbook_keeps_the_20d_reserved_entities():
assert '- fromEntities: ["host", "remote-node", "health", "kube-apiserver"]' in _PLAYBOOK
def test_playbook_does_not_enable_a_host_firewall():
assert "enable-host-firewall" not in _PLAYBOOK
assert "CiliumHostFirewall" not in _PLAYBOOK
def test_clusterrole_can_manage_expose_services():
# Delete always removes `{name}-expose`. Without this verb the API
# gets 403 (not 404) and every sandbox delete through k7-api is 400.
assert '"services"' in _CLUSTERROLE
+4 -4
View File
@@ -1,4 +1,4 @@
"""Unit tests for the ``k7 pause`` CLI command (Spec 10c).
"""Unit tests for the ``k7 pause`` CLI command.
These verify the cleaned-up CLI surface:
- ``k7 pause NAME`` → no snapshot, snapshot_name=None forwarded to Core.
@@ -31,7 +31,7 @@ def test_pause_no_snapshot():
patcher, fake = _mock_pause_returning_success()
try:
# ``--core`` forces the K7Core path so the mocked K7Core in
# ``_mock_pause_returning_success`` is exercised. Spec 10g default
# ``_mock_pause_returning_success`` is exercised. Default
# routing is through the HTTP API (see test_cli_routing.py).
result = runner.invoke(app, ["--core", "pause", "demo"])
finally:
@@ -106,7 +106,7 @@ def test_pause_with_namespace():
def test_pause_unknown_pvc_flag_rejected():
"""Spec 10c: --pvc was removed. Passing it must fail fast."""
"""--pvc was removed. Passing it must fail fast."""
result = runner.invoke(app, ["pause", "demo", "--pvc=demo-root-lh"])
assert result.exit_code != 0
combined = (result.output or "") + (result.stderr or "")
@@ -114,7 +114,7 @@ def test_pause_unknown_pvc_flag_rejected():
def test_pause_unknown_snapshot_class_flag_rejected():
"""Spec 10c: --snapshot-class was removed. Passing it must fail fast."""
"""--snapshot-class was removed. Passing it must fail fast."""
result = runner.invoke(app, ["pause", "demo", "--snapshot-class=longhorn"])
assert result.exit_code != 0
combined = (result.output or "") + (result.stderr or "")
+57 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for the Spec 10g CLI ↔ API client plumbing.
"""Unit tests for the CLI ↔ API client plumbing.
Coverage:
@@ -170,6 +170,62 @@ def test_resolve_client_exits_when_key_missing(isolated_config_dir, monkeypatch)
cli_client.resolve_client("https://flag.local:31000", None)
def test_resolve_api_ca_chain(isolated_config_dir, monkeypatch, tmp_path):
flag = tmp_path / "flag.pem"
envp = tmp_path / "env.pem"
cfgp = tmp_path / "cfg.pem"
etcp = tmp_path / "etc.pem"
for p in (flag, envp, cfgp, etcp):
p.write_text("ca")
monkeypatch.setattr(cli_client, "_ETC_API_CA", etcp)
assert cli_client._resolve_api_ca(str(flag)) == str(flag)
monkeypatch.setenv("K7_API_CA", str(envp))
assert cli_client._resolve_api_ca(None) == str(envp)
monkeypatch.delenv("K7_API_CA")
cli_config.set_config_value("api.ca", str(cfgp))
assert cli_client._resolve_api_ca(None) == str(cfgp)
# Config wins over /etc; wipe config by pointing at a fresh dir.
monkeypatch.setenv(cli_config.CONFIG_DIR_ENV, str(tmp_path / "empty-cfg"))
assert cli_client._resolve_api_ca(None) == str(etcp)
def test_verify_ssl_for_url_never_disables_https():
assert cli_client.verify_ssl_for_url("https://10.0.0.1:31007", None) is True
assert cli_client.verify_ssl_for_url("https://10.0.0.1:31007", "/etc/k7/tls/ca.crt") == "/etc/k7/tls/ca.crt"
assert cli_client.verify_ssl_for_url("http://10.0.0.1:31007", "/etc/k7/tls/ca.crt") is False
def test_resolve_client_https_uses_ca_path(isolated_config_dir, monkeypatch, tmp_path):
ca = tmp_path / "ca.crt"
ca.write_text("dummy")
monkeypatch.delenv("K7_API_CA", raising=False)
monkeypatch.setattr(cli_client, "_ETC_API_CA", tmp_path / "missing")
with patch("k7.cli._client.Client") as client_cls:
cli_client.resolve_client("https://10.0.0.1:31007", "k", str(ca))
assert client_cls.call_args.kwargs["verify_ssl"] == str(ca)
def test_resolve_client_https_without_ca_uses_system_store(isolated_config_dir, monkeypatch, tmp_path):
monkeypatch.delenv("K7_API_CA", raising=False)
monkeypatch.setattr(cli_client, "_ETC_API_CA", tmp_path / "missing")
with patch("k7.cli._client.Client") as client_cls:
cli_client.resolve_client("https://api.example.com:31007", "k")
assert client_cls.call_args.kwargs["verify_ssl"] is True
def test_resolve_client_http_disables_verify(isolated_config_dir, monkeypatch, tmp_path):
monkeypatch.delenv("K7_API_CA", raising=False)
monkeypatch.setattr(cli_client, "_ETC_API_CA", tmp_path / "missing")
with patch("k7.cli._client.Client") as client_cls:
cli_client.resolve_client("http://10.0.0.1:31007", "k")
assert client_cls.call_args.kwargs["verify_ssl"] is False
def test_config_accepts_api_ca(isolated_config_dir):
cli_config.set_config_value("api.ca", "/tmp/ca.crt")
assert cli_config.get_config_value("api.ca") == "/tmp/ca.crt"
# ---------------------------------------------------------------------------
# CLI routing — --core flag vs default API path.
# ---------------------------------------------------------------------------
+23
View File
@@ -1,6 +1,7 @@
"""Unit tests for pure CLI utility functions."""
import json
from pathlib import Path
from unittest.mock import MagicMock, patch
import pytest
@@ -75,6 +76,12 @@ class TestNormalizeBackend:
def test_k7_alias_maps_to_k7d(self):
assert _normalize_backend("k7") == "k7d"
def test_k7d_fc_accepted(self):
assert _normalize_backend("k7d-fc") == "k7d-fc"
def test_k7_fc_alias_maps_to_k7d_fc(self):
assert _normalize_backend("k7-fc") == "k7d-fc"
# --- _kubectl_cmd ---
@@ -109,6 +116,11 @@ _DUAL_STACK = json.dumps(
class TestReadApiEndpoint:
@pytest.fixture(autouse=True)
def _ignore_on_node_endpoint_file(self, tmp_path: Path):
with patch("k7.cli.k7._ETC_K7_API_ENDPOINT", tmp_path / "missing"):
yield
def test_returns_endpoint_on_success(self):
port_proc = MagicMock(returncode=0, stdout="31007")
node_proc = MagicMock(returncode=0, stdout=_SINGLE_STACK)
@@ -139,6 +151,14 @@ class TestReadApiEndpoint:
result = _read_api_endpoint(["kubectl"])
assert result == "http://localhost:31007"
def test_https_when_service_targets_8443(self):
port_proc = MagicMock(returncode=0, stdout="31007,8443")
node_proc = MagicMock(returncode=0, stdout=_SINGLE_STACK)
with patch("k7.cli.k7.subprocess.run", side_effect=[port_proc, node_proc]):
result = _read_api_endpoint(["kubectl"])
assert result == "https://10.0.0.1:31007"
# --- _build_default_inventory ---
@@ -240,6 +260,9 @@ class TestParseBackends:
def test_aliases(self):
assert _parse_backends("kfd,kql") == ["kata-firecracker-devmapper", "kata-qemu-longhorn"]
def test_k7d_fc_alias(self):
assert _parse_backends("k7d,k7-fc") == ["k7d", "k7d-fc"]
def test_deprecated_aliases(self):
assert _parse_backends("fd,ql") == ["kata-firecracker-devmapper", "kata-qemu-longhorn"]
+24 -2
View File
@@ -1,7 +1,7 @@
"""Unit tests for the k7_sdk client (sync + async).
These tests verify URL/body construction and response unwrapping for the
pause/resume/fork SDK additions from Spec 10a. The HTTP layer is mocked so
pause/resume/fork SDK additions. The HTTP layer is mocked so
the tests run without a live API.
"""
@@ -12,6 +12,23 @@ import pytest
from k7_sdk.client import AsyncClient, Client, SandboxProxy
def test_sync_client_accepts_ca_path(tmp_path):
ca = tmp_path / "ca.crt"
ca.write_text("dummy")
c = Client(endpoint="https://api.test", api_key="k", verify_ssl=str(ca))
assert c.session.verify == str(ca)
def test_sync_client_refuses_verify_false_on_https():
with pytest.raises(ValueError, match="not allowed for https"):
Client(endpoint="https://api.test", api_key="k", verify_ssl=False)
def test_sync_client_allows_verify_false_on_http():
c = Client(endpoint="http://api.test", api_key="k", verify_ssl=False)
assert c.session.verify is False
def _mock_response(payload: dict, status_code: int = 200) -> MagicMock:
r = MagicMock()
r.status_code = status_code
@@ -50,7 +67,7 @@ class TestSyncClientPause:
assert body == {"namespace": "ns", "snapshot": "snap1"}
def test_pause_does_not_accept_legacy_kwargs(self):
"""Spec 10c: pvc and snapshot_class were removed from the SDK surface."""
"""pvc and snapshot_class were removed from the SDK surface."""
import inspect
sig = inspect.signature(Client.pause)
@@ -178,6 +195,11 @@ class TestAsyncClient:
# --- Smoke: AsyncClient construction requires httpx --------------------------
def test_async_client_refuses_verify_false_on_https():
with pytest.raises(ValueError, match="not allowed for https"):
AsyncClient(endpoint="https://api.test", api_key="k", verify_ssl=False)
def test_async_client_requires_httpx(monkeypatch):
"""When httpx is unavailable, AsyncClient raises RuntimeError loudly."""
import k7_sdk.client as mod
+199 -3
View File
@@ -1,4 +1,4 @@
"""Unit tests for the k7d backend paths in K7Core (spec 9a M11).
"""Unit tests for the k7d backend paths in K7Core.
Everything here is mocked — the daemon socket, crictl, and the k8s API
are never touched. The real end-to-end behaviour is covered by
@@ -13,14 +13,15 @@ from k7.core.core import (
K7D_ANN_FORK_SOURCE_CLUSTER,
K7D_ANN_FORK_SOURCE_VM,
K7D_ANN_PAUSED,
K7D_FC_VIRTIOFS_REFUSED,
K7Core,
)
from k7.core.models import SandboxConfig
from k7.core.models import OperationResult, SandboxConfig
from tests.unit.conftest import mock_deployment, mock_pod
def _local_node(core: K7Core):
"""Patch the spec-18g node resolution so the sandbox appears LOCAL —
"""Patch the node resolution so the sandbox appears LOCAL —
these tests exercise the direct daemon-socket path, not forwarding."""
return patch.object(core, "_k7d_sandbox_node", new=AsyncMock(return_value=core._k7d_local_node()))
@@ -55,6 +56,20 @@ class TestK7dBackendDetection:
assert await core._detect_backend("test-sb", "default") == "k7d"
def test_k7_fc_alias_canonicalizes_to_k7d_fc(self):
assert K7Core._canonicalize_backend("k7-fc") == "k7d-fc"
def test_k7d_fc_passes_through(self):
assert K7Core._canonicalize_backend("k7d-fc") == "k7d-fc"
async def test_detect_backend_accepts_k7d_fc_annotation(self, core: K7Core):
dep = mock_deployment(annotations={"k7.katakate.org/backend": "k7d-fc"})
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
core._apps_v1_client = mock_apps
assert await core._detect_backend("test-sb", "default") == "k7d-fc"
# --- create_sandbox rendering ---
@@ -86,6 +101,62 @@ class TestK7dCreateSandbox:
# Deployment carries the backend annotation for later detection.
assert deployment.metadata.annotations["k7.katakate.org/backend"] == "k7d"
async def test_create_k7d_fc_uses_runtime_class_k7_fc(self, core: K7Core):
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
mock_net = AsyncMock()
core._apps_v1_client = mock_apps
core._core_v1_client = mock_v1
core._networking_v1_client = mock_net
sched_ok = MagicMock()
sched_ok.success = True
with patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)):
cfg = SandboxConfig(name="k7d-fc-sb", image="alpine:3.20", backend="k7d-fc")
result = await core.create_sandbox(cfg)
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
pod_spec = deployment.spec.template.spec
assert pod_spec.runtime_class_name == "k7-fc"
assert pod_spec.node_selector == {"k7.katakate.org/backend-k7d-fc": "true"}
assert deployment.metadata.annotations["k7.katakate.org/backend"] == "k7d-fc"
async def test_create_k7d_fc_refuses_hostpath_before_pod(self, core: K7Core):
core._apps_v1_client = AsyncMock()
core._core_v1_client = AsyncMock()
core._networking_v1_client = AsyncMock()
cfg = SandboxConfig(
name="k7d-fc-hp",
image="alpine:3.20",
backend="k7d-fc",
volumes=[{"name": "hp", "hostPath": {"path": "/var/log"}}],
)
result = await core.create_sandbox(cfg)
assert not result.success
assert result.error == K7D_FC_VIRTIOFS_REFUSED
core._apps_v1_client.create_namespaced_deployment.assert_not_called()
async def test_docker_stamps_annotations_on_k7d_fc(self, core: K7Core):
mock_apps = AsyncMock()
core._apps_v1_client = mock_apps
core._core_v1_client = AsyncMock()
core._networking_v1_client = AsyncMock()
sched_ok = MagicMock()
sched_ok.success = True
with (
patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)),
patch("k7.core.core.k7d_supports_docker", return_value=True),
):
cfg = SandboxConfig(name="k7d-fc-dock", image="ubuntu:24.04", backend="k7-fc", docker=True)
result = await core.create_sandbox(cfg)
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
assert deployment.spec.template.spec.runtime_class_name == "k7-fc"
assert deployment.spec.template.metadata.annotations["k7d.katakate.org/docker"] == "true"
async def test_create_skips_kata_memory_annotation(self, core: K7Core):
mock_apps = AsyncMock()
core._apps_v1_client = mock_apps
@@ -108,6 +179,70 @@ class TestK7dCreateSandbox:
container = deployment.spec.template.spec.containers[0]
assert container.resources.limits == {"memory": "512Mi", "cpu": "1"}
async def test_docker_stamps_annotations_no_sidecar_no_privileged(self, core: K7Core):
mock_apps = AsyncMock()
core._apps_v1_client = mock_apps
core._core_v1_client = AsyncMock()
core._networking_v1_client = AsyncMock()
sched_ok = MagicMock()
sched_ok.success = True
with (
patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched_ok)),
patch("k7.core.core.k7d_supports_docker", return_value=True),
):
cfg = SandboxConfig(
name="k7d-dock",
image="ubuntu:24.04",
backend="k7d",
docker=True,
docker_disk="40Gi",
)
result = await core.create_sandbox(cfg)
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
ann = deployment.metadata.annotations
pod_ann = deployment.spec.template.metadata.annotations
assert ann["k7.katakate.org/docker"] == "true"
assert pod_ann["k7d.katakate.org/docker"] == "true"
assert pod_ann["k7d.katakate.org/docker-disk"] == "40Gi"
assert "k7.katakate.org/sidecar" not in ann
assert "k7.katakate.org/sidecar" not in pod_ann
containers = deployment.spec.template.spec.containers
assert [c.name for c in containers] == ["sandbox"]
sandbox = containers[0]
assert sandbox.security_context.privileged is not True
assert sandbox.security_context.allow_privilege_escalation is False
assert sandbox.security_context.capabilities.drop == ["ALL"]
assert sandbox.security_context.seccomp_profile.type == "RuntimeDefault"
async def test_docker_against_old_k7d_fails_loud(self, core: K7Core):
core._apps_v1_client = AsyncMock()
core._core_v1_client = AsyncMock()
core._networking_v1_client = AsyncMock()
with patch("k7.core.core.k7d_supports_docker", return_value=False):
cfg = SandboxConfig(name="k7d-old", image="ubuntu:24.04", backend="k7d", docker=True)
result = await core.create_sandbox(cfg)
assert not result.success
assert "no docker service" in result.error
assert "upgrade" in result.error
async def test_docker_and_sidecar_mutually_exclusive(self, core: K7Core):
core._apps_v1_client = AsyncMock()
core._core_v1_client = AsyncMock()
core._networking_v1_client = AsyncMock()
cfg = SandboxConfig(
name="both",
image="ubuntu:24.04",
backend="k7d",
docker=True,
sidecar="docker",
)
result = await core.create_sandbox(cfg)
assert not result.success
assert "mutually exclusive" in result.error
# --- pause / resume via the daemon socket ---
@@ -135,6 +270,22 @@ class TestK7dPauseResume:
patch_body = mock_apps.patch_namespaced_deployment.call_args.kwargs["body"]
assert patch_body["metadata"]["annotations"][K7D_ANN_PAUSED] == "true"
async def test_pause_k7d_fc_uses_daemon_socket(self, core: K7Core):
mock_apps = AsyncMock()
dep = mock_deployment(name="fc-sb", runtime_class="k7-fc", annotations={"k7.katakate.org/backend": "k7d-fc"})
mock_apps.read_namespaced_deployment.return_value = dep
core._apps_v1_client = mock_apps
vm = {"vm_id": "vm-fc-0", "guest_cid": 88, "sandbox_id": "cri-fc"}
with (
_local_node(core),
patch.object(core, "_k7d_vm_for_sandbox", new=AsyncMock(return_value=vm)),
patch.object(core, "_k7d_request", new=AsyncMock(return_value={"status": "ok"})) as request,
):
result = await core.pause_sandbox("fc-sb")
assert result.success, result.error
request.assert_awaited_once_with({"op": "pause_vm", "vm_id": "vm-fc-0"})
mock_apps.patch_namespaced_deployment_scale.assert_not_called()
async def test_pause_with_snapshot_is_loud_error(self, core: K7Core):
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = _k7d_deployment("k7d-sb")
@@ -196,10 +347,22 @@ class TestK7dFork:
patch.object(core, "_k7d_vm_for_sandbox", new=AsyncMock(return_value=vm)),
patch.object(core, "_k7d_running_pod", new=AsyncMock(return_value=src_pod)),
patch.object(core, "_wait_for_pod_container_started", new=AsyncMock(return_value="dst-sb-xyz")),
patch.object(
core, "_read_sandbox_egress_whitelist", new=AsyncMock(return_value=["example.com"])
) as read_egress,
patch.object(core, "_read_sandbox_ingress_rules", new=AsyncMock(return_value=([8000], ["sandbox:a"]))),
patch.object(
core, "_apply_sandbox_network_policies", new=AsyncMock(return_value=OperationResult(True))
) as apply_policies,
):
result = await core.fork_sandbox("src-sb", "dst-sb")
assert result.success, result.error
# The warm fork inherits the source's policy too.
read_egress.assert_awaited_once_with("src-sb", "default")
assert apply_policies.call_args.kwargs["egress_whitelist"] == ["example.com"]
assert apply_policies.call_args.kwargs["ingress_ports"] == [8000]
assert apply_policies.call_args.kwargs["ingress_from"] == ["sandbox:a"]
assert "warm fork" in result.message
new_dep = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
assert new_dep.metadata.name == "dst-sb"
@@ -219,8 +382,41 @@ class TestK7dFork:
result = await core.fork_sandbox("src-sb", "dst-sb")
assert not result.success
assert "sidecar" in result.error
assert "real CRI sidecar" in result.error
mock_apps.create_namespaced_deployment.assert_not_called()
async def test_fork_docker_annotation_alone_is_not_a_sidecar(self, core: K7Core):
mock_apps = AsyncMock()
annotations = {
"k7.katakate.org/backend": "k7d",
"k7.katakate.org/docker": "true",
"k7d.katakate.org/docker": "true",
}
dep = mock_deployment(name="src-sb", runtime_class="k7", annotations=annotations)
dep.spec.template.metadata.annotations = dict(annotations)
dep.spec.template.metadata.labels = {"app": "src-sb", "katakate.org/sandbox": "src-sb"}
dep.metadata.labels = {"app": "src-sb", "runtime": "kata", "katakate.org/sandbox": "src-sb"}
dep.spec.selector.match_labels = {"app": "src-sb"}
mock_apps.read_namespaced_deployment.return_value = dep
core._apps_v1_client = mock_apps
vm = {"vm_id": "vm-1-0", "guest_cid": 77, "sandbox_id": "cri-src", "cluster_id": "cri-src"}
src_pod = mock_pod(name="src-sb-abc")
src_pod.spec.node_name = "k7-node-01"
with (
_local_node(core),
patch.object(core, "_k7d_vm_for_sandbox", new=AsyncMock(return_value=vm)),
patch.object(core, "_k7d_running_pod", new=AsyncMock(return_value=src_pod)),
patch.object(core, "_wait_for_pod_container_started", new=AsyncMock(return_value="dst-sb-xyz")),
patch.object(core, "_read_sandbox_egress_whitelist", new=AsyncMock(return_value=[])),
patch.object(core, "_read_sandbox_ingress_rules", new=AsyncMock(return_value=([], []))),
patch.object(core, "_apply_sandbox_network_policies", new=AsyncMock(return_value=OperationResult(True))),
):
result = await core.fork_sandbox("src-sb", "dst-sb")
assert result.success, result.error
mock_apps.create_namespaced_deployment.assert_called_once()
async def test_fork_with_snapshot_name_is_loud_error(self, core: K7Core):
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = _k7d_deployment("src-sb")
+330 -11
View File
@@ -94,10 +94,10 @@ class TestDeleteSandbox:
assert result.success is True
mock_apps.delete_namespaced_deployment.assert_called_once_with(name="my-sb", namespace="default")
mock_v1.delete_namespaced_secret.assert_called_once_with(name="my-sb-env", namespace="default")
mock_v1.delete_namespaced_config_map.assert_called_once_with(name="my-sb-persist-wrapper", namespace="default")
mock_v1.delete_namespaced_persistent_volume_claim.assert_called_once_with(
name="my-sb-root-lh", namespace="default"
)
mock_v1.delete_namespaced_config_map.assert_any_call(name="my-sb-persist-wrapper", namespace="default")
mock_v1.delete_namespaced_config_map.assert_any_call(name="my-sb-docker-vehicle", namespace="default")
mock_v1.delete_namespaced_persistent_volume_claim.assert_any_call(name="my-sb-root-lh", namespace="default")
mock_v1.delete_namespaced_persistent_volume_claim.assert_any_call(name="my-sb-docker-lh", namespace="default")
async def test_delete_ignores_404s(self, core: K7Core):
mock_apps = AsyncMock()
@@ -383,11 +383,11 @@ class TestPauseSandbox:
assert snap_mock.call_args.kwargs["snapshot_name"] == "snap1"
async def test_snapshot_only_defaults_pvc_to_root(self, core: K7Core):
"""Spec 10c regression: snapshot_name alone triggers snapshot.
"""Regression: snapshot_name alone triggers snapshot.
Pre-10c the snapshot only fired when ``pvc_name AND snapshot_name`` were
The snapshot used to fire only when ``pvc_name AND snapshot_name`` were
both set, so ``k7 pause foo --snapshot bar`` silently did nothing.
After 10c, ``snapshot_name`` alone is sufficient and ``pvc_name``
Now ``snapshot_name`` alone is sufficient and ``pvc_name``
defaults to ``_root_pvc_name(name)``.
"""
mock_apps = AsyncMock()
@@ -536,7 +536,12 @@ class TestForkSandbox:
"""Set up mock clients for fork tests and return them."""
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
_setup_clients(core, apps=mock_apps, v1=mock_v1)
# Source has no egress policy at all → the fork inherits open egress.
mock_net = AsyncMock()
mock_net.read_namespaced_network_policy.side_effect = ApiException(status=404)
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
_setup_clients(core, apps=mock_apps, v1=mock_v1, net=mock_net, custom=mock_custom)
src_dep = MagicMock()
src_dep.metadata.name = "src"
@@ -671,8 +676,305 @@ class TestForkSandbox:
assert "Forked" in result.message
mock_apps.create_namespaced_deployment.assert_called_once()
def _patch_fork_storage(self, core: K7Core):
return (
patch.object(core, "_create_volume_snapshot", new_callable=AsyncMock, return_value=OperationResult(True)),
patch.object(core, "_wait_for_snapshot_ready", new_callable=AsyncMock, return_value=OperationResult(True)),
patch.object(core, "_wait_for_pvc_bound", new_callable=AsyncMock, return_value=OperationResult(True)),
)
# --- spec 18g: per-node agent forwarding ---
async def test_fork_applies_network_policies(self, core: K7Core):
"""a fork used to come up with no policy at all."""
self._setup_fork(core)
core._custom_objects_client.get_namespaced_custom_object.side_effect = None
core._custom_objects_client.get_namespaced_custom_object.return_value = {
"spec": {"egress": [{"toFQDNs": [{"matchName": "example.com"}]}, {"toCIDR": ["10.0.0.0/8"]}]}
}
snap, ready, bound = self._patch_fork_storage(core)
with (
snap,
ready,
bound,
patch.object(
core,
"_apply_sandbox_network_policies",
new_callable=AsyncMock,
return_value=OperationResult(True),
) as apply_policies,
):
result = await core.fork_sandbox("src", "dst")
assert result.success is True
assert apply_policies.call_args.kwargs["name"] == "dst"
assert apply_policies.call_args.kwargs["egress_whitelist"] == ["example.com", "10.0.0.0/8"]
async def test_fork_of_open_egress_source_inherits_open_egress(self, core: K7Core):
self._setup_fork(core)
snap, ready, bound = self._patch_fork_storage(core)
with (
snap,
ready,
bound,
patch.object(
core,
"_apply_sandbox_network_policies",
new_callable=AsyncMock,
return_value=OperationResult(True),
) as apply_policies,
):
result = await core.fork_sandbox("src", "dst")
assert result.success is True
assert apply_policies.call_args.kwargs["egress_whitelist"] is None
async def test_fork_policy_failure_rolls_back_deployment(self, core: K7Core):
self._setup_fork(core)
snap, ready, bound = self._patch_fork_storage(core)
with (
snap,
ready,
bound,
patch.object(
core,
"_apply_sandbox_network_policies",
new_callable=AsyncMock,
return_value=OperationResult(False, error="boom"),
),
patch.object(
core,
"_delete_sandbox_resources",
new_callable=AsyncMock,
return_value=OperationResult(True),
) as rollback,
):
result = await core.fork_sandbox("src", "dst")
assert result.success is False
assert "rolled back" in result.error and "boom" in result.error
rollback.assert_awaited_once_with("dst", "default")
class TestApplySandboxNetworkPolicies:
"""opt-in ingress, deny-all by default."""
def _net(self, core: K7Core):
mock_net = AsyncMock()
_setup_clients(core, net=mock_net)
return mock_net
def _ingress_body(self, mock_net):
for call in mock_net.create_namespaced_network_policy.call_args_list:
body = call.kwargs["body"]
if body.metadata.name.endswith("-deny-ingress"):
return body
raise AssertionError("no deny-ingress policy created")
async def test_default_is_deny_all(self, core: K7Core):
mock_net = self._net(core)
result = await core._apply_sandbox_network_policies(name="sb", namespace="default", egress_whitelist=None)
assert result.success is True
body = self._ingress_body(mock_net)
assert body.spec.policy_types == ["Ingress"]
assert body.spec.ingress == []
assert body.spec.pod_selector.match_labels == {"katakate.org/sandbox": "sb"}
async def test_ports_without_sources_allow_same_namespace_sandboxes(self, core: K7Core):
mock_net = self._net(core)
await core._apply_sandbox_network_policies(
name="sb", namespace="default", egress_whitelist=None, ingress_ports=[8000, 8001]
)
rule = self._ingress_body(mock_net).spec.ingress[0]
assert [(p.protocol, p.port) for p in rule.ports] == [("TCP", 8000), ("TCP", 8001)]
expr = rule._from[0].pod_selector.match_expressions[0]
assert (expr.key, expr.operator) == ("katakate.org/sandbox", "Exists")
async def test_explicit_sources_are_scoped(self, core: K7Core):
mock_net = self._net(core)
await core._apply_sandbox_network_policies(
name="sb",
namespace="default",
egress_whitelist=None,
ingress_ports=[8000],
ingress_from=["sandbox:alice", "cidr:203.0.113.0/24"],
)
peers = self._ingress_body(mock_net).spec.ingress[0]._from
assert peers[0].pod_selector.match_labels == {"katakate.org/sandbox": "alice"}
assert peers[1].ip_block.cidr == "203.0.113.0/24"
async def test_bad_source_fails_without_creating_the_policy(self, core: K7Core):
mock_net = self._net(core)
result = await core._apply_sandbox_network_policies(
name="sb", namespace="default", egress_whitelist=None, ingress_ports=[8000], ingress_from=["nonsense"]
)
assert result.success is False
assert "nonsense" in result.error
mock_net.create_namespaced_network_policy.assert_not_called()
async def test_policy_name_is_unchanged_when_it_carries_allow_rules(self, core: K7Core):
"""Renaming it would strand policies created by older k7 versions,
which `_delete_sandbox_resources` deletes by name."""
mock_net = self._net(core)
await core._apply_sandbox_network_policies(
name="sb", namespace="default", egress_whitelist=None, ingress_ports=[8000]
)
assert self._ingress_body(mock_net).metadata.name == "sb-deny-ingress"
class TestExposeSandboxPorts:
"""NodePort Service in front of matching ingress rules."""
async def test_service_uses_local_external_traffic_policy(self, core: K7Core):
mock_v1 = AsyncMock()
created = MagicMock()
port = MagicMock()
port.port = 8000
port.node_port = 31234
created.spec.ports = [port]
mock_v1.create_namespaced_service.return_value = created
_setup_clients(core, v1=mock_v1)
result = await core._apply_sandbox_expose_service("sb", "default", [8000])
assert result.success is True
assert result.data == {"node_ports": {8000: 31234}}
body = mock_v1.create_namespaced_service.call_args.kwargs["body"]
assert body.metadata.name == "sb-expose"
assert body.spec.type == "NodePort"
# Without Local the client's source IP is SNAT'd and cidr: rules are theatre.
assert body.spec.external_traffic_policy == "Local"
assert body.spec.selector == {"app": "sb"}
async def test_unallocated_node_port_is_a_loud_failure(self, core: K7Core):
mock_v1 = AsyncMock()
created = MagicMock()
port = MagicMock()
port.port = 8000
port.node_port = None
created.spec.ports = [port]
mock_v1.create_namespaced_service.return_value = created
_setup_clients(core, v1=mock_v1)
result = await core._apply_sandbox_expose_service("sb", "default", [8000])
assert result.success is False
assert "no node port" in result.error
async def test_expose_without_matching_ingress_port_is_rejected(self, core: K7Core):
mock_apps = AsyncMock()
_setup_clients(core, apps=mock_apps, v1=AsyncMock(), net=AsyncMock())
cfg = SandboxConfig(name="sb1", image="alpine:3.20", ingress_ports=[8000], expose_ports=[9000])
result = await core.create_sandbox(cfg)
assert result.success is False
assert "[9000]" in result.error
mock_apps.create_namespaced_deployment.assert_not_called()
class TestReadSandboxIngressRules:
async def test_deny_all_reads_back_as_none(self, core: K7Core):
np = MagicMock()
np.spec.ingress = []
mock_net = AsyncMock()
mock_net.read_namespaced_network_policy.return_value = np
_setup_clients(core, net=mock_net)
assert await core._read_sandbox_ingress_rules("sb", "default") == (None, None)
async def test_round_trips_ports_and_sources(self, core: K7Core):
"""A fork must re-derive the source's policy from what it reads back."""
core_net = AsyncMock()
_setup_clients(core, net=core_net)
await core._apply_sandbox_network_policies(
name="sb",
namespace="default",
egress_whitelist=None,
ingress_ports=[8000],
ingress_from=["sandbox:alice", "namespace:team-a", "cidr:203.0.113.0/24"],
)
applied = [
c.kwargs["body"]
for c in core_net.create_namespaced_network_policy.call_args_list
if c.kwargs["body"].metadata.name == "sb-deny-ingress"
][0]
core_net.read_namespaced_network_policy.return_value = applied
ports, sources = await core._read_sandbox_ingress_rules("sb", "default")
assert ports == [8000]
assert sources == ["sandbox:alice", "namespace:team-a", "cidr:203.0.113.0/24"]
async def test_default_source_reads_back_as_empty_list(self, core: K7Core):
core_net = AsyncMock()
_setup_clients(core, net=core_net)
await core._apply_sandbox_network_policies(
name="sb", namespace="default", egress_whitelist=None, ingress_ports=[8000]
)
applied = [
c.kwargs["body"]
for c in core_net.create_namespaced_network_policy.call_args_list
if c.kwargs["body"].metadata.name == "sb-deny-ingress"
][0]
core_net.read_namespaced_network_policy.return_value = applied
assert await core._read_sandbox_ingress_rules("sb", "default") == ([8000], [])
class TestReadSandboxEgressWhitelist:
async def test_cilium_wildcard_round_trips(self, core: K7Core):
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.return_value = {
"spec": {"egress": [{"toFQDNs": [{"matchPattern": "**.docker.com"}, {"matchName": "docker.io"}]}]}
}
_setup_clients(core, custom=mock_custom)
assert await core._read_sandbox_egress_whitelist("sb", "default") == ["*.docker.com", "docker.io"]
async def test_v1_netpol_cidrs(self, core: K7Core):
np = MagicMock()
peer = MagicMock()
peer.ip_block.cidr = "10.0.0.0/8"
rule = MagicMock()
rule.to = [peer]
np.spec.egress = [rule]
mock_net = AsyncMock()
mock_net.read_namespaced_network_policy.return_value = np
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
_setup_clients(core, net=mock_net, custom=mock_custom)
assert await core._read_sandbox_egress_whitelist("sb", "default") == ["10.0.0.0/8"]
async def test_block_all_egress_round_trips_as_empty_list(self, core: K7Core):
"""`egress_whitelist=[]` (block everything) must not degrade to None
(open) when a fork inherits it."""
np = MagicMock()
np.spec.egress = []
mock_net = AsyncMock()
mock_net.read_namespaced_network_policy.return_value = np
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
_setup_clients(core, net=mock_net, custom=mock_custom)
assert await core._read_sandbox_egress_whitelist("sb", "default") == []
async def test_no_policy_means_open_egress(self, core: K7Core):
mock_net = AsyncMock()
mock_net.read_namespaced_network_policy.side_effect = ApiException(status=404)
mock_custom = AsyncMock()
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
_setup_clients(core, net=mock_net, custom=mock_custom)
assert await core._read_sandbox_egress_whitelist("sb", "default") is None
# --- per-node agent forwarding ---
class TestK7dAgentForwarding:
@@ -1084,6 +1386,24 @@ class TestCreateSandboxValidation:
assert result.success is False
assert "empty or invalid" in result.error
async def test_ingress_from_without_ports_rejected(self, core: K7Core):
_setup_clients(core, apps=AsyncMock(), v1=AsyncMock(), net=AsyncMock())
cfg = SandboxConfig(name="sb1", image="alpine:3.18", ingress_from=["sandbox:alice"])
result = await core.create_sandbox(cfg)
assert result.success is False
assert "ingress_from requires ingress_ports" in result.error
async def test_unparseable_ingress_source_rejected_before_provisioning(self, core: K7Core):
mock_apps = AsyncMock()
_setup_clients(core, apps=mock_apps, v1=AsyncMock(), net=AsyncMock())
cfg = SandboxConfig(name="sb1", image="alpine:3.18", ingress_ports=[8000], ingress_from=["nonsense"])
result = await core.create_sandbox(cfg)
assert result.success is False
assert "nonsense" in result.error
mock_apps.create_namespaced_deployment.assert_not_called()
# --- _wait_for_pod_container_started ---
@@ -1351,8 +1671,7 @@ class TestApplyCiliumEgressPolicy:
assert {"matchName": "api.openai.com"} in fqdn_rule["toFQDNs"]
# Leading `*.` must be translated to Cilium's multi-label wildcard
# `**.` — a single `*` never crosses label boundaries, silently
# breaking nested subdomains like production.cloudfront.docker.com
# (spec 18f issue 3).
# breaking nested subdomains like production.cloudfront.docker.com.
assert {"matchPattern": "**.huggingface.co"} in fqdn_rule["toFQDNs"]
assert any("toCIDR" in r and r["toCIDR"] == ["10.0.0.0/8"] for r in egress)
# DNS proxy allowance is present
+32 -1
View File
@@ -2,7 +2,7 @@
import pytest
from k7.core.core import K7Core, _classify_egress_entries
from k7.core.core import K7Core, _classify_egress_entries, _classify_ingress_sources
# --- _memory_limit_to_mib ---
@@ -268,3 +268,34 @@ class TestClassifyEgressEntries:
cidrs, fqdns = _classify_egress_entries(["10.0.0.5/8"])
assert cidrs == ["10.0.0.0/8"]
assert fqdns == []
# --- _classify_ingress_sources ---
class TestClassifyIngressSources:
def test_empty_list(self):
assert _classify_ingress_sources([]) == []
def test_sandbox_becomes_pod_selector(self):
peers = _classify_ingress_sources(["sandbox:alice"])
assert peers[0].pod_selector.match_labels == {"katakate.org/sandbox": "alice"}
assert peers[0].namespace_selector is None
assert peers[0].ip_block is None
def test_namespace_becomes_namespace_selector(self):
peers = _classify_ingress_sources(["namespace:team-a"])
assert peers[0].namespace_selector.match_labels == {"kubernetes.io/metadata.name": "team-a"}
def test_cidr_becomes_ip_block_and_is_normalized(self):
peers = _classify_ingress_sources(["cidr:10.0.0.5/8"])
assert peers[0].ip_block.cidr == "10.0.0.0/8"
@pytest.mark.parametrize(
"entry",
["nonsense", "sandbox:", "pod:alice", "cidr:not-a-cidr", "10.0.0.0/8"],
)
def test_unparseable_source_is_a_hard_error(self, entry: str):
"""No default interpretation: an unreadable security rule must fail loudly."""
with pytest.raises(ValueError):
_classify_ingress_sources([entry])
+40
View File
@@ -0,0 +1,40 @@
"""Unit tests: k7 user-facing Docker pins match the pinned k7d snapshot."""
from pathlib import Path
import pytest
from k7.core.docker import (
k7d_supports_docker,
parse_docker_disk,
pins_from_env_text,
user_facing_pins,
)
_SNAPSHOT = Path(__file__).parent / "fixtures" / "k7d-guest-docker-pins.env"
def test_user_facing_pins_match_pinned_k7d_snapshot():
"""Fail loud if src/k7/core/docker.py drifts from k7d guest/docker/pins.env."""
snapshot = pins_from_env_text(_SNAPSHOT.read_text(encoding="utf-8"))
user = user_facing_pins()
missing = [k for k in user if k not in snapshot]
assert not missing, f"snapshot missing pins {missing}"
drifted = {k: (user[k], snapshot[k]) for k in user if snapshot[k] != user[k]}
assert not drifted, f"k7 docker.py pins drifted from pinned k7d pins.env: {drifted}"
def test_parse_docker_disk_accepts_gi():
assert parse_docker_disk("40Gi") == "40Gi"
assert parse_docker_disk(" 20Gi ") == "20Gi"
def test_parse_docker_disk_rejects_garbage():
with pytest.raises(ValueError, match="invalid"):
parse_docker_disk("lots")
def test_k7d_supports_docker_old_recorded_version(tmp_path: Path):
version_file = tmp_path / "k7d_version"
version_file.write_text("0.2.1\n")
assert k7d_supports_docker(str(version_file)) is False
+124
View File
@@ -0,0 +1,124 @@
"""Unit tests for ``k8s-object-presence.sh``.
Unknown (timeout / 503 / unable to handle) must never classify as absent,
or the playbook would run ``cilium install`` on a live cluster.
"""
import os
import stat
import subprocess
from pathlib import Path
HELPER = Path(__file__).resolve().parents[2] / "src/k7/deploy/k8s-object-presence.sh"
PLAYBOOK = Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml"
def _run(kubectl: Path, retries: str = "3", delay: str = "0") -> subprocess.CompletedProcess[str]:
env = os.environ.copy()
env["K7_KUBECTL"] = str(kubectl)
env["K7_PRESENCE_RETRIES"] = retries
env["K7_PRESENCE_DELAY"] = delay
return subprocess.run(
["bash", str(HELPER), "kube-system", "daemonset", "cilium"],
capture_output=True,
text=True,
env=env,
check=False,
)
def _write_kubectl(tmp_path: Path, body: str) -> Path:
path = tmp_path / "kubectl"
path.write_text("#!/bin/bash\n" + body)
path.chmod(path.stat().st_mode | stat.S_IEXEC)
return path
def test_helper_is_executable_script():
assert HELPER.is_file()
assert HELPER.stat().st_mode & stat.S_IXUSR
def test_present_on_success(tmp_path: Path):
kubectl = _write_kubectl(tmp_path, "echo NAME READY; exit 0\n")
result = _run(kubectl)
assert result.returncode == 0
assert result.stdout.strip() == "present"
def test_absent_on_not_found(tmp_path: Path):
kubectl = _write_kubectl(
tmp_path,
"echo 'Error from server (NotFound): daemonsets.apps \"cilium\" not found' >&2; exit 1\n",
)
result = _run(kubectl)
assert result.returncode == 0
assert result.stdout.strip() == "absent"
def test_unknown_on_unable_to_handle_does_not_look_absent(tmp_path: Path):
kubectl = _write_kubectl(
tmp_path,
'echo "The connection to the server localhost:6443 was refused" >&2; exit 1\n',
)
result = _run(kubectl, retries="2", delay="0")
assert result.returncode == 2
assert "present" not in result.stdout
assert "absent" not in result.stdout
assert "unknown" in result.stderr
def test_unknown_on_timeout(tmp_path: Path):
kubectl = _write_kubectl(
tmp_path, 'echo "Unable to connect to the server: net/http: request canceled" >&2; exit 1\n'
)
result = _run(kubectl, retries="2", delay="0")
assert result.returncode == 2
assert result.stdout.strip() != "absent"
def test_unknown_on_503(tmp_path: Path):
kubectl = _write_kubectl(
tmp_path,
'echo "Error from server (ServiceUnavailable): the server is currently unable to handle the request" >&2; exit 1\n',
)
result = _run(kubectl, retries="2", delay="0")
assert result.returncode == 2
assert result.stdout.strip() != "absent"
def test_retries_then_present(tmp_path: Path):
state = tmp_path / "n"
state.write_text("0")
kubectl = _write_kubectl(
tmp_path,
f"""
n=$(cat '{state}')
n=$((n + 1))
echo "$n" > '{state}'
if [ "$n" -lt 3 ]; then
echo "unable to handle the request" >&2
exit 1
fi
exit 0
""",
)
result = _run(kubectl, retries="5", delay="0")
assert result.returncode == 0
assert result.stdout.strip() == "present"
def test_playbook_installs_cilium_only_when_absent():
text = PLAYBOOK.read_text()
assert "cilium_installed_check.rc != 0" not in text
assert 'cilium_presence.stdout == "absent"' in text
assert 'hubble_presence.stdout == "absent"' in text
assert "cilium hubble enable" in text
def test_playbook_does_not_fail_reinstall_on_status_warnings_alone():
text = PLAYBOOK.read_text()
assert "Confirm Cilium agent is Ready (reinstall)" in text
assert 'cilium_presence.stdout == "present"' in text
# First-install wait is still the strict --wait path.
assert "Wait for Cilium to report ready (first install)" in text
+284
View File
@@ -0,0 +1,284 @@
"""Unit tests for the Kata docker vehicle."""
from unittest.mock import AsyncMock, MagicMock, patch
from kubernetes_asyncio.client.exceptions import ApiException
from k7.core.core import K7Core
from k7.core.docker import (
ANN_DOCKER_PVC,
ANN_K7_DOCKER,
CLI_PATH,
CLI_STAGING_DIR,
DIND_IMAGE,
DOCKER_HOST_URL,
GRAPH_DEVICE_PATH,
KATA_FORK_GRAPH_REJECT,
KATA_SOCKET_DIR,
KFD_DOCKER_STORAGE_CLASS,
VEHICLE_CONTAINER_NAME,
)
from k7.core.models import ExecResult, SandboxConfig
def _setup(core, apps=None, v1=None, net=None):
core._config_loaded = True
if apps is not None:
core._apps_v1_client = apps
if v1 is not None:
core._core_v1_client = v1
if net is not None:
core._networking_v1_client = net
def _ok():
r = MagicMock()
r.success = True
r.error = ""
r.data = {"created": True, "cm_name": "x"}
return r
async def _create_kata_docker(core: K7Core, backend: str):
mock_apps = AsyncMock()
mock_v1 = AsyncMock()
mock_v1.read_namespaced_persistent_volume_claim.side_effect = ApiException(status=404)
mock_v1.read_namespaced_config_map.side_effect = ApiException(status=404)
mock_net = AsyncMock()
_setup(core, apps=mock_apps, v1=mock_v1, net=mock_net)
sched = _ok()
overlay = _ok()
with (
patch.object(core, "_check_scheduling", new=AsyncMock(return_value=sched)),
patch.object(core, "_wait_kata_docker_overlay2", new=AsyncMock(return_value=overlay)),
patch.object(core, "_load_persist_bind_script", return_value='#!/bin/sh\nexec "$@"'),
patch.object(core, "_load_docker_vehicle_script", return_value="#!/bin/sh\n"),
patch.object(core, "_load_docker_cli_copy_script", return_value="#!/bin/sh\n"),
patch.object(core, "_load_docker_cli_stage_script", return_value='#!/bin/sh\nexec "$@"\n'),
patch.object(core, "_get_image_entrypoint_cmd", new=AsyncMock(return_value=([], ["/bin/sh"]))),
patch.object(core, "_apply_sandbox_network_policies", new=AsyncMock(return_value=_ok())),
):
cfg = SandboxConfig(
name="dock-kata",
image="ubuntu:24.04",
backend=backend,
docker=True,
docker_disk="20Gi",
)
result = await core.create_sandbox(cfg)
return result, mock_apps
class TestKataDockerPod:
async def test_kql_vehicle_block_socket_cli_no_sidecar_ann(self, core: K7Core):
result, mock_apps = await _create_kata_docker(core, "kata-qemu-longhorn")
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
pod = deployment.spec.template.spec
names = [c.name for c in pod.containers]
assert names == ["sandbox", VEHICLE_CONTAINER_NAME]
vehicle = next(c for c in pod.containers if c.name == VEHICLE_CONTAINER_NAME)
sandbox = next(c for c in pod.containers if c.name == "sandbox")
assert vehicle.security_context.privileged is True
assert sandbox.security_context.privileged is True # persist-bind, not docker
devices = vehicle.volume_devices or []
assert [d.device_path for d in devices] == [GRAPH_DEVICE_PATH]
graph_mounts = [m for m in (vehicle.volume_mounts or []) if m.mount_path == "/var/lib/docker"]
assert graph_mounts == []
sock_mounts = [m.mount_path for m in (sandbox.volume_mounts or [])]
assert KATA_SOCKET_DIR in sock_mounts
assert CLI_STAGING_DIR in sock_mounts
assert "/tmp" in sock_mounts
assert "/var/run" not in sock_mounts
tmp_vol = {m.name for m in (sandbox.volume_mounts or []) if m.mount_path == "/tmp"}
vehicle_tmp = {m.name for m in (vehicle.volume_mounts or []) if m.mount_path == "/tmp"}
assert tmp_vol == {"path-share-tmp"}
assert vehicle_tmp == {"path-share-tmp"}
env = {e.name: e.value for e in (sandbox.env or [])}
assert env.get("DOCKER_HOST") == DOCKER_HOST_URL
cli = [m for m in sandbox.volume_mounts if m.mount_path == CLI_PATH]
assert cli and cli[0].read_only is True
assert pod.share_process_namespace is True
assert not pod.host_pid
assert all(v.host_path is None for v in (pod.volumes or []))
assert deployment.metadata.annotations.get(ANN_K7_DOCKER) == "true"
assert "k7.katakate.org/sidecar" not in (deployment.metadata.annotations or {})
assert deployment.metadata.annotations.get(ANN_DOCKER_PVC) == "dock-kata-docker-lh"
inits = [c.name for c in (pod.init_containers or [])]
assert "docker-cli-copy" in inits
assert DIND_IMAGE in {vehicle.image, *(c.image for c in pod.init_containers or [])}
def _exec_ok():
return ExecResult(exit_code=0, stdout="", stderr="", duration_ms=0)
class TestKataDockerSnapshot:
async def test_create_snapshot_waits_for_root_and_docker(self, core: K7Core):
dep = MagicMock()
dep.metadata.annotations = {
ANN_K7_DOCKER: "true",
ANN_DOCKER_PVC: "sb-docker-lh",
}
dep.status.ready_replicas = 1
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup(core, apps=mock_apps, v1=AsyncMock())
wait = AsyncMock(return_value=_ok())
exec_cmd = AsyncMock(return_value=_exec_ok())
with (
patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")),
patch.object(core, "_create_volume_snapshot", new=AsyncMock(return_value=_ok())),
patch.object(core, "_wait_for_snapshot_ready", new=wait),
patch.object(core, "exec_command", new=exec_cmd),
):
result = await core.create_snapshot("sb", "snap1")
assert result.success, result.error
assert wait.await_count == 2
names = [c.args[0] for c in wait.await_args_list]
assert names == ["snap1", "snap1-docker"]
assert exec_cmd.await_count == 2
assert exec_cmd.await_args_list[0].kwargs.get("container") in (None, "sandbox")
assert exec_cmd.await_args_list[1].kwargs.get("container") == VEHICLE_CONTAINER_NAME
async def test_create_snapshot_skips_sync_when_paused(self, core: K7Core):
dep = MagicMock()
dep.metadata.annotations = {ANN_K7_DOCKER: "true", ANN_DOCKER_PVC: "sb-docker-lh"}
dep.status.ready_replicas = 0
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup(core, apps=mock_apps, v1=AsyncMock())
exec_cmd = AsyncMock(return_value=_exec_ok())
with (
patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-qemu-longhorn")),
patch.object(core, "_create_volume_snapshot", new=AsyncMock(return_value=_ok())),
patch.object(core, "_wait_for_snapshot_ready", new=AsyncMock(return_value=_ok())),
patch.object(core, "exec_command", new=exec_cmd),
):
result = await core.create_snapshot("sb", "snap1")
assert result.success, result.error
exec_cmd.assert_not_awaited()
class TestKataDockerRestore:
async def test_restore_clones_docker_block_pvc(self, core: K7Core):
snap_info = MagicMock()
snap_info.ready_to_use = True
custom = AsyncMock()
custom.get_namespaced_custom_object.return_value = {
"metadata": {
"annotations": {
"k7.io/source-image": "ubuntu:24.04",
"k7.io/source-backend": "kata-qemu-longhorn",
"k7.io/source-docker": "true",
"k7.io/source-docker-disk": "20Gi",
}
}
}
apps = AsyncMock()
apps.read_namespaced_deployment.side_effect = ApiException(status=404)
_setup(core, apps=apps, v1=AsyncMock())
core._custom_objects_client = custom
clones: list[dict] = []
async def fake_clone(**kwargs):
clones.append(kwargs)
return _ok()
with (
patch.object(core, "get_snapshot", new=AsyncMock(return_value=snap_info)),
patch.object(core, "_create_pvc_from_snapshot", new=fake_clone),
patch.object(core, "create_sandbox", new=AsyncMock(return_value=_ok())),
):
result = await core.restore_sandbox("snap1", "dst")
assert result.success, result.error
assert [c["snapshot_name"] for c in clones] == ["snap1", "snap1-docker"]
assert clones[1]["volume_mode"] == "Block"
assert clones[1]["target_pvc_name"] == "dst-docker-lh"
async def test_restore_without_docker_skips_docker_pvc(self, core: K7Core):
snap_info = MagicMock()
snap_info.ready_to_use = True
custom = AsyncMock()
custom.get_namespaced_custom_object.return_value = {
"metadata": {
"annotations": {
"k7.io/source-image": "ubuntu:24.04",
"k7.io/source-backend": "kata-qemu-longhorn",
}
}
}
apps = AsyncMock()
apps.read_namespaced_deployment.side_effect = ApiException(status=404)
_setup(core, apps=apps, v1=AsyncMock())
core._custom_objects_client = custom
clones: list[dict] = []
async def fake_clone(**kwargs):
clones.append(kwargs)
return _ok()
with (
patch.object(core, "get_snapshot", new=AsyncMock(return_value=snap_info)),
patch.object(core, "_create_pvc_from_snapshot", new=fake_clone),
patch.object(core, "create_sandbox", new=AsyncMock(return_value=_ok())),
):
result = await core.restore_sandbox("snap1", "dst")
assert result.success, result.error
assert [c["snapshot_name"] for c in clones] == ["snap1"]
async def test_kfd_ephemeral_block_not_privileged_sandbox(self, core: K7Core):
result, mock_apps = await _create_kata_docker(core, "kata-firecracker-devmapper")
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
pod = deployment.spec.template.spec
sandbox = next(c for c in pod.containers if c.name == "sandbox")
vehicle = next(c for c in pod.containers if c.name == VEHICLE_CONTAINER_NAME)
assert sandbox.security_context.privileged is not True
assert sandbox.security_context.allow_privilege_escalation is False
assert sandbox.security_context.capabilities.drop == ["ALL"]
assert sandbox.security_context.seccomp_profile.type == "RuntimeDefault"
assert vehicle.security_context.privileged is True
graph = next(v for v in pod.volumes if v.name == "docker-graph")
assert graph.ephemeral is not None
claim = graph.ephemeral.volume_claim_template.spec
assert claim.volume_mode == "Block"
assert claim.storage_class_name == KFD_DOCKER_STORAGE_CLASS
assert ANN_DOCKER_PVC not in (deployment.metadata.annotations or {})
assert sandbox.command == ["/bin/sh", "/opt/k7/docker/k7-docker-cli-stage.sh"]
assert sandbox.args == ["/bin/sh", "-c", "sleep 365d"]
async def test_sidecar_docker_alias_is_vehicle_not_dind(self, core: K7Core):
"""API/core path with docker=True (what the CLI alias produces)."""
result, mock_apps = await _create_kata_docker(core, "kata-qemu-longhorn")
assert result.success, result.error
deployment = mock_apps.create_namespaced_deployment.call_args.kwargs["body"]
names = [c.name for c in deployment.spec.template.spec.containers]
assert "sidecar" not in names
assert VEHICLE_CONTAINER_NAME in names
images = [c.image for c in deployment.spec.template.spec.containers]
assert "docker:27.5-dind" not in images
class TestKataDockerForkReject:
async def test_kfd_docker_fork_rejected(self, core: K7Core):
dep = MagicMock()
dep.metadata.annotations = {
"k7.katakate.org/backend": "kata-firecracker-devmapper",
ANN_K7_DOCKER: "true",
}
mock_apps = AsyncMock()
mock_apps.read_namespaced_deployment.return_value = dep
_setup(core, apps=mock_apps, v1=AsyncMock())
with patch.object(core, "_detect_backend", new=AsyncMock(return_value="kata-firecracker-devmapper")):
result = await core.fork_sandbox("src", "dst")
assert not result.success
assert "cannot be cloned" in result.error
assert result.error == KATA_FORK_GRAPH_REJECT
class TestDindImagePin:
def test_dind_image_is_digest_pinned(self):
assert DIND_IMAGE.startswith("docker:27.5.1-dind@sha256:")
digest = DIND_IMAGE.split("sha256:", 1)[1]
assert len(digest) == 64
+1 -1
View File
@@ -1,4 +1,4 @@
"""Spec 11a: ``katakate`` PyPI shim re-exports ``k7_sdk`` with a deprecation warning."""
"""``katakate`` PyPI shim re-exports ``k7_sdk`` with a deprecation warning."""
import warnings
+30
View File
@@ -0,0 +1,30 @@
"""Playbook turns off Kata's guest seccomp bypass."""
from pathlib import Path
PLAYBOOK = Path(__file__).resolve().parents[2] / "src/k7/deploy/k7-install-node.yaml"
def test_playbook_disables_guest_seccomp_bypass_on_both_kata_configs():
text = PLAYBOOK.read_text()
assert "path: /opt/kata/share/defaults/kata-containers/configuration-qemu.toml" in text
assert "path: /opt/kata/share/defaults/kata-containers/configuration-fc.toml" in text
assert "regexp: '^#?\\s*disable_guest_seccomp\\s*='" in text
assert text.count("line: 'disable_guest_seccomp = false'") == 2
assert "line: 'disable_guest_seccomp = true'" not in text
assert "qemu_guest_seccomp.rc != 0" in text
assert "fc_guest_seccomp.rc != 0" in text
assert "Refusing to leave Kata guest seccomp bypassed" in text
qemu = text.split("Assert disable_guest_seccomp exists in kata-qemu config", 1)[1]
qemu = qemu.split("Assert disable_guest_seccomp exists in kata-fc config", 1)[0]
fc = text.split("Assert disable_guest_seccomp exists in kata-fc config", 1)[1]
fc = fc.split("Create containerd shim symlink", 1)[0]
assert "when: k7_has_longhorn" in qemu
assert "when: k7_has_devmapper" in fc
assert "k7_has_longhorn" not in fc.split("when:", 1)[1]
def test_playbook_does_not_regress_virtiofsd_thread_pool():
"""CHALLENGES.md #10: single-threaded virtiofsd wedges the kata agent."""
text = PLAYBOOK.read_text()
assert 'virtio_fs_extra_args = ["--thread-pool-size=16", "--announce-submounts"]' in text
+1 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for the control-plane registry SSRF guard (spec 10h)."""
"""Unit tests for the control-plane registry SSRF guard."""
from unittest.mock import AsyncMock, MagicMock, patch
+21
View File
@@ -206,3 +206,24 @@ def test_render_known_byte_stable_snapshot(tmp_path: Path, render):
"""
)
assert out == expected, f"--- expected ---\n{expected}\n--- got ---\n{out}"
def test_render_k7d_fc_fork_rows(tmp_path: Path, render):
"""k7d-fc column and fork ops appear only when the CSV has them."""
rows = [
("k7d", "run_io", 1, 1.2),
("k7d", "run_io", 2, 1.3),
("k7d", "fork_to_ready", 1, 2.4),
("k7d-fc", "run_io", 1, 1.4),
("k7d-fc", "run_io", 2, 1.5),
("k7d-fc", "fork_to_ready", 1, 2.6),
]
csv = _write_csv(tmp_path, rows)
out = render(csv, title="k7d-fc", hardware_note="hw")
assert "| k7d |" in out or "| k7d | k7d-fc |" in out
assert "k7d-fc" in out
assert "fork → Ready + overlay2" in out
assert "fork warm engine" not in out # not in this CSV
io_row = next(line for line in out.splitlines() if line.startswith("| run io"))
assert "1.25 s" in io_row
assert "1.45 s" in io_row
+15 -1
View File
@@ -1,4 +1,4 @@
"""Unit tests for the Spec 10f restore-from-snapshot surface.
"""Unit tests for the restore-from-snapshot surface.
Coverage:
@@ -106,6 +106,20 @@ class TestRehydrateConfigFromSnapshot:
# Falls back to default backend when neither annotation nor override set.
assert result.data.backend == "kata-qemu-longhorn"
def test_docker_annotation_sets_docker_true(self, core: K7Core):
ann = {
"k7.io/source-image": "ubuntu:24.04",
"k7.io/source-backend": "kata-qemu-longhorn",
"k7.io/source-docker": "true",
"k7.io/source-docker-disk": "20Gi",
}
result = core._rehydrate_config_from_snapshot(
annotations=ann, new_name="r", namespace="default", overrides=None
)
assert result.success
assert result.data.docker is True
assert result.data.docker_disk == "20Gi"
def test_malformed_limits_annotation_falls_back_to_none(self, core: K7Core):
ann = {"k7.io/source-image": "alpine:3.20", "k7.io/source-limits": "not json"}
result = core._rehydrate_config_from_snapshot(
+84 -3
View File
@@ -1,4 +1,4 @@
"""Unit tests for the Spec 10e snapshot lifecycle (kind inference + gc filters).
"""Unit tests for the snapshot lifecycle (kind inference + gc filters).
These exercise ``K7Core`` directly with mocked Kubernetes custom-objects API
calls, so no cluster is required. They cover:
@@ -12,7 +12,9 @@ calls, so no cluster is required. They cover:
"""
from datetime import datetime, timedelta, timezone
from unittest.mock import AsyncMock, patch
from unittest.mock import AsyncMock, MagicMock, patch
from kubernetes_asyncio.client.exceptions import ApiException
from k7.core.core import K7Core
from k7.core.models import (
@@ -107,7 +109,13 @@ def _patch_list(core: K7Core, items: list[dict]) -> None:
"""Patch the dynamic K8s client so ``list_snapshots`` returns ``items``."""
mock_custom = AsyncMock()
mock_custom.list_namespaced_custom_object.return_value = {"items": items}
mock_custom.list_cluster_custom_object.return_value = {"items": items}
async def _list_cluster(*, plural: str = "", **_kwargs):
if plural == "volumesnapshotcontents":
return {"items": []}
return {"items": items}
mock_custom.list_cluster_custom_object.side_effect = _list_cluster
core._custom_objects_client = mock_custom
core._config_loaded = True
@@ -211,3 +219,76 @@ class TestGcSnapshots:
assert len(result.data) == 1
assert result.data[0]["deleted"] is False
assert "transient" in result.data[0]["error"]
async def test_reaps_orphan_volumesnapshotcontent(self, core: K7Core):
items = [
_make_snap(
"demo-fork-1700000000",
annotations={"k7.io/kind": SNAPSHOT_KIND_FORK, "k7.io/source-sandbox": "demo"},
age_minutes=5,
),
]
vsc = {
"kind": "VolumeSnapshotContent",
"metadata": {"name": "snapcontent-orphan", "deletionTimestamp": "2026-09-07T22:00:00Z"},
"spec": {"volumeSnapshotRef": {"name": "gone-snap", "namespace": "k7-test-abcd"}},
}
mock_custom = AsyncMock()
async def _list_cluster(*, plural: str, **_kwargs):
if plural == "volumesnapshotcontents":
return {"items": [vsc]}
return {"items": items}
mock_custom.list_cluster_custom_object.side_effect = _list_cluster
mock_custom.list_namespaced_custom_object.return_value = {"items": items}
mock_custom.get_namespaced_custom_object.side_effect = ApiException(status=404)
mock_custom.patch_cluster_custom_object.return_value = {}
mock_custom.delete_cluster_custom_object.return_value = {}
core._custom_objects_client = mock_custom
core._config_loaded = True
v1 = MagicMock()
v1.list_namespaced_persistent_volume_claim = AsyncMock(return_value=MagicMock(items=[]))
core._core_v1_client = v1
with patch.object(core, "delete_snapshot", new_callable=AsyncMock) as del_snap:
result = await core.gc_snapshots(namespace="k7-test-abcd", keep_fork_for=timedelta(minutes=10))
del_snap.assert_not_called()
assert result.success is True
assert [r["name"] for r in result.data] == ["snapcontent-orphan"]
assert result.data[0]["kind"] == "volumesnapshotcontent"
assert result.data[0]["deleted"] is True
mock_custom.delete_cluster_custom_object.assert_awaited()
patch_kwargs = mock_custom.patch_cluster_custom_object.await_args.kwargs
assert patch_kwargs.get("_content_type") == "application/merge-patch+json"
async def test_leaves_volumesnapshotcontent_for_live_snapshot(self, core: K7Core):
live = _make_snap(
"exp-baseline",
annotations={"k7.io/kind": SNAPSHOT_KIND_NAMED, "k7.io/source-sandbox": "demo"},
source_pvc="demo-root-lh",
)
vsc = {
"kind": "VolumeSnapshotContent",
"metadata": {"name": "snapcontent-live"},
"spec": {"volumeSnapshotRef": {"name": "exp-baseline", "namespace": "default"}},
}
mock_custom = AsyncMock()
async def _list_cluster(*, plural: str, **_kwargs):
if plural == "volumesnapshotcontents":
return {"items": [vsc]}
return {"items": [live]}
mock_custom.list_cluster_custom_object.side_effect = _list_cluster
mock_custom.list_namespaced_custom_object.return_value = {"items": [live]}
mock_custom.get_namespaced_custom_object.return_value = live
core._custom_objects_client = mock_custom
core._config_loaded = True
result = await core.gc_snapshots()
assert result.data == []
mock_custom.delete_cluster_custom_object.assert_not_called()
+2 -2
View File
@@ -80,9 +80,9 @@ sudo apt update
sudo apt install k7
# dual-NVMe box: let the playbook auto-detect the raw spare disk.
# PPA k7 0.2.1 still defaults k7d to 0.1.0 — pin the public v0.2.1 release.
# PPA k7 0.2.1 still defaults k7d to 0.1.0 — this tree pins k7d 0.6.0.
# Run from a checkout of this repo (or Katakate/k7) so k7-api:local can build.
sudo k7 install --backend kfd,kql,k7d --k7d-version 0.2.1
sudo k7 install --backend kfd,kql,k7d
```
`k7 install` provisions the LVM thin-pool on that disk for the `kfd` backend.
Generated
+1 -1
View File
@@ -1069,7 +1069,7 @@ wheels = [
[[package]]
name = "k7"
version = "0.2.2"
version = "0.3.0"
source = { editable = "." }
dependencies = [
{ name = "fastapi" },