Observability, usage metering, tenants and quotas, and roles with OIDC login. A Helm chart installs K7 on a cluster you already run, API keys carry a role, and k7-agent no longer needs host privileges.
26 KiB
Security Policy
Supported Versions
This project is pre-1.0 and under active development; breaking changes may occur until 1.0.0. Security fixes land on the latest release line only.
| Version | Supported |
|---|---|
| 0.2.1 and later | Yes |
| 0.2.0 and earlier | No — upgrade to 0.2.1 |
Reporting a Vulnerability
If you believe you have found a security vulnerability, please email:
- security@katakate.org (preferred)
- Or open a private security advisory via GitHub (Security → Advisories → Report a vulnerability)
Please include:
- A detailed description of the issue and potential impact
- Steps to reproduce or proof-of-concept
- Affected versions / commit SHAs and environment details
We aim to acknowledge reports within 72 hours and provide a remediation plan or mitigation timeline when applicable.
Do not open a public issue for security-sensitive reports.
Scope and current model
-
Nodes run K3s. A cluster (or a single node) can install multiple sandbox backends; each sandbox picks one:
kfd(kata-firecracker-devmapper) — Firecracker via Kata. The Firecracker process runs inside the jailer (chroot + dropped capabilities + seccomp). An integration test asserts the jailer is active after install.kql(kata-qemu-longhorn) — QEMU via Kata with a Longhorn PVC root (durable disk, named snapshots / restore / disk-only fork).k7d— Katakate/k7d (runtimeClassName: k7); CoW sibling-fork isolation differs — see k7d'sSECURITY.md.k7d-fc(runtimeClassName: k7-fc) — same daemon and shim; stock Firecracker under the stock jailer. What that buys (and does not) is k7d'sSECURITY.mdFirecracker profile table. hostPath volumes are refused before the pod is created.
-
Sandbox containers run as non-root with restricted capabilities on top of the VM boundary.
-
The control plane API uses API keys with hashed storage and expiry, stored in the Kubernetes Secret
kube-system/k7-api-keys(protect etcd: enable K3s--secrets-encryptionor KMS envelope encryption on managed clusters). Tokens arek7_<id>_<secret>; the store holdssha256(secret). A plain SHA-256 is deliberate: the secret is 256 random bits, so the threat is store disclosure, not guessing, and a slow KDF adds nothing. The<id>is safe to log and makes leaked keys greppable. The API never writes the key store on authenticate; it serves a snapshot refreshed every 5 s and returns 503 rather than trusting a snapshot older than 60 s. Revocation reaches every replica within one refresh. Thek7-apiServiceAccount can onlygetandpatchthe key Secret (namespaced Role;patchis for the admin key routes); it cannot read Secrets elsewhere. Every key carries a role (see Roles and OIDC below). Keys may additionally be narrowed to one or more namespaces (k7 generate-api-key -n <ns>); they may also be scoped to one or more Kubernetes nodes (k7 generate-api-key --node <node>). Absent/empty narrowing leaves the role's own scope (its tenant, or the cluster). Narrowed keys are enforced on every namespace-bearing and cluster-scoped control-plane route — they cannot list across all namespaces, touch namespaces outside their list, or read cluster-wide topology (e.g.GET /api/v1/nodes/storage). A node-scoped key may only create/restore/fork sandboxes onto its listed nodes (a single-node scope auto-pins; several nodes require an explicitnode_name). Namespace scope is the control-plane tenancy boundary; node scope is the placement boundary (API-key--nodeis a Kubernetes node name fromk7 nodes list/kubectl get nodes— the Linux hostname K3s registered — joined via the kubelet-stampedkubernetes.io/hostnamelabel; inventory.ini does not write that label). Together they keep a tenant off another tenant's k7d daemon only if that node is also dedicated (k7 nodes dedicate NODE --tenant IDwritesk7.katakate.org/tenantas a label and a NoSchedule taint). Without the taint, an unscoped key can still land on the same node. Anoperatororadminkey without narrowing has cross-namespace control-plane access and unrestricted placement. -
Roles and OIDC. K7 roles are not Kubernetes RBAC. They are application roles checked inside k7-api; the k7-api ServiceAccount stays the only Kubernetes actor, and nobody gets a Kubernetes identity through K7. Four fixed roles:
read-only(list/get sandboxes and snapshots) anduser(plus create, exec, logs, pause/resume, fork, snapshots) are granted per tenant (theTenantCR'sspec.namespaces);operator(cluster-wide read, snapshot GC, node storage, node dedication; no exec, no keys) andadmin(everything, including keys and role bindings) are cluster-wide. One table,PERMISSIONSinsrc/k7/api/authz.py, maps every route to its roles; a route missing from it answers 500 and fails CI. Checks compose: route role, then tenant, then the key's namespace/node narrowing; the first failure is a 403 that names the missing role or scope.k7 api keys migrategives keys from before roles a role:admin: true→admin, namespace-scoped →user(still narrowed to those namespaces), unscoped →adminflaggedunscoped_legacy(list-api-keyswarns; re-issue them with a real role). A key without a role is refused (403) until migrated. OIDC (k7 install --oidc-issuer/--oidc-audience/--oidc-client-id/ --oidc-groups-claim/--oidc-username-claim, optional--oidc-jwks-ttl, or chartauth.oidc; all five or none;https://issuers only) lets humans and workloads sendAuthorization: Bearer <JWT>. k7-api validates statelessly: signature against the issuer's JWKS (cached forK7_OIDC_JWKS_TTL_SECONDS, default 3600, minimum 10; also refetched on an unknownkidat most every 10 s). A failed refresh is 503, including for the next 10 s, and an expired cache is not used.algis RS256 or ES256 only and must match the key type (none, HS256 and key confusion are 401).iss,aud,exp(60 s leeway) andsubare required. When the username claim isemail,email_verifiedmust be boolean true. Tokens startingk7_and theX-API-Keyheader always take the key path; a bad key is never retried as OIDC. Groups, usernames or subjects map to roles through ConfigMapk7-role-bindings(k7 role-bindings get|apply,PUT /api/v1/role-bindings, admin only), read on every request, so an edit or adeny-subjectsentry applies on the next request on every replica. Several matching bindings combine as the highest role per tenant plus the highest cluster role. An unbound subject gets a 403 that echoes its groups (to debug claim shapes); bindings validate tenants, role/tenant pairs and duplicates, and a write that leaves noadminbinding while no unexpiredadminkey exists is refused. k7-api stores no sessions and never sees refresh tokens:k7 login(device flow, or--browserfor Authorization Code + PKCE) caches them in~/.config/k7/credentials.json(0600) and refreshes them itself;k7 logoutdeletes the cache, then revokes at the IdP. Configure the IdP for short access tokens (1 h or less): a JWT stays valid until it expires, anddeny-subjectsis the immediate stop. Keep anadminbreak-glass key offline: keys keep working during an IdP outage, and the server-node bootstrap path (k7 generate-api-keywith the node's kubeconfig) always works. Reinstalling without the--oidc-*flags turns OIDC off (like--api-allow-cidr); so does the deprecated remotePOST /api/v1/install. -
The control-plane API is on a public NodePort (
31007) on a public node, over HTTPS by default. A Caddy sidecar terminates TLS; the API container and its probes stay HTTP on:8000. The default cert is a playbook-minted cluster CA at/etc/k7/tls/ca.crt(Let's Encrypt cannot issue for a bare IP). Clients trust it withk7 config set api.ca,--api-ca, orK7_API_CA.--api-hostname(Let's Encrypt via Caddy) and--api-tls-cert/--api-tls-keyare the other tracks;--api-insecure-httprestores a plain HTTP NodePort and must be treated as such (verify=Falseon anhttps://URL is an error).k7 install --api-allow-cidr <cidr>(repeatable, Cilium only) restricts who can reach it: aCiliumNetworkPolicyon thek7-apipod allows the operator's CIDRs plus the reservedhost/remote-node/health/kube-apiserverpeers (the kubelet probe path), and the Service switches toexternalTrafficPolicy: Localso the pod sees the real client IP. This is defence in depth for operators who know their client CIDRs, not a secure API: there is still no rate limiting. The flag is off by default and affects nothing when unset. It is a pod-level policy — SSH and the k3s API port are never touched, so a wrong CIDR costs API access from your workstation and nothing else (kubectl -n kube-system delete ciliumnetworkpolicy k7-api-ingressundoes it). AfromCIDRpeer is not evaluated for Cilium-managed sources, so the allowlist constrains external clients only; in-cluster access is governed by the pod policies. -
Audit log. Every mutating API request and every 401/403 produces one JSON line on the API pod's stdout (
k7.auditlogger) with the principal type (key/oidc), subject (key id or OIDCsub), role, tenant, IdP issuer, key id and name, route template, namespace, sandbox, status, latency and client IP.X-Forwarded-Foris trusted only from the localhost Caddy sidecar. Tokens, emails, groups, bodies and exec command text are never logged (exec: sha256 + length). Emission failure fails the request closed (500). Ship stdout with your collector (Fluent Bit, CloudWatch, Loki); on exec-heavy workloads expect one line per exec and size log rotation accordingly. -
Least-privilege control-plane pods.
k7-agentis not privileged: uid 0 plus thek7dgroup, pinned to gid 770 on every node (for the root-owned containerd socket and thek7d:k7dk7d socket), all capabilities dropped, no privilege escalation, read-only root filesystem,RuntimeDefaultseccomp. Its only host mounts are the containerd socket directory,/run/k7d,/var/lib/k7d(ro) and/run/k7(ro);/devis not mounted. kfd thin-pool stats come from the host timerk7-storage-report.timer(/run/k7/storage.json, 500 if older than 90 s).k7-apiruns as uid 1000 with no hostPath and forwards every k7d VM op to the node'sk7-agent. No K7 pod opens/dev/kvm(the shims and VMMs run on the host), so there is no KVM device plugin. -
Control-plane OCI registry inspection (used to resolve image entrypoint/cmd) rejects registry hosts that are not on an allowlist (default:
registry-1.docker.io,ghcr.io,quay.io,public.ecr.aws; extend viaK7_REGISTRY_ALLOWLIST) and rejects any host that resolves to loopback/private/link-local/metadata/reserved addresses. Redirect following is disabled. The previouslocalhost→httpdowngrade path has been removed. -
Ingress to sandboxes is denied by default (NetworkPolicy) and opt-in per sandbox:
--ingress-portopens TCP ports,--ingress-from(sandbox:/namespace:/cidr:) scopes who may connect. With no source given, an opened port is reachable only from sandboxes in the same namespace — never from the internet. Acidr:source does not scope in-cluster peers on Cilium: anipBlockpeer is not evaluated for pod-to-pod traffic, so such a rule leaves the opened port reachable from any sandbox in the cluster.k7 createwarns when one is used; scope in-cluster access withsandbox:/namespace:instead. Seedocs/BACKENDS.mdfor the measured behaviour.--expose-portadditionally publishes a port outside the cluster (NodePort withexternalTrafficPolicy: Local, so the pod sees the real client IP); it requires a matching--ingress-port, and on a public node it puts the sandbox on a public IP. Acidr:allowlist there does not constrain clients on the cluster's own nodes. Egress is per-sandbox: open, blocked, CIDR allowlist, or FQDN allowlist when Cilium is the CNI (default). DNS follows the egress mode:- Blocked and CIDR-only sandboxes have no DNS at all.
- FQDN sandboxes resolve only their allowlisted names. Cilium's
DNS proxy forwards those to CoreDNS and answers every other query
itself with NXDOMAIN (
dnsProxy.dnsRejectResponseCode=nameError), so a sandbox cannot exfiltrate data by encoding it into queries for<data>.attacker.tld. k7-api rewrites policies created by older versions (which allowed every name) when it starts. - Open sandboxes resolve anything, as they reach anything.
Residual channels inside an allowlist are the allowlist's own:
- A broad wildcard over a zone someone else controls (
*.nip.io, a delegated subdomain,*.amazonaws.com) admits names that party chooses, and HTTPS to them. - Allowlisting a public resolver (
dns.google,1.1.1.1/32) gives that resolver directly, DoH included. - An allowlisted name on a shared CDN IP can front for other tenants of that CDN.
Without Cilium (
--cni flannel, chartnetworkPolicy.provider=kubernetes), domain entries are refused. A CIDR entry containing the cluster DNS server (thekube-dnsService or CoreDNS pod IPs, checked at create and fork) is refused too: those engines match pod IPs, so it would give a recursive resolver. The check is against current IPs only. A CoreDNS pod rescheduled into an allowlisted pod-network range later re-opens the path, so do not allowlist pod or service ranges there. -
Sandboxes never reach the platform. With Cilium (default) a cluster-wide deny policy (
k7-sandbox-platform-deny) stops every sandbox — in every egress mode, including--egress-open— from dialing the node it runs on, the other nodes, the Kubernetes API (node IP or10.43.0.1), cloud metadata (169.254.0.0/16), and the pods ofkube-system/longhorn-system(k7-api, Longhorn, ...), CoreDNS excepted. "Open" egress means the internet, not the cluster. This is deny-only and composes with the per-sandbox egress policies; it does not defend against a VM escape. Caveat:k7 install --cni flannelclusters do not get this isolation (no Cilium policy engine) — an--egress-opensandbox there can reach the node and the API server. -
Multi-node clusters are supported (Ansible inventory; Longhorn for the QEMU/
kqlpath). Cilium FQDN egress applies cluster-wide. -
Root on a K3s server is full cluster control. Nodes in
[k7_servers](including the 3-node--haexample, where every host is a server) have/etc/rancher/k3s/k3s.yamlas cluster-admin (playbook--write-kubeconfig-mode 644), the join token, and etcd. The first master also has/etc/k7/agent_token. Node pins andk7 nodes dedicatedo not contain that: they stop two tenants sharing a k7d daemon, they do not survive a VM escape onto a master. Put tenant sandboxes on[k7_agents]; do not dedicate a tenant onto a server runningk7-api. Root on an agent is not the admin kubeconfig.k7-agenthas no ServiceAccount token and ingress fromremote-nodeis denied, so that breakout owns that node's k7d (that tenant, if dedicated) — not other agents and not the Kubernetes API. Per-node isolation is the blast radius for a worker compromise; it is not useless. The shared agent token remains on every node as defence-in-depth after the CNP. -
k7-node-installeris root on every opted-in node, by design (kata-deploy pattern;src/k7/deploy/manifests/k7-node-installer/). It is privileged,hostPID, mounts/at/hostand runs Ansible in PID 1's namespaces. It runs only on nodes labelledk7.katakate.org/node-pool=trueand is separate fromk7-agent. Its image carries every release artifact, checksummed at build time; it downloads nothing at runtime (OS packages still come from the node's own package repos). Node labelling: RBAC cannot scopepatch nodesto one node, so the ServiceAccount getsget/patchon all nodes and a ValidatingAdmissionPolicy (k7-node-installer-own-node) denies any patch that is not to the node in the caller's bound token (authentication.kubernetes.io/node-name, Kubernetes >= 1.32), touches the node spec, annotations, finalizers or ownerReferences, or changes a label outsidek7.katakate.org/. That is defence in depth, not a boundary: root on the node can read that node's kubelet credentials. It does stop a leaked installer token from relabelling other nodes, for example into a tenant's pool. The cleanup ServiceAccount addslist/deleteon pods (to drain sandboxes with--force) and is bound only by the cleanup DaemonSet; a second policy (k7-node-cleanup-own-node-pods) lets it delete only K7 sandbox pods (RuntimeClassk7,k7-fc,kata,kata-qemu) on its own node. Itslist podsstays cluster-wide (admission does not see reads), so a leaked cleanup token can read every pod spec in the cluster. The release publishes the installer image signed, scanned and pinned by digest ink7-node-installer.yaml; the git-tree manifests carry a placeholder. -
Runtime annotations are k7's alone. containerd hands pod annotations to the k7d shim and to Kata, which act on them:
k7d.katakate.org/cluster-idskips the pod netns (egress leaves as the node, outside Cilium),fork-source-*/tree-*clone any VM on the node by id,io.katacontainers.config_pathandio.katacontainers.config.runtime.*override the hardened Kata config. The ValidatingAdmissionPolicyk7-runtime-annotationsdenies adding, changing or removing anyk7d.katakate.org/orio.katacontainers.annotation on Pods and built-in pod templates (Deployment, ReplicaSet, StatefulSet, DaemonSet, Job, CronJob, ReplicationController) unless the caller is the k7-api ServiceAccount, insystem:masters, or kube-controller-manager. Grant tenants namespace RBAC freely; this is what keeps them on the pod network. Behind it, containerd forwards only the keys k7 sets (docker,docker-disk,fork-source-cluster,fork-source-vm, Katadefault_memoryand the tenant rate limiters), and Kata enables only those three hypervisor annotations. Workloads created through other controllers (CRDs) are denied if their pods carry these keys.
See also the docs: security model, networking, and backends comparison.
Tenants and quotas
api.modeis chosen at install, never defaulted (k7 install --api-mode production|development, chartapi.mode).productionrefuses to start while the key store holds an unscoped key and names each one: no role (from before roles), a migratedunscoped_legacyadmin, or a namespace-only key with no tenant. It refuses--api-insecure-http(K7_API_TRANSPORT) andK7_API_ENABLE_REMOTE_INSTALL=1, and returns 403 to those keys on every route at request time (including key minting, role bindings and/install), with the commands that re-scope them. Every response carriesX-K7-Mode;/healthreports it.developmentkeeps them working (unscoped_legacyas admin).- A
Tenant(tenants.k7.katakate.org/v1alpha1, cluster-scoped) claims namespaces (labelk7.katakate.org/tenant), dedicated nodes, adefaultSize, quotas and per-VM limits. A--tenantkey reaches the tenant's namespaces intersected with its own-nlist, read from the Tenant on every request, so adding or removing a namespace applies at once and a key whose tenant is deleted reaches nothing. Cross-tenant requests are 403 before any cluster call, on every namespaced route. Fork and restore never leave their namespace. A namespace claimed by two Tenants (only possible by writing CRs by hand) fails every request there with 500TenantConflictnaming both. Creates carry the tenant andk7.katakate.org/api-key-idlabels on every object; tenants withspec.nodesland only there, through a selector and toleration. Tenant routes (/api/v1/tenants):operatorandadminread, onlyadmincreates, updates and deletes (PERMISSIONS). - Quotas (
sandboxes,vcpu,memory,disk,snapshotStorage) are checked at K7's two choke points, the sandbox Deployment create and the VolumeSnapshot create, so create, fork, restore, pause-with-snapshot and named snapshots are all covered. The check and the create run under a per-tenantcoordination.k8s.ioLease (k7-quota-<tenant>, TTL 10 s), so concurrent creates on any number of API replicas cannot overshoot. A Lease not acquired within 5 s is 503QuotaLockTimeoutand creates nothing; a Lease lost mid-check is 503QuotaLockLost. Over-quota is 403QuotaExceededwith the dimension, limit, usage and request, and is audit-logged. Usage is reserved, not used: a paused sandbox keeps its vCPU, memory and disk until it is deleted. kfd counts its 10 GB devmapper root; kql counts the PVCs its Deployment mounts. The leader replica writesTenant.status.usageevery 60 s. - Enforcement is at the K7 API.
kubectl applyby a cluster admin bypasses it. For a single-namespace tenant aResourceQuota(k7-tenant-quota:count/deployments.apps,requests.storage) is the Kubernetes-side backstop; multi-namespace tenants have none, since a ResourceQuota cannot span namespaces. - Per-VM limits fail closed.
limits.network(egressBps,ingressBps, bits/s) is set as Kata'stx_/rx_rate_limiter_max_rateannotations (Firecracker token buckets on kfd, tc on the tap for kata-qemu). On kfd, K7 encodes the value for Kata's Firecracker unit conversion, which would otherwise run 4.8% fast. Firecracker's bucket starts full, so the first second can burst to twice the cap; the 30-second average stays within 10%. This replaces the spec's Cilium Bandwidth Manager, which exists only under Cilium (not--cni flannelor a chart install on another CNI) and is aimed at pod egress; the hypervisor limiter caps both directions per VM whatever the CNI. A request may lower a cap, never raise it. A backend that cannot enforce a configured limit refuses the create with 400LimitNotEnforceableinstead of running unlimited:diskIOon every backend (Kata 3.24's runtime has no block rate-limiter annotation, and k7d has no disk limiter yet),networkon k7d and k7d-fc, and adiskquota ormaxSandboxSize.diskon k7d and k7d-fc (K7 cannot read a k7d VM's writable-disk size).maxSandboxSizeis 400; production mode needs a CPU and memory limit on every sandbox, filled fromdefaultSize.
Known limitations (pre-1.0)
- Default
k7 installservesk7-apiover HTTPS with a playbook-minted cluster CA, so API keys no longer travel in cleartext. There is still no rate limiting.--api-insecure-httpis the escape hatch back to a plain HTTP NodePort and must be treated as such.--api-allow-cidris defence in depth on who can connect; it is not a secure-API claim. - No rate limiting or abuse protection at the API layer yet.
- API keys are only as safe as the cluster's Secrets: anyone with
get secretsinkube-system(or root on a server) reads the hashes. Tenant roles are the tenancy boundary on top of that model. Migratedunscoped_legacykeys are clusteradmin(key minting, role bindings, remote install) only indevelopmentmode;productionmode refuses them at startup and on every request until they are re-scoped. Node scoping is the matching opt-in placement boundary: without--nodeon the key, two namespaces can still share a node's k7d daemon. Pair-nand--node, and runk7 nodes dedicateon that node, for tenant isolation on k7d. The dedicate step is a label + taint (k7.katakate.org/tenant); a hostname pin alone does not keep other sandboxes off the node. Neither contains a host compromise: root on[k7_servers]is cluster-admin. - Until the release after 0.4.0,
POST /api/v1/installaccepted any valid key and request-supplied playbook content. It is now 404 unless the cluster was installed with--api-remote-install, cluster-wideadminonly, and takes only node names: it labels them into the k7-node-installer pool, whose image and config are fixed at deploy time. An admin can therefore make the installer root-prepare any existing node it names (servers only with--node-installer-allow-server-restart). Turning it off removes that path: a playbook re-install deletes the installer stack and pool labels; on the chart, whose installer is core, Helm drops the API's Role and the pre-install/pre-upgrade hook, which may patch nodes while it runs, removes the remote-install marker and the records. The nodes the API added stay in the pool and stay prepared. - An OIDC token is valid until it expires; there is no revocation list
beyond
deny-subjects. Entra ID group overage (more than ~200 groups replaced by a link) is not followed: bind such users byuserorsubject, or emit group ids in the token. - Young project; no independent security audit yet.
- The
k7dbackend has a different isolation trade-off for CoW sibling forks — see k7d'sSECURITY.md. - Prefer a dedicated RBAC-restricted kubeconfig for the API rather than cluster-admin credentials in production.
- Host compromise is cluster-wide on servers.
--node/dedicateisolate k7d placement, not a root shell on[k7_servers]. The HA inventory example makes every node a server. Tenant boxes belong in[k7_agents].k7-agenthas no ServiceAccount token and noremote-nodeingress, so root on an agent is that node (that tenant if dedicated), not k7-api RBAC on the rest of the cluster.
Responsible Disclosure
Do not publicly disclose vulnerabilities before we have had a reasonable time to investigate and release fixes. We appreciate coordinated disclosure and will credit reporters unless anonymity is requested.