mirror of
https://github.com/Katakate/k7.git
synced 2026-11-20 20:34:24 +00:00
Observability, usage metering, tenants and quotas, and roles with OIDC login. A Helm chart installs K7 on a cluster you already run, API keys carry a role, and k7-agent no longer needs host privileges.
1.2 KiB
1.2 KiB
K7OperationSlow
The p95 of a successful lifecycle op (k7_operation_duration_seconds,
label op) has been over twice its budget for 15 minutes. The budgets come
from PERFORMANCE.md, and the chart's observability.thresholds.opP95BudgetSeconds
overrides them.
First three commands
kubectl -n kube-system logs -l app=k7-api -c k7-api --since=30m | jq -c 'select(.logger == "k7.op") | {op, backend, duration_ms, sandbox, namespace}' | sort | tail -20
kubectl get pods -A -l katakate.org/sandbox -o wide | grep -v Running
kubectl -n longhorn-system get volumes.longhorn.io -o wide | grep -v healthy
Reading them
- One backend is slow:
kata-qemu-longhornforks and creates wait on Longhorn snapshots and clones. Degraded or rebuilding volumes slow them down.k7dops wait on the k7d daemon (k7d_rpc_duration_secondson the k7-agent of that node). - One node is slow: compare the
nodeof the slow sandboxes. A full thin pool or a busy disk raises every op on that node; see K7StoragePoolFilling. - Every op is slow: the Kubernetes API is slow. Check
k7_kube_api_errors_totaland the apiserver's own latency.