KRNALI LABS
We build tools that make evidence useful. krnali Ops helps teams understand what is happening in their Kubernetes environment. Our EUDI message verifier explores credential checks inside a conversation. Read the project notes, inspect the code we have published, and see what still needs testing.
From the lab Ops in private preview. EUDI verifier in sandbox testing.
Clear project status · Evidence you can inspect · Practical testing
PROJECTS & UPDATES
Two projects, with different purposes and data boundaries. Each update explains what is available, what has been tested and what is still being built.
What has changed in the operator and portal: clearer investigations, honest lifecycle states and separately authorised, bounded fixes.
Current capabilities and work still in integration, explained separately.
A single-use link requests a name presentation from an EUDI test wallet and returns the result to the operator handling the conversation.
Real sandbox integration. Successful phone presentation and iPhone compatibility still need confirmation.
This historical record predates the current forecast bounds described in the
Ops update. It is one object, captured with
kubectl get finding -o json from
the cluster krnali runs against every day, and printed here field for field.
Nothing is rewritten for the page. It is in the HTML, so it is here with
JavaScript off.
FINDING krnali-b21b7eb38360a758 NAMESPACE krnali-portal SUBJECT krnali-portal/portal is trending toward its memory limit SEVERITY warning KIND prediction FAILURE CLASS OOMKill LEAD TIME 485h28m (range 388h22m to 582h34m) PROVENANCE deterministic CONFIDENCE 40 / medium OBSERVED SIGNALS memory.limit 256.0 MiB workload memory.workingSet 12.5 MiB metrics.k8s.io growthRate 513.3 KiB/h baseline sampleCount 256 baseline windowSpan 21m0s baseline DEVIATION 12.5 MiB observed against 11.4 MiB baseline memory.workingSet working set is 1.1 MiB above the learned baseline EVIDENCE CHAIN 1. working-set memory rising on container portal latest working set 12.5 MiB against a 256.0 MiB memory limit 2. baseline EWMA mean 11.4 MiB over 256 samples trend observed over 21m0s 3. deviation from learned-normal 1.1 MiB above baseline (1.1σ); upward slope sustained beyond 15m0s (Mann-Kendall Z=11.9) 4. coincident change: ReplicaSet krnali-portal-794dfcdbd6 (revision 5) rolled out at 2026-08-03T09:10:51Z (outside the correlation window); images: krnali-portal:main-2465dc8 5. projected to reach the memory limit in ~485h28m0s linear extrapolation of the working-set trend (513.3 KiB/h, R²=0.27) crosses 256.0 MiB 6. same symptom already recorded 132 time(s), first seen 2026-07-31T17:05:27Z ADVISORY raise the container memory limit or reduce its footprint yourself; krnali does not edit workloads $ kubectl set resources deployment/krnali-portal -c portal --limits=memory=<above the projected peak> target deployment/krnali-portal reversible yes blast radius workload rollback: restore the previous memory limit with the same command using the prior value fix at source find and fix the memory growth (leak, unbounded cache, or a recent change) rather than only raising the limit if the growth is expected load, size the container memory limit to the real working set verification working-set memory levels off below the limit and the breach projection clears on the next baseline FINGERPRINT krnali-portal/krnali-portal/portal:memory-breach RECURRENCE 132 occurrences, first seen 2026-07-31, last seen 2026-08-03 ACTIVE False TrendNoLongerRising the working-set series is judgeable and no longer shows the sustained rise this forecast was built on OBSERVABLE True SeriesJudgeable the series behind this finding is present and sufficient for a verdict
View sanitised Finding YAML · How this prediction was calculated
The lead time is a straight line, extended. krnali samples container working-set memory from metrics-server into a per-series store, fits an ordinary least-squares regression over the window, and extends that line until it crosses the container memory limit. In this record the fit gives 513.3 KiB/h against a 256.0 MiB limit, which is the 485h28m estimate. The range is a fixed band of plus or minus 20 percent around the estimate, documented as a band rather than a standard error because the fit does not expose one.
A slope alone is not a trend. Before a forecast is emitted the series has to clear a gate ladder: enough samples and history to be out of cold start, a usable fit over the whole window and over the trailing window, a Mann-Kendall rank test agreeing the rise is monotonic rather than the work of two outliers, a limit the line actually reaches, and a confidence above the emission floor. This record shows Mann-Kendall Z=11.9 in evidence step 3.
Confidence is a published formula, not a judgement. It is
60 · R² + 25 · min(span / 60min, 1) + 15 · min(recurrence / 3, 1),
rounded and banded into low, medium or high. You can check this record by
hand. R² is 0.27, the window span is 21m and the symptom had recurred 132
times, so 60 × 0.27 + 25 × 0.35 + 15 × 1 = 39.95, which rounds to the 40 the
object carries. Same evidence, same number, every time.
That number is deliberately not high. A 21 minute window with an R² of 0.27 is a weak fit, and the formula says so rather than rounding it up. A tool that scored this 90 would be lying.
The forecast was withdrawn later, for a different reason. A weak fit lowers
confidence; it does not close a record. The operator set
Active=False with the reason TrendNoLongerRising
because the sustained rise the forecast was built on had stopped. That is
the condition printed at the foot of the record above.
Provenance on the lead time reads deterministic because a
model was not involved in producing any figure on this object.
The API server will tell you a pod restarted. It will not tell you whether that matters, what changed at the same time, or what you should do about it. That gap is the whole product.
Crash loops, image and configuration failures, HPA saturation, node pressure, and rollout regressions. Findings deduplicate on a stable fingerprint and count recurrence.
Trend forecasts are withdrawn when their supporting rise stops. Reactive detectors now resolve their own findings after the symptom has been absent for a measured quiet window. Missing or stale evidence does not establish recovery. Acknowledgement in the portal is separate from symptom resolution.
Three forecasts today, all deterministic: a container trending toward its memory limit, a ResourceQuota running out of headroom, and a rollout that is regressing while it is still progressing. The first two are arithmetic over a learned baseline that survives operator restarts; the third carries a lead time only while the rollout is still in flight, and is filed as a reactive finding once it has completed.
Those three are the whole of it. None of them is a general prediction of anything else, and the page will say so when there are more.
Every finding carries an ordered evidence chain, the signals it read and where each came from, the deviation from learned normal, the change that coincided with it, and an advisory whose steps state their blast radius, their rollback, and whether they are reversible. The advisory is assembled from computed facts by fixed templates. The portal can optionally explain a workload through a customer-run model endpoint; measurements and confidence remain deterministic.
Different shapes, same cluster, all captured on the same day. They are printed here including the parts that are unflattering, because a page of curated output would prove nothing.
A cluster component, not a workload of ours, restarting on a fatal startup error. This is what a reactive finding contains: the restart count, the backoff state, the last termination reason, a bounded tail of the previous container's log, the change that coincided with it, and a two line advisory. No root cause, no stack analysis, no action steps. The product does not claim more than it observed.
FINDING krnali-09ff37ddf61003dd NAMESPACE local-path-storage SUBJECT local-path-provisioner-7b8c8ddbd6-9m4nq is crashlooping (4 restarts) SEVERITY info KIND reactive OBSERVED SIGNALS restartCount 4 pod/local-path-provisioner-7b8c8ddbd6-9m4nq lastState.terminated.reason Error pod/local-path-provisioner-7b8c8ddbd6-9m4nq EVIDENCE CHAIN 1. container local-path-provisioner restarted 4 times in CrashLoopBackOff=false; last termination: Error 2. coincident change: ReplicaSet local-path-provisioner-7b8c8ddbd6 (revision 1) rolled out at 2026-07-24T13:10:06Z (outside the correlation window); images: docker.io/kindest/local-path-provisioner:v20250214-acbabc1a 3. previous log tail for container local-path-provisioner time="2026-07-29T19:51:37Z" level=fatal msg="Error starting daemon: invalid empty flag helper-pod-file and it also does not exist at ConfigMap local-path-storage/local-path-config with err: Get \"https://10.96.0.1:443/api/v1/namespaces/local-path-storage/configmaps/local-path-config\": dial tcp 10.96.0.1:443: i/o timeout" 4. same symptom already recorded 599 time(s), first seen 2026-07-29T19:51:52Z ADVISORY inspect the previous-container log tail and the coincident change; roll back the change if it is the cause verification container local-path-provisioner stops restarting and stays Ready with no CrashLoopBackOff FINGERPRINT local-path-storage/local-path-provisioner:crashloop RECURRENCE 600 occurrences, first seen 2026-07-29, last seen 2026-08-03 ACTIVE True SymptomObserved the symptom that raised this finding is currently observed OBSERVABLE True SeriesJudgeable the series behind this finding is present and sufficient for a verdict
An archived record. It was last reconciled on 2026-07-30, before the operator could record lifecycle conditions, so it carries none and its state today is unknown. It is here because the evidence is the point: a real registry error, an explicit contrast against the prior revision, a change inside the correlation window, and two action steps that are both reversible and both yours to run.
FINDING krnali-4066cf03b05203ef NAMESPACE krnali-portal SUBJECT krnali-portal: new revision 3 is regressing (ImagePullBackOff) SEVERITY critical KIND reactive OBSERVED SIGNALS newRevision 3 deployment/krnali-portal signature ImagePullBackOff pod/krnali-portal-6cf85594cb-lsmmj oldRevision 2 deployment/krnali-portal DEVIATION 3 observed against 2 baseline rollout.revision new ReplicaSet revision 3 shows ImagePullBackOff the prior revision did not EVIDENCE CHAIN 1. new revision 3 failing on container portal: ImagePullBackOff Back-off pulling image "krnali-portal:dogfood-757b657": ErrImagePull: failed to pull and unpack image "docker.io/library/krnali-portal:dogfood-757b657": failed to resolve reference "docker.io/library/krnali-portal:dogfood-757b657": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed 2. prior revision 2 did not exhibit this signature the signature appeared with the new rollout, not before it, so it is a regression not a pre-existing fault 3. coincident change: ReplicaSet krnali-portal-6cf85594cb (revision 3) rolled out at 2026-07-30T05:59:35Z (within the correlation window); images: krnali-portal:dogfood-757b657 4. the new revision is already carrying full traffic the rollout has completed, so the regression is live on every replica 5. same symptom already recorded 1 time(s), first seen 2026-07-30T05:59:37Z ADVISORY roll back to the prior revision or fix the new image/config; krnali does not roll anything back itself $ kubectl rollout undo deployment/krnali-portal target deployment/krnali-portal reversible yes blast radius workload rollback: re-apply the new revision once its image/config is fixed (kubectl rollout undo to it, or re-apply the manifest) $ kubectl rollout pause deployment/krnali-portal target deployment/krnali-portal reversible yes blast radius workload rollback: kubectl rollout resume deployment/krnali-portal fix at source fix the container image or config the new revision introduced, then re-roll forward if the change is unsafe to run at all, hold the rollout until the image is corrected verification the new revision's error signature is absent from its pods kubectl rollout status completes cleanly with all replicas ready on the corrected revision FINGERPRINT krnali-portal/krnali-portal:rollout-regression RECURRENCE 2 occurrences, first seen 2026-07-30, last seen 2026-07-30 LIFECYCLE no conditions recorded on this object
Rendered in full further up the page. It is a closed record: the operator withdrew its own forecast at 2026-08-03T09:54:13Z with the reason TrendNoLongerRising, because the rise it was built on had stopped. Read the full record.
nali verify-rbac prints what the account can and cannot do.
The claim is checkable in three places. The ClusterRole in the chart, which you
can read with helm template before you install anything. The
guardrail suite in the repository, which fails the build when a verb outside
get, list and watch appears on a resource krnali does not own. And
nali verify-rbac, which you run on demand against your own cluster:
step 4 of the install notes exists to make you run it.
The Ops observer writes its own Finding and Baseline resources. It cannot patch,
delete, scale, restart, or execute inside customer workloads. pods/exec
does not appear in any rule of the ClusterRole. nali verify-rbac
prints the proof for your own audit, and it ships with a negative control
that fails loudly if every answer comes back allowed.
Cluster telemetry remains inside the customer-controlled boundary by default. krnali sends data externally only through destinations the customer explicitly configures, such as Jira or a customer-owned model endpoint. There is no phone-home telemetry, and no krnali-held corpus is built from your cluster.
The portal ships with a default-deny NetworkPolicy, a request body cap, an SSRF fence, and an auth gate that fails closed on any non-loopback listener. Customer resources remain read-only. Verified decisions may write to explicitly permitted krnali-owned ActionRequests and AutomationPolicies. Workload changes use the separately installed and authorised krnali-act.
When krnali cannot reach a verdict on a series it marks the finding unobservable and names the gate it failed: the series is still cold, the source stopped delivering samples, there is no usable fit over the window, or the window is too sparse to support a sustained-trend verdict. Only a series that clears every one of those gates is allowed to resolve a finding, so silence is never read as recovery. Absence of evidence is never rendered as health.
Ops and Axiom use the same evidence-bound reasoning architecture. It gathers attributable evidence, compares competing explanations, records uncertainty, and emits typed proposals. It cannot authorise or execute its own conclusions. A model may explain or investigate the evidence when configured. It never authors measurements, thresholds, confidence, or predicted lead time. Operator advisories use fixed templates. The portal's optional workload explanation uses a configured customer-run model endpoint and rejects numbers absent from its evidence pack. This does not verify every prose claim.
krnali is not a metrics store, a dashboard, or a pager. It sits on top of the systems you already run and answers a different question.
The initial cohort is deliberately narrower than the long-term product.
Founding teams get the full private-preview product free for twelve months, direct engineering support from the people building it, and influence over the roadmap. In exchange we ask for a 30-minute onboarding session and regular feedback during the first three months.
Applications arrive by email, at [email protected]. The button opens a message with the subject and a short template already filled in. If your mail client does not open, write to the same address directly.
Tell us four things:
What happens next: we read every application and reply, including when the answer is no. Teams we take on get an architecture call, an installation plan, and private chart access. This page is static, runs no analytics, sets no cookies, and stores nothing about you.
Engineering assurance for planned change. Axiom discovers the target environment, qualifies application requirements against customer standards, produces governed implementation plans, verifies the result, and maintains a living evidence-backed as-built.
Approved direction. No code. Ops is the product you can run.
Cluster telemetry remains inside the customer-controlled boundary by default. krnali sends data externally only through destinations you explicitly configure, such as Jira or a customer-owned model endpoint. There is no krnali-operated service in the data path and no phone-home telemetry.
The Ops observer does not modify customer workloads. A separate, explicitly authorised krnali-act installation can perform registered actions with approval, verification and rollback handling. The current action raises a Deployment container's memory limit. Read the Ops update for that boundary.
A read-only ServiceAccount: get, list and watch on the resources it observes, and full verbs only on its own krnali.io group. pods/exec appears in no rule. The operator does not announce any of this at startup, and it should not have to be believed: run nali verify-rbac, which step 4 of the install notes tells you to run. It asks your cluster's own authorizer what the ServiceAccount may do, prints every answer, and exits non-zero if any of them is unexpected.
No. Detection, baselining, predictions, evidence construction and reports work without a model. The optional portal workload explanation uses a customer-run, Ollama-compatible endpoint and is off by default. No public-model fallback is used.
Yes. Core observations, measurements, confidence scores and forecasts are deterministic. Operator advisories use fixed templates. An optional model explanation in the portal does not author those values or gain authority to act.
Yes. Detection and the memory baseline need the Kubernetes API and metrics-server, and nothing else. Prometheus is optional, and today it adds workload panels in the portal rather than feeding the operator's own reasoning.
The finding is marked unobservable and the specific gate is named: the series is still cold, the source stopped delivering samples, it has no usable fit, or the window is too sparse to judge. The finding's state is left as it was rather than resolved. A source going dark is never rendered as health.
Not publicly. The chart and container images are unpublished and the source repository is private, so there is no anonymous pull and no copyable command to give you. Founding teams get private chart access and direct installation support.
The chart is cloud neutral. Any conformant Kubernetes cluster.
Ops watches what is running. Axiom governs what is about to change. Ops is in private preview. Axiom is a proposed direction with no code behind it.
The Karnali, one of four rivers flowing from Mount Kailash. Pronounced kar nah lee. The CLI is nali.