# How liken works

This page gives the model in one read: what a machine runs, how a
fleet changes, and where your part of the system lives. The guides
assume this model.

## One image, plus your layer

Every machine runs the same operating system image. The image is a
read-only squashfs file, built by the project and published in every
release. Nothing on a machine edits it.

Everything that makes a cluster yours lives in one small archive: the
deployment layer. It holds your Cluster document, your Machine
manifests, and the cluster's identity, which is its certificate
authorities and its join token.
[`liken layer`](/docs/reference/cli/#liken-layer) packs the archive,
and every boot loads the image and your layer together.

A machine has no package manager and no configuration files to edit.
What a machine runs is the image plus the layer, and both are
declared.

## Two boot slots

Each machine keeps two copies of the operating system, in slots named
A and B. The machine runs from one slot and downloads upgrades into
the other. An upgrade reboots into the new slot one time, as a trial.
A trial that fails, in any way, ends with the machine back on the
slot it already proved. [Roll back](/docs/guides/rollback/) describes
the fallback paths.

An installation writes slot A. The first upgrade fills slot B, and
after that the slots alternate.

## Two resources drive everything

A `liken` cluster is an ordinary Kubernetes cluster with two extra
resources:

* A [Cluster](/docs/reference/cluster/) declares the fleet in one
  document: the release that every machine runs, the network, and
  the settings that the machines share.
* A [Machine](/docs/reference/machine/) declares one machine: its
  disks, its network ports, and its kernel modules. The machine
  reports what it observes in the resource's `status`.

You operate the fleet when you edit these two resources with
`kubectl`. A machine has no shell and no SSH server. Each machine
reads the two documents and converges to them.

A change applies with the smallest disruption that it needs. Some
values apply live, within seconds. Some restart k3s in place, and
the pods stay up. Some wait for the machine's next boot, whatever
causes it. Some need a reboot of their own. A machine that needs a
disruption stages the change, reports it in the Machine's
`status.pending`, and waits for its turn under the cluster's
disruption budget.

A machine that agrees with every document still reboots for one
reason: because you asked it to.
[`liken request-reboot`](/docs/reference/cli/#liken-request-reboot)
is that request, for a driver that bound the wrong device or a
machine you are experimenting on. It takes the same turn, the same
cordon, and the same drain as every other reboot.

## Releases and the channel

A release is a set of files with a version such as `2026.07.20-001`.
Releases are published on
[the release channel](/docs/reference/release-channel/), a directory
that any web server can share. Your Cluster names the channel, pins
each release by the digest of its release document, and points
`spec.version` at the release to run. Each machine downloads from
the channel, verifies every byte against the pinned digest, and
takes its turn to reboot. [Upgrade the
fleet](/docs/guides/upgrade/) is one edit to the Cluster.

## Where to go next

[Install a cluster](/docs/guides/install/) gives the steps from a
downloaded release to `kubectl get nodes`. When something does not
go to plan, [Troubleshoot](/docs/guides/troubleshoot/) maps each
symptom to the field that explains it.


# The hardware operators

The hardware operators are operators that you install on a `liken`
cluster. Each one publishes one kind of machine hardware as DRA
devices, so a workload claims that hardware the way
[Give a workload a device](/docs/guides/devices/) shows, with no
privilege and no host path.

Each operator is a DRA driver of its own, separate from the
operating system's driver that [Devices](/docs/reference/devices/)
describes. The operating system publishes the hardware a machine
holds, such as a Bluetooth radio, a GPU, or a sound card. A hardware
operator publishes what that hardware serves, at the grain a
workload asks for: a paired controller, a monitor output, an audio
output.

The name of an operator's device class is also the hostname of its
manual. Each manual gives the install steps and the claims for its
devices:

* [bluetooth.liken.sh](https://bluetooth.liken.sh) publishes paired
  Bluetooth controllers. The source is
  [liken-sh/bluetooth-operator](https://github.com/liken-sh/bluetooth-operator).
* [display.liken.sh](https://display.liken.sh) publishes monitor
  outputs. The source is
  [liken-sh/display-operator](https://github.com/liken-sh/display-operator).
* [audio.liken.sh](https://audio.liken.sh) publishes audio outputs.
  The source is
  [liken-sh/audio-operator](https://github.com/liken-sh/audio-operator).


# Install a cluster

This guide gives the steps from a downloaded release to `kubectl get
nodes`. You do not need the repository, and you do not need to build
the system. The release contains all the files, including the `liken`
toolkit that does these steps.

You need:

* One or more x86-64 machines with blank disks. UEFI firmware and
  BIOS firmware both work. A machine needs 1 GB of memory, and more
  for its workloads. The default disk layout asks for about 20 GB.
  The installation erases the disks it claims.
* A USB stick. The installation image overwrites it.
* A Linux workstation with `kubectl`.
* A keyboard and a screen on each machine, for the installation. A
  person selects the boot entry and answers the message at the end. If
  a machine has no screen, use a serial console. Build the stick with
  `-console`, as step 4 shows.

After the installation, the machine needs nothing attached. It boots,
joins the cluster, and gets its configuration from the cluster.

## 1. Download a release

Releases are at [releases.liken.sh](https://releases.liken.sh/).
Each release is a directory with the name of its version, for example
`2026.07.20-001/`. Select a version, then download the toolkit and the
release document:

    curl -fLO https://releases.liken.sh/<version>/liken
    curl -fLO https://releases.liken.sh/<version>/release.yaml

Verify the files that you downloaded. `release.yaml` contains a
`sha256:` line for each file in the release. The release's page on
GitHub gives the digest of `release.yaml` itself. Thus the chain of
trust starts with a value that you can read:

    sha256sum release.yaml liken

Compare the first digest with the release page. Compare the second
digest with the `liken` entry in `release.yaml`. Then make the toolkit
executable, and download and verify the rest of the release:

    chmod +x liken
    ./liken fetch -digest sha256:<hex> https://releases.liken.sh <version> channel

The `-digest` value is the digest of `release.yaml` from the release
page. [`liken fetch`](/docs/reference/cli/#liken-fetch) writes the
whole release into `channel/<version>/`, and it checks each byte
against the document. [The release
channel](/docs/reference/release-channel/) describes the layout and
the trust chain.

## 2. Describe your cluster

    ./liken new mycluster

[`liken new`](/docs/reference/cli/#liken-new) asks a short series of
plain questions: the names of your machines, which machines are
leaders, their addresses, and their disks. Then it writes `mycluster/`:
a `cluster.yaml` file and one manifest for each machine. Each field has
a comment that explains what the field means. Keep `mycluster/` in
version control. That directory is the declaration of your cluster.

If you do not know a machine's disks, its network interface names, or
the extra drivers that it needs, write your best estimate now. The
hardware report in step 5 boots the machine, examines its hardware, and
writes a corrected manifest.

## 3. Mint the cluster's identity

    ./liken mint mycluster/identity

[`liken mint`](/docs/reference/cli/#liken-mint) creates the identity:
the set of certificate authorities and the join token that make your
machines into one cluster. The files include private keys. The
scaffold's `.gitignore` keeps them out of version control.

If you already run a k3s cluster, do not run `liken mint`. Use
[`liken adopt`](/docs/reference/cli/#liken-adopt) instead, to join
machines to that cluster. [Adopt an existing k3s
cluster](/docs/guides/adopt/) gives the steps.

## 4. Build the install stick

    ./liken layer mycluster mycluster/identity mycluster/deployment.cpio
    ./liken stick channel/<version> mycluster/deployment.cpio mycluster/install.img

[`liken layer`](/docs/reference/cli/#liken-layer) packs the layer: the
small archive that contains your manifests and your identity.
[`liken stick`](/docs/reference/cli/#liken-stick) joins the release
with your layer into one bootable disk image.

For a machine with no screen, name its serial port when you build the
stick:

    ./liken stick -console ttyS0 channel/<version> mycluster/deployment.cpio mycluster/install.img

The boot menu and all the messages then go to that port. Thus you can
install the machine through a serial cable or a remote console. The
machines keep the setting, so their consoles stay available after the
installation.

Check the device name of your USB stick before the next command. The
command overwrites the device.

    sudo dd if=mycluster/install.img of=/dev/YOUR-STICK bs=4M oflag=direct status=progress

## 5. Boot each machine from the stick

Connect the stick and boot the machine. For the first boot, it is
possible that you must open the firmware's boot-device menu. The
stick's menu appears. It has two entries for each machine, and it ends
with one entry for the report:

    install as big
    wipe and reinstall as big
    install as little
    wipe and reinstall as little
    liken hardware report

The menu has no timeout. You must select an entry.

### First, run the hardware report

Select `liken hardware report`. This boot makes no changes to the
machine's disks. It loads the drivers for the disks and the network
ports, records the devices that appear, and writes a proposed manifest
to the stick as `hardware-report.yaml`. It prints the full proposal,
then it holds:

    liken: this report was written to the stick as hardware-report.yaml; press Enter to reboot.

The report loads storage drivers and network drivers only. A machine
needs these two types of driver to install itself and to join a
cluster. A driver for other hardware would change the machine while a
person is in front of it. A display driver, for example, takes control
of the screen that the report prints to. The machine reports its other
hardware in the node's status after the node runs, in the list of
unclaimed hardware.

Press Enter to reboot the machine. The report changes no disk, so you
can run it as many times as necessary: after you install a disk, after
you connect a cable, or to examine a change before you install.

Take the stick to your workstation and read `hardware-report.yaml`. The
file is a valid Machine manifest. Each line has a comment with the data
that the report found: the drivers that each device needs, in load
order; each disk's size, model, and path; each interface's name, MAC
address, and link state. Copy the `spec.modules`, `spec.network`, and
`spec.storage` sections into this machine's manifest in `mycluster/`.
Edit the parts marked `CHANGE-ME`. Then build the stick again with step
4, so that it contains the corrected manifest.

Three parts of the proposal need your judgement.

The storage sizes agree with the disks that the report measured, so you
can install with them unchanged. Two roles still need your attention.
`clusterState` contains the k3s database, the TLS files, and the
containerd image store. Thus the workloads on the node set its size.
Increase it if this machine runs many images, or large images. Select
this number carefully. The Cluster's
[`spec.runtime.kubelet.imageGC`](/docs/reference/cluster/#specruntimekubeletimagegc)
section controls how much of this filesystem the image store keeps. `podStorage` comes after
`clusterState` on the disk, so `clusterState` cannot become larger
after the installation.
Set `podStorage` to the size that your workloads' volumes need. If the
report had to decrease either size, it says so in the file.

The proposal declares only the network ports that had a cable when the
report ran. It lists the ports with no cable below them, as comments,
with their names and MAC addresses. Remove the comment marks from a
port after you connect its cable. A declared port with no cable delays
each boot, because the machine waits a maximum of thirty seconds for
its DHCP lease.

The report can give a warning that a disk needs a driver that this
image does not contain on its boot path. You cannot put such a driver
in `spec.modules`, because the machine reads that list only after it
finds its disks. That machine needs an image built with the driver in
its boot modules. The proposal gives the name of the driver, and it
does not put that disk in the layout.

Run the report on each new machine. It gives you the disks, the
interface names, and the drivers that a datasheet does not give.

### Then install the machine

Select `install as <name>` for the machine in front of you. The machine
partitions its blank disks, copies the operating system onto them,
registers itself with its firmware, then it holds:

    liken: installed to slot A; remove the stick, then press Enter to power off; the next power-on boots from the disk.

Remove the stick before you press Enter. The stick is first in the boot
order. If you power on the machine with the stick connected, the menu
appears again. Press Enter. The machine powers off. Power it on again.
From that time, it boots from its own disk.

If the installation does not complete, it prints the error, lists the
machine's disks, and holds:

    liken: press Enter to power off

Correct the cause and boot the installation again. An installation is
idempotent, so a second attempt is safe.

### To reinstall a machine that liken installed

`install as <name>` claims only blank disks. It refuses a disk that it
does not recognize, so it does not erase data that it did not write.
This protects your data, but it also prevents the plain installation
from replacing an installation that liken made. To replace one, select
`wipe and reinstall as <name>`. In one boot, it erases the disks that
this machine's manifest declares, then it installs the machine. Your
selection of the entry at the keyboard is the confirmation. It ends
with the same held messages as a plain installation.

A reinstallation erases all the data that it claims, on each disk that
the manifest declares. It erases the cluster state: this node's copy of
the k3s database, its certificates, and the images that it unpacked. It
also erases the volumes that your workloads claimed on this node. The
machine returns as a new member with the same name, and not as the
member it was.

A disk replacement is different. If you install a new blank system disk
and select the plain `install as <name>`, the installation claims the
blank disk and recognizes the data disk that it wrote before. Thus the
cluster state on that disk stays. Only `wipe and reinstall` erases a
disk that liken claimed.

If you reinstall with a different layout, the machine's document in the
cluster continues to describe the old layout, and the two disagree.
Storage roles can only increase in size, so the machine reports
`SpecConverged: False` with the reason `StagingRejected`, and it gives
the size that the disk now has. Correct this in this order. First, let
the machine boot and publish its status. Then edit the Machine resource
in the cluster to the layout that it now has. The rule compares your
spec with the sizes that the machine last booted with, so the cluster
accepts the edit only after the machine reports them. Also change the
machine's manifest in `mycluster/` to the same layout, because the next
stick and the next reinstallation start from that copy.

Use the same stick for each machine. Start with the first leader. The
machines find each other at the addresses that you declared. The
leaders make the control plane, and the followers join it.

## 6. Talk to your cluster

    ./liken kubeconfig mycluster

[`liken kubeconfig`](/docs/reference/cli/#liken-kubeconfig) writes
`mycluster/identity/kubeconfig`, an administrator credential. It points
at the `endpoint:` in your `cluster.yaml`. If your workstation reaches
the cluster at a different address, pass `-server`:

    ./liken kubeconfig -server https://203.0.113.5:6443 mycluster

Then:

    kubectl --kubeconfig mycluster/identity/kubeconfig get nodes

Each machine shows as Ready. The cluster is now an ordinary Kubernetes
cluster, with two more `liken` resources:

    kubectl get clusters      what the fleet is, as one document
    kubectl get machines      each machine, as the OS sees it

Edit those resources to make configuration changes. The
[Machine](/docs/reference/machine/) and
[Cluster](/docs/reference/cluster/) pages describe each field. When a
new release is available, [Upgrade the fleet](/docs/guides/upgrade/)
moves each machine to it with one edit. If a step did not go as this
guide says, [Troubleshoot](/docs/guides/troubleshoot/) starts from
the symptom.


# Adopt an existing k3s cluster

Adoption joins `liken` machines to a k3s cluster that `liken` did not
create. You can replace the cluster's machines one at a time while the
cluster continues to serve. The procedure does not export or restore
cluster state. Each new member receives the data through usual
replication.

Adoption works with any k3s cluster that uses embedded etcd, on any
operating system.

Read this behavior before you start. A `liken` server disables the
bundled k3s components (traefik, servicelb, metrics-server), unless
your `cluster.yaml` declares them as features. The change applies to
the whole cluster when the first `liken` server joins. If your
workloads need a bundled component, declare its feature in your
`cluster.yaml`, or start a replacement before that first join.

## 1. Harvest the identity

On any server of the existing cluster, as root:

    cd /var/lib/rancher/k3s/server
    tar czf /tmp/identity.tgz token \
        tls/server-ca.{crt,key} \
        tls/client-ca.{crt,key} \
        tls/request-header-ca.{crt,key} \
        tls/service.key \
        tls/etcd/server-ca.{crt,key} \
        tls/etcd/peer-ca.{crt,key}

Copy the archive to your workstation and unpack it into a private
directory, for example `harvest/`. The archive contains only the
certificate authorities and the join token. It does not contain the
server's leaf certificates, because each server signs its own
certificates with the shared roots.

## 2. Arrange the identity

    ./liken new mycluster
    ./liken adopt harvest mycluster/identity

[`liken adopt`](/docs/reference/cli/#liken-adopt) puts the harvested
files into the identity directory in the same arrangement as
[`liken mint`](/docs/reference/cli/#liken-mint). It refuses an
incomplete harvest, and it makes sure that the token agrees with the
harvested certificate authority. After this step, the later steps are
the same for a minted identity and an adopted identity.

## 3. Declare the adoption

Edit `mycluster/cluster.yaml`:

* Set [`spec.origin`](/docs/reference/cluster/#spec--origin) to
  `Adopted`.
* Set `spec.endpoint` to the existing cluster's join URL.

The datastore of an adopted cluster already exists. Each `liken` leader
joins it through the endpoint, and no `liken` machine initializes a new
datastore. A second datastore next to a live one divides the cluster
into two clusters.

### The document claims the whole cluster

[`spec.features`](/docs/reference/cluster/#spec--features) is an
opt-in list, and `liken` reads it as a statement
about the cluster, not only about the machines you install. A feature
that the document does not name is a feature that the cluster
retracts, and a retracted feature is one `liken` tears down.

The `flux` feature shows what this means for an adopted cluster. If
the cluster already runs Flux, and your `cluster.yaml` does not
declare `flux`, that omission reads as a retraction.

`liken` does not delete that Flux. It removes a Flux installation only
when the `flux-system` namespace carries a `liken.sh/feature=flux`
annotation, and that annotation arrives only when the document
declares the feature. A document that never declared `flux` never put
it there, so the teardown deletes nothing and the Cluster reports the
refusal:

    kubectl describe cluster

The `FluxTeardown` condition names what the teardown declined, and the
one command that gives the installation to `liken`. Decide before you
install the first machine:

* To keep the existing Flux and run it yourself, leave `flux` out of
  `spec.features`. The sync keeps running. The condition stays, and it
  is the report that `liken` manages none of it.
* To run the fleet from git with `liken`, declare the `flux` feature.
  [Run the fleet from git](/docs/guides/gitops/) has the steps, and
  its last section covers an installation that already exists.

## 4. Install the liken machines

Build the stick and install each machine as in
[Install a cluster](/docs/guides/install/). Start with the first
leader. Each machine joins the existing cluster directly. The existing
machines continue to serve during the procedure.

`liken` machines have a `liken.sh/machine=true` node label, and the OS
workloads schedule only onto nodes with that label. Thus the foreign
nodes get no changes.

## 5. Rotate the old servers out

Remove the foreign servers one at a time:

    kubectl delete node <old-server>

For a k3s server, the deletion of the node also removes the etcd
member. Wait for the cluster to become stable before you remove the
next server, to keep the quorum. If `spec.endpoint` gives a foreign
server, change it to a `liken` leader's address before you remove that
server.

## 6. Promote the cluster

After you remove the last foreign member, edit the Cluster resource and
set `spec.origin` to `Founded`:

    kubectl edit cluster

This is the only permitted edit to the field. The promotion makes no
change to the running fleet. It has an effect only if you build the
cluster again: the founding leader of a founded cluster can create the
datastore again.

Only a boot reads `spec.origin`, so the promotion reboots nothing. Each
machine stages the new document and reports `StagedForNextBoot` on its
`ClusterConverged` condition. The machine applies the document at its
next boot, whatever causes that boot. The same holds for an edit to
`spec.endpoint`.


# Add machines to a cluster

You add one declared machine at a time. Describe the new machine, build
the install stick again, and boot the machine from it. This procedure
makes no changes to the running machines.

## 1. Describe the new machine

Add a manifest file to your deployment directory, with the manifests
that `liken new` wrote. Copy an existing machine's file and change the
name, the addresses, and the storage for the new hardware. The comments
in the file explain each field.

## 2. If the new machine is a leader

If the new machine will run a control plane, add its name to
[`spec.leaders`](/docs/reference/cluster/#spec) in two places:

* On the live cluster, with `kubectl edit cluster`. The machines read
  this document, and each machine gets its role from it.
* In your `mycluster/cluster.yaml`. The file applies only to a new
  cluster. But if you keep it correct, a new build gives the cluster
  that you run.

Make the live edit before you install the machine. Keep an odd number
of leaders, so that the datastore can always make a majority.

## 3. Rebuild the install stick

A change to the deployment layer needs new install media. Use the
release that your fleet runs. `kubectl get clusters` shows it in the
VERSION column. Pack the layer and the stick again with
[`liken layer`](/docs/reference/cli/#liken-layer) and
[`liken stick`](/docs/reference/cli/#liken-stick):

    ./liken layer mycluster mycluster/identity mycluster/deployment.cpio
    ./liken stick channel/<version> mycluster/deployment.cpio mycluster/install.img

Check the device name before the next command. The command
overwrites the device.

    sudo dd if=mycluster/install.img of=/dev/YOUR-STICK bs=4M oflag=direct status=progress

## 4. Boot the new machine from the stick

The stick's menu now lists the new machine, with an `install as
<name>` entry and a `wipe and reinstall as <name>` entry. If you do not
know this hardware, boot `liken hardware report` first. It writes
`hardware-report.yaml` to the stick, so you can correct the machine's
disks, interfaces, and drivers before you install. [Install a
cluster](/docs/guides/install/#5-boot-each-machine-from-the-stick)
describes the report and the held console messages fully.

Select `install as <name>`. The machine installs itself and holds the
console:

    liken: installed to slot A; remove the stick, then press Enter to power off; the next power-on boots from the disk.

Remove the stick, then press Enter. The machine powers off. Power it on
again, and it boots from its own disk.

## 5. Watch it join

    kubectl get machines

The new machine appears, then it becomes Ready. It also appears in
`kubectl get nodes`. The rows of the other machines do not change.


# Upgrade the fleet

One edit to the Cluster moves every machine to a new release. Each
machine downloads the release, verifies every artifact, and reboots
when the cluster grants its turn. You do not rebuild the media, and
you do not touch the machines.

## 1. Find the release

The channel at [releases.liken.sh](https://releases.liken.sh/) lists
every release. The cluster also polls the channel:

    kubectl get clusters

The AVAILABLE column shows the latest version that the channel
announces. To make
the cluster poll the channel now, set the `liken.sh/check-releases`
annotation to a new value:

    kubectl annotate cluster --all --overwrite liken.sh/check-releases="$(date -Is)"

The value of the annotation has no meaning. The change of the value is
the request.

An upgrade needs two facts: the version, and the digest of that
release's `release.yaml`. The release's page on GitHub gives both, as
a catalog entry you can copy. To compute the digest yourself:

    curl -fsSL https://releases.liken.sh/<version>/release.yaml | sha256sum

## 2. Catalog the release and set the target

    kubectl edit cluster

Add the release to
[`spec.releases.catalog`](/docs/reference/cluster/#specreleasescatalog),
and point [`spec.version`](/docs/reference/cluster/#spec--version)
at it:

    spec:
      version: "2026.07.20-001"
      releases:
        source: https://releases.liken.sh
        catalog:
          - version: "2026.07.20-001"
            digest: sha256:<hex>

If `spec.version` names no catalog entry, the API refuses the change
while your edit is still open. The digest is the start of
[the trust chain](/docs/reference/release-channel/#the-trust-chain).
The digest names the release document, and the release document names
the artifacts. Each machine checks every downloaded byte against one
or the other.

## 3. The rollout

Each machine that runs a different version:

1. Downloads the release into the boot slot it does not run from, and
   verifies each artifact against the digest chain.
2. Stages the change and asks the cluster for a reboot turn.
3. Cordons and drains its node when the cluster grants the turn. The
   PodDisruptionBudgets of the workloads apply during the drain. A pod
   that holds a DRA claim leaves before the pod that serves its
   driver. The driver stays to answer the kubelet's unprepare call.
4. Reboots into the new slot one time, as a trial. The trial is a
   success when the OS starts and rejoins the cluster, and the machine
   then boots that slot from then on. If the trial fails, the machine
   returns to the other slot without help:
   [Roll back](/docs/guides/rollback/) describes this.

The cluster grants turns within
[`spec.disruption.maxUnavailable`](/docs/reference/cluster/#specdisruption)
(the default is one machine at a time). Only one leader is down at a
time, whatever the budget says, because the datastore needs a majority
of the leaders.

If a machine's
[`rebootPolicy`](/docs/reference/machine/#spec--rebootpolicy) is
`Manual` (the default), the machine stages the change, reports
`RebootPending`, and waits for you. Grant the reboot with
[`liken approve-reboot`](/docs/reference/cli/#liken-approve-reboot):

    liken approve-reboot mycluster <machine>

The machine then takes its turn under the same disruption budget as
an `Auto` machine: it drains first, and only one leader is ever down
at a time. Set `rebootPolicy: Auto` on machines that must take their
turn without an operator.

## 4. Watch the rollout

    kubectl get machines

The LIKEN column changes to the new version one machine at a time, and
the phase of each machine shows its step in the rollout. The phase of
the Cluster shows Updating during the rollout, and Ready when the
rollout is complete.

If a machine with a granted turn does not return, the cluster sets its
`Progressing` condition to `False` with the reason `RolloutStalled`,
and grants no more turns until you examine the machine.


# Roll back

Two mechanisms return a machine to a version that works. If a new
version fails its first boot, the machine returns to the previous
version without help. You roll the fleet back deliberately when you
point the Cluster at an earlier version.

## The automatic fallback

Every machine keeps two boot slots, A and B. The machine runs from one
slot and writes downloaded releases into the other. An upgrade reboots
into the new slot one time only, as a trial:

* If the new kernel panics, the machine resets, and the firmware boots
  the proven slot. No software is involved.
* If the new version boots but does not rejoin the cluster in ten
  minutes, a watchdog reboots the machine, and the machine starts
  again on the proven slot.

In both cases, the machine serves on the version it ran before. Its
phase shows Blocked, its conditions show `RejectedLastBoot`, and
[`status.boot.systemRejection`](/docs/reference/machine/#statusbootsystemrejection)
records what happened. The rejection stays until you point
[`spec.version`](/docs/reference/cluster/#spec--version) at a
different version, so the machine does not try the failed version
again.

A bad release is never published again. To correct a bad release,
publish a release with the next serial number, add it to the catalog,
and point `spec.version` at it.

## Deliberate rollback

To move the fleet back to an earlier release, point `spec.version` at
that release:

    kubectl edit cluster

The version must still be in
[`spec.releases.catalog`](/docs/reference/cluster/#specreleasescatalog).
This is a reason to keep the old entries. The rollout is the same as
for an upgrade: each machine downloads the older release into its
inactive slot, verifies it, and reboots on its granted turn, one
machine at a time. The cluster continues to serve during the rollout.


# Run the fleet from git

The `flux` feature connects a cluster to a git repository. The cluster
runs [Flux](https://fluxcd.io), syncs the repository, and applies what
the repository holds. The repository then declares everything: the
Cluster document, the Machine documents, and your workloads. You make
each change with a commit, and the fleet converges to the repository.

You need:

* A cluster, either running or ready to install. [Install a
  cluster](/docs/guides/install/) has those steps.
* A private, empty git repository at a forge you can reach over SSH.
* The [Flux CLI](https://fluxcd.io/flux/installation/) on your
  workstation, for one command in step 1.

## 1. Lay out the repository

The cluster syncs one path of the repository. The `path` parameter
selects the path, and the default is the repository root. Use this
layout:

    flux-system/
      gotk-components.yaml    the Flux engine
    liken/
      cluster.yaml            the Cluster document
      node-1.yaml             one file per Machine

Export the engine manifest with the Flux CLI:

    flux install --export \
      --components=source-controller,kustomize-controller \
      > flux-system/gotk-components.yaml

The repository holds the engine manifest, because the repository
controls the engine. liken installs a pinned copy of these two
controllers one time, and only to make the first sync possible. After
the first sync, the cluster runs the engine from your repository. To
upgrade Flux, commit a change to this file. To add a controller,
commit a change to the same file.

Copy your `cluster.yaml` and machine manifests into `liken/`. These
are the files `liken new` wrote. Then add this annotation to the
Cluster document and to every Machine document:

    metadata:
      annotations:
        kustomize.toolkit.fluxcd.io/prune: disabled

The annotation prevents Flux from deleting the fleet's documents. Flux
deletes the objects that are no longer in the synced path. Without the
annotation, a commit that removes these documents also removes the
fleet's declaration from the live cluster. With the annotation, Flux
keeps the marked documents, and you must remove the fleet
deliberately.

Do not add `GitRepository` or `Kustomization` objects to the
repository. liken renders these two sync objects from the feature's
parameters. A copy in the repository conflicts with the rendered
object.

## 2. Declare the feature

Collect the forge's SSH host keys, and compare them with the keys the
forge publishes:

    ssh-keyscan github.com

Then declare the feature on the Cluster. On a running cluster, use
`kubectl edit cluster`. On a cluster you did not install yet, put the
same block in your `cluster.yaml` before you build the stick. The
fleet then syncs from the first boot.

    spec:
      features:
        flux:
          repository: ssh://git@github.com/you/fleet.git
          knownHosts: |
            github.com ssh-ed25519 AAAAC3NzaC1lZDI1...
            github.com ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNo...
            github.com ssh-rsa AAAAB3NzaC1yc2EAAA...

`repository` is required. The default for `branch` is `main`, and the
default for `path` is the repository root. `knownHosts` holds the
forge's host keys, one key per line. The keys are public material, so
they belong in the spec. They let the first clone verify the forge.
The [Cluster reference](/docs/reference/cluster/#spec--features)
describes each parameter.

On a running cluster, k3s restarts in place on each machine, one
machine at a time, to apply this edit. The machines and their pods
stay up.

## 3. Register the deploy key

The cluster creates its own SSH deploy key. The private half stays in
the cluster. Read the public half:

    kubectl get cluster -o jsonpath='{.items[0].status.flux.publicKey}{"\n"}'

Register this value at the forge as a deploy key for the repository,
and give it write access. On GitHub, the setting is in the
repository's Settings, then Deploy keys. The sync starts when the
forge accepts the key.
[`status.flux`](/docs/reference/cluster/#statusflux) describes the
lifecycle of the key.

## 4. Watch the first sync

    kubectl --namespace flux-system get gitrepositories,kustomizations

When both objects show Ready, the repository controls the fleet. The
first sync also replaces the installed engine with the copy from your
repository.

## 5. Work by commit

From now on, change the fleet with a commit. Add a workload manifest
under the synced path, and the cluster runs it. Edit a feature or a
Machine's disks in `liken/`, and the fleet converges as it does for a
live edit. To [upgrade the fleet](/docs/guides/upgrade/), commit the
new catalog entry and `spec.version` in `liken/cluster.yaml`.

## Rules for safe operation

* Never commit the sync objects. liken owns `GitRepository` and
  `Kustomization`.
* Keep the `prune: disabled` annotation on the Cluster and Machine
  documents.
* Monitor the memory. The repository sets how many controllers run.
  The `source-controller` and the `kustomize-controller` fit on a 1 GB
  machine with its workloads. Each controller you add uses memory that the
  workloads need. Add components one at a time, and look at
  `kubectl top nodes` after each one.
* A live edit changes field ownership. The API server records you as
  an owner of each field you change with `kubectl`, and git cannot
  delete a field that you own. After each manual repair, commit the
  same state to the repository. If a later commit must remove a field
  that you changed live, also remove that field live.

## Rotate, retract, recover

To rotate the deploy key, delete the Secret. The cluster creates a new
key pair in seconds, and you register the new public half at the
forge:

    kubectl --namespace flux-system delete secret flux-system

To turn the feature off, remove `flux` from `spec.features`. The sync
stops, and the cluster removes the engine, its namespace, and the
deploy key. The workloads that the repository deployed continue to
run. Retraction stops the sync. It does not remove the workloads. To
turn the feature on again, declare it again and register the new key
it creates.

Retraction removes only the Flux that `liken` installed. The next
section says how `liken` separates the two cases.

If someone deletes the engine by accident, the cluster installs it
again in about a minute, and the next sync restores the copy from the
repository.

## What liken owns

Declaring the feature puts this annotation on the `flux-system`
namespace:

    liken.sh/feature: flux

The annotation is the record that this installation is `liken`'s. The
teardown reads it before it deletes anything, and removes only what
carries it.

A cluster that never declares the feature never gets the annotation.
`liken` seeds nothing there, so a Flux that was running before `liken`
arrived keeps running, and retracting a feature the document never
declared deletes nothing. The Cluster says so:

    kubectl describe cluster

The `FluxTeardown` condition names what the teardown declined. A
cluster that `liken` founded before this record existed reports the
same thing until its namespace carries the annotation.

Declaring the feature is how you hand an existing installation to
`liken`. The namespace does not have to be new: `liken` applies its
own copy over whatever is there, and the annotation lands with it.
From that point retraction removes the engine, its namespace, and the
deploy key, exactly as it does for an installation `liken` made.
Writing the annotation by hand does the same thing:

    kubectl annotate namespace flux-system liken.sh/feature=flux


# Give a workload a device

This guide gives a pod a piece of hardware: a GPU for a transcoder,
and a USB adapter that one pod holds alone. No step here needs a
privileged pod or a host path.

[Devices](/docs/reference/devices/) describes what liken publishes and
why. The [hardware operators](/docs/concepts/hardware-operators/)
publish devices the operating system does not: paired Bluetooth
controllers, monitor outputs, and audio outputs.

## 1. See what a node offers

    kubectl get resourceslices
    kubectl get resourceslice <node>-liken.sh -o yaml

Each entry is one device you can claim:

    - name: pci-0000-00-02-0
      allowMultipleAllocations: true
      attributes:
        address: {string: "0000:00:02.0"}
        bus: {string: pci}
        class: {string: display}
        classCode: {string: "030000"}
        driver: {string: i915}
        modalias: {string: "pci:v00008086d000046D2..."}
        name: {string: Alder Lake-N [UHD Graphics]}
        product: {string: 46d2}
        renderNode: {bool: true}
        subsystem: {string: drm}
        vendor: {string: "8086"}
    - name: pci-0000-00-02-0-display
      attributes:
        address: {string: "0000:00:02.0"}
        bus: {string: pci}
        class: {string: display}
        classCode: {string: "030000"}
        displayNode: {bool: true}
        driver: {string: i915}
        modalias: {string: "pci:v00008086d000046D2..."}
        name: {string: Alder Lake-N [UHD Graphics]}
        product: {string: 46d2}
        subsystem: {string: drm}
        vendor: {string: "8086"}
    - name: pci-0000-00-02-0-i2c-dev
      attributes:
        address: {string: "0000:00:02.0"}
        bus: {string: pci}
        class: {string: display}
        classCode: {string: "030000"}
        driver: {string: i915}
        modalias: {string: "pci:v00008086d000046D2..."}
        name: {string: Alder Lake-N [UHD Graphics]}
        product: {string: 46d2}
        subsystem: {string: i2c-dev}
        vendor: {string: "8086"}

This machine publishes three devices for one GPU.

* `pci-0000-00-02-0` delivers the render node,
  `/dev/dri/renderD128`. More than one claim can allocate it, so
  several workloads encode, decode, and compute on this GPU at the
  same time.
* `pci-0000-00-02-0-display` delivers the card node,
  `/dev/dri/card0`, which carries modesetting. One claim allocates it,
  because one process at a time can drive a display.
* `pci-0000-00-02-0-i2c-dev` delivers the i2c monitor-control buses
  that i915 registers for each display output. One claim allocates it,
  because those buses are raw wires.

All three carry the same `address`, because they are one card. A claim
uses that to ask for halves of the same GPU. See
[Two requests, one card](#two-requests-one-card).

If the hardware you expect is not in the list, no driver is bound to
it. Look at the hardware the machine reports that it cannot drive:

    kubectl get machine <name> -o jsonpath='{.status.hardware.unclaimed}'

## 2. Declare the driver

A device becomes available for a claim when a driver binds it. Add the
module to [`spec.modules`](/docs/reference/machine/#spec--modules)
in the machine's manifest:

    spec:
      modules:
        - i915

Apply the manifest, and the machine loads the module. Most modules
load without a reboot. The Machine's `SpecConverged` condition shows
when the change has an effect. The device is in the node's slice in a
few seconds.

The hardware report names these modules for you, as comments, when you
install the machine. See
[Install a cluster](/docs/guides/install/#first-run-the-hardware-report).

## 3. Say what your workload needs

A `DeviceClass` is a named set of conditions on hardware. Write one
for each kind of device that your deployments ask for.

    apiVersion: resource.k8s.io/v1
    kind: DeviceClass
    metadata:
      name: gpu-render
    spec:
      selectors:
        - cel:
            expression: |
              device.driver == "liken.sh" &&
              has(device.attributes["liken.sh"].renderNode)

liken publishes an attribute only when it is true of the hardware, so
`has()` is the complete test. Guard an attribute a device may lack
with `has()` before you read it. The Kubernetes API treats an
unguarded read of a missing attribute as an evaluation error, and an
evaluation error aborts the whole allocation.

Do not select on `driver` alone, and do not select on `subsystem`
alone. Every device that liken publishes for one GPU carries the same
`driver`, and the card node carries the same `subsystem: drm` as the
render node. A class that matches more than one of them can allocate
the monitor buses, or the display, to your transcoder. `renderNode`
is the fact that names the half you want, and only that half carries
it.

This class matches any GPU with a DRM render node, on any machine in
the fleet. A selector that asks for a vendor and a product ID names
one model, and it stops matching on the next machine you buy.

## 4. Claim the GPU for a deployment

The deployment writes a claim against the class, and the claim
allocates the GPU. The claim is also what places the pod: the
scheduler picks a machine whose `ResourceSlice` offers a matching
device.

    apiVersion: resource.k8s.io/v1
    kind: ResourceClaim
    metadata:
      name: transcoder-gpu
      namespace: media
    spec:
      devices:
        requests:
          - name: gpu
            exactly:
              deviceClassName: gpu-render

The pod names the claim, and each container that needs the device
names the pod's entry:

    spec:
      template:
        spec:
          resourceClaims:
            - name: gpu
              resourceClaimName: transcoder-gpu
          containers:
            - name: ffmpeg
              image: ...
              resources:
                claims:
                  - name: gpu

The container receives every node the published device delivers. For
the GPU above, that is `/dev/dri/renderD128` only. The card node and
the `/dev/i2c-*` monitor-control nodes belong to the companion
devices, and each companion needs its own request. Nothing else
changes: no privilege, and no host mount. Your image supplies the
userspace driver, and the image's user must be able to open the node.
Check what the container received:

    kubectl exec deploy/transcoder -- ls -l /dev/dri

A device that carries `allowMultipleAllocations: true` serves more
than one claim: a second deployment writes its own `ResourceClaim`
against the same `DeviceClass`, and both deployments run. The
integrated GPU above carries it, so more than one transcoder can
allocate the render node at once.

Give a deployment that holds a claim like this the `Recreate`
strategy. A rolling update runs the old pod and the new pod at once,
both name the same claim, and Kubernetes gives a claim's device to
every pod that names the claim. `Recreate` stops the old pod first,
so one pod holds the device at a time.

### A workload that drives a display

A player or a kiosk sets the video mode, so it needs the card node as
well. Write a second `DeviceClass` for the display half:

    apiVersion: resource.k8s.io/v1
    kind: DeviceClass
    metadata:
      name: gpu-display
    spec:
      selectors:
        - cel:
            expression: |
              device.driver == "liken.sh" &&
              has(device.attributes["liken.sh"].displayNode)

Then put both requests in one claim. The container receives the nodes
of both devices:

    apiVersion: resource.k8s.io/v1
    kind: ResourceClaim
    metadata:
      name: player-gpu
      namespace: media
    spec:
      devices:
        requests:
          - name: render
            exactly:
              deviceClassName: gpu-render
          - name: display
            exactly:
              deviceClassName: gpu-display

The display device allocates to one claim, so a second player waits
until the first claim ends. This is the correct behavior: the kernel
gives modesetting to one process for each card, and a second player
that started would fail when it opened the card node.

Sound is a third device, and a different one: the audio controller
has its own PCI address, so a player that plays HDMI audio adds a
request for it. That device is shareable, so a second player does not
wait for it.

### Two requests, one card

On a machine with two GPUs, the two requests above can allocate halves
of different cards. A `constraints` block pairs them. Each constraint
names the requests it applies to and one attribute, and the scheduler
then allocates devices whose value for that attribute is the same.
The attribute to use is `address`, which is the device's address on
its bus:

    spec:
      devices:
        requests:
          - name: render
            exactly:
              deviceClassName: gpu-render
          - name: display
            exactly:
              deviceClassName: gpu-display
        constraints:
          - requests: [render, display]
            matchAttribute: liken.sh/address

Every device that liken publishes for one card carries that card's
address, and two cards in a machine have different addresses. The
render node and the card node the claim allocates are now halves of
the same GPU.

Leave the audio controller out of the list. It has an address of its
own, so a constraint that included its request would never match.

## 5. Give one pod a device alone

Most devices are not shareable. A USB adapter has one control
endpoint, so liken publishes it as a device that allocates one time.
Select the device by its identity:

    apiVersion: resource.k8s.io/v1
    kind: DeviceClass
    metadata:
      name: zigbee-adapter
    spec:
      selectors:
        - cel:
            expression: |
              device.driver == "liken.sh" &&
              device.attributes["liken.sh"].vendor == "10c4" &&
              device.attributes["liken.sh"].product == "ea60"

Add `device.attributes["liken.sh"].serial == "..."` when a machine has
two adapters of the same model and you must select one of them. The
name of the device follows the port, not the unit, so a replacement
adapter in the same port has the same name.

Then give each pod its own claim:

    apiVersion: resource.k8s.io/v1
    kind: ResourceClaimTemplate
    metadata:
      name: zigbee
      namespace: home
    spec:
      spec:
        devices:
          requests:
            - name: adapter
              exactly:
                deviceClassName: zigbee-adapter

    # in the pod template:
          resourceClaims:
            - name: adapter
              resourceClaimTemplateName: zigbee

A template makes one claim for each pod, and the device allocates to
one claim, so only one pod holds the adapter. A rolling update waits
until the old pod releases the adapter. This is correct for hardware
that one process must own.

Do not point two pods at one `ResourceClaim` for a device of this
kind. Both pods receive it. Kubernetes shares a claim with every pod
that names it, by design, and liken does not refuse the second pod. A
device node does not grant exclusive access by itself. The kernel is
what enforces exclusive access, through `O_EXCL` and the driver's own
open path. Use a template to give the device to one pod only.

## When a claim does not schedule

If a pod stays in `Pending` with a claim that is not allocated, then
usually no device matched the claim. Do these checks in this order:

1. `kubectl get resourceslices -o yaml` shows if the device is
   published. If the device is not there, no driver is bound to it:
   go back to step 2.
2. Compare your selector with the device's attributes. A selector that
   reads an attribute the device does not have errors on that device
   and aborts the allocation, so it never matches. Guard the reference
   with `has()` so an absent attribute means "does not match" instead.
3. If the device is published and it matches, another claim can
   already hold it. Find the holder:

       kubectl get resourceclaims -A -o wide

   A device that is not shareable allocates one time, and the second
   claim waits until the first claim releases it.


# Troubleshoot

A `liken` machine reports its problems in the Machine resource. Read
in this order: the phase, then the condition that is `False`, then
the status field that the condition's message names.

    kubectl get machines
    kubectl describe machine <name>

The phase gives the machine's state in one word. `Ready` means every
condition is true. `Blocked` means a change exists that the system
refuses to apply, and more time will not fix it. `Lost` means the
machine's heartbeat stopped. The sections below start from the
symptom you see.

## The machine stays at the stick menu

The menu has no timeout, so a machine at the menu waits for a
person. Select an entry.

If the machine returned to the menu after an installation, the stick
is still connected. The stick is first in the boot order. Power the
machine off, remove the stick, and power it on again. It then boots
from its own disk.

## The installation refuses a disk

`install as <name>` claims blank disks only, so it does not erase
data that it did not write. To replace an installation that `liken`
made, select `wipe and reinstall as <name>` instead.
[Install a cluster](/docs/guides/install/#to-reinstall-a-machine-that-liken-installed)
explains both paths, and the disk-replacement case between them.

## The installation fails and holds

A failed installation prints its error, lists the machine's disks,
and holds the console. Correct the cause and boot the installation
again. An installation is idempotent, so a second attempt is safe.

If the error says that a disk needs a driver the boot path does not
carry, run the hardware report. The report names the driver and
keeps that disk out of the proposed layout.
[Install a cluster](/docs/guides/install/#first-run-the-hardware-report)
describes the report.

## A machine does not join the cluster

Install the first leader first. A follower that boots before any
leader serves waits for one.

A declared network port with no cable delays each boot, because the
machine waits a maximum of thirty seconds for that port. Remove the
port from the manifest, or connect the cable.

If the machine still does not appear in `kubectl get machines`,
watch its console. Every step of the boot prints there, and a boot
that fails holds its reason on the screen.

## A machine shows Lost

The machine stopped sending its heartbeat, so the cluster operator
wrote the phase on its behalf. The machine is powered off, or it
cannot reach the cluster. When the machine returns, its next report
overwrites the phase. If the machine is running and `Lost`, check
its network path to the leaders.

## A machine shows Blocked

A change exists that the system refuses to stage. The condition that
is `False` names the reason:

* `StagingRejected`: the spec asks for something the machine cannot
  do. The common case is a storage role that is smaller in the spec
  than on the disk, after a reinstallation with a different layout.
  The condition's message gives the sizes.
  [Install a cluster](/docs/guides/install/#to-reinstall-a-machine-that-liken-installed)
  gives the order of the corrections.
* `RejectedLastBoot`: the machine tried the change in a boot, and
  the boot failed. See the next section.

## A machine returned to the old version

The machine tried the new version one time, the trial failed, and
the machine fell back to the slot it had proved. Its conditions show
`RejectedLastBoot`, and
[`status.boot.systemRejection`](/docs/reference/machine/#statusbootsystemrejection)
records what happened. The machine does not try that version again
until [`spec.version`](/docs/reference/cluster/#spec--version)
points at a different release. [Roll back](/docs/guides/rollback/) describes the
fallback and the correction.

## The fleet stays on the old version

`kubectl get machines` shows each machine's version in the LIKEN
column. For a machine that did not move, read its conditions:

* `RebootPending`: the machine's `rebootPolicy` is `Manual`, which is
  the default, and the machine waits for you. Read what it waits for
  and grant the reboot with
  [`liken approve-reboot`](/docs/reference/cli/#liken-approve-reboot).
* `AwaitingTurn`: the machine waits for the cluster's disruption
  budget. The default budget is one machine at a time, and only one
  leader is down at a time whatever the budget says.
* `Downloading`: the machine still fetches or verifies the release.
  A slow link makes this step long. The machine retries a failed
  download on its own.
* `AwaitingPodRefresh`: the machine runs the new release and waits
  for its operator pod to be recreated from the new template, which
  happens after a leader boots the release.

If the Cluster's `Progressing` condition is `False` with the reason
`RolloutStalled`, a machine with a granted turn did not return. The
cluster grants no more turns until you examine that machine, so the
rest of the fleet is safe while you do.

## A machine crashed

A kernel crash survives the reboot. The next boot reads the crash
from the machine's crash journal and reports it in the Machine's
[`status.lastCrash`](/docs/reference/machine/#statuslastcrash), with
the log lines that the kernel wrote as it failed:

    kubectl get machine <name> -o jsonpath='{.status.lastCrash}' | jq

## A machine needs a reboot and nothing is staged

A machine converges to its documents, so a machine that already
matches them stages nothing and reboots for nothing. Some faults
clear only at boot anyway. A kernel driver that bound the wrong
device holds it until the machine restarts, and no edit takes it
back.

[`liken request-reboot`](/docs/reference/cli/#liken-request-reboot)
asks for that boot:

    ./liken request-reboot mycluster <name>

The machine waits for its reboot turn, cordons, and drains, the same
as a machine applying a staged change. If its `rebootPolicy` is
`Manual`, it reports `RebootPending` and waits for
[`liken approve-reboot`](/docs/reference/cli/#liken-approve-reboot).
The `RebootRequestHonored` condition reports where the request is.

## A pod with a device claim stays Pending

[Give a workload a device](/docs/guides/devices/#when-a-claim-does-not-schedule)
gives the checks, in order.

## Reading logs

[`liken stern`](/docs/reference/cli/#liken-stern) tails the logs of
many pods at once, with the deployment's credential:

    ./liken stern mycluster <pod-name-pattern>

When the logs do not explain a problem, the Cluster's
[`spec.runtime`](/docs/reference/cluster/#specruntime) section raises
the log level of k3s or containerd, one field each, and says what
each level costs.


<!-- Generated from cluster/manifests/clusters-crd.yaml by docs/crdref. Do not edit. -->

A Cluster declares the fleet in one document: the release every
machine runs, the network, the shared runtime settings, and the
disruption budget for changes. [Upgrade the
fleet](/docs/guides/upgrade/) moves every machine with one edit to
`spec.version`.

A Cluster is the group that liken machines form together. It holds the topology, which is the set of machines that run control planes, and the network facts that every node must agree on.

## spec

The declared state of this cluster: the release the fleet runs, which machines lead it, the network and the time sources that every node agrees on, the limits a rollout keeps to, the optional features, the registries, and the k3s runtime settings. One document serves every machine, so each field here states a fact that the whole fleet shares. A fact about one machine belongs on that Machine instead.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="spec--version"></span>`version` | string | no | The liken release that the fleet moves toward. An upgrade edits this one field. A Machine spec holds no version of its own. Each machine's operator compares its running version (status.version.liken) with this target, and moves toward the target with the same staged change and granted reboot that every other change uses. This value must name an entry in releases.catalog. Pattern: `^\d{4}\.\d{2}\.\d{2}-\d{3}$`. |
| <span id="spec--releases"></span>`releases` | [object](#specreleases) | no | Where release artifacts come from, and which releases exist. The operator reads this section live on every pass, as it reads version. The cluster document that machines stage and reboot into does not include it. A published release must move machines through downloads and sequenced reboots. It must never stage a configuration reboot of the whole fleet by itself. The fleet observer polls the channel's root document at a slow interval. To request an immediate poll, set the Cluster's liken.sh/check-releases annotation to any new value, for example a timestamp. The value itself means nothing: the edit is the request. This is an annotation and not a spec field, because it asks for one action and does not declare a standing state. kubectl uses the same shape to request a Deployment rollout. |
| <span id="spec--origin"></span>`origin` | string | no | How the cluster's datastore came to exist. Founded, the default when the field is unset, means that liken created the datastore, through the founding leader's cluster-init. Adopted means that the datastore is already in a cluster that liken did not create. Machines built with that cluster's identity join it as members, and this includes the founding leader. No machine initializes a new datastore. A deployment declares adoption when the document is created. The only legal edit is the promotion from Adopted to Founded, made after the last foreign member is gone. Promotion changes nothing on a running fleet, because k3s ignores cluster-init when the datastore already exists. Promotion is important when the cluster is built again from nothing: the founder of a founded cluster can create the datastore again. Only a boot reads this field, so the edit costs no reboot and no restart. Each machine holds the new document and applies it at its next boot, and reports StagedForNextBoot on its ClusterConverged condition until then. One of: `Founded`, `Adopted`. |
| <span id="spec--leaders"></span>`leaders` | []string | no | The machines that run control planes, by Machine name. A machine gets its role from this list. A machine that is named here is a leader, and a machine that is absent is a follower. The first entry is the founding leader, and the other leaders join through its address. |
| <span id="spec--endpoint"></span>`endpoint` | string | no | The URL that followers join the cluster through, for example https://10.10.0.1:6443. If the cluster has one leader, this is that leader's address on the node network. Only a boot reads this field: a machine that has joined keeps the address of every leader and never asks the endpoint again. So an edit costs no reboot and no restart. Each machine holds the new document and applies it at its next boot, and reports StagedForNextBoot on its ClusterConverged condition until then. Two things still hold the old address until a machine boots. A follower keeps the endpoint's host in its time sources, as the last resort behind each leader's own address. And a machine that was down during the edit has not staged the new document, so it boots with the old endpoint. Keep the old address reachable until every machine reports Converged. Pattern: `^https://.+$`. |
| <span id="spec--network"></span>`network` | [object](#specnetwork) | no | The cluster's address plan: the facts that k3s requires every node to agree on, declared once here and not repeated on each machine. An unset field keeps k3s's own default. Every field here is immutable after it is set, except nodePortCIDRs. k3s cannot change these values on a live cluster, so an edit would change nothing, and the difference would appear only at the next reboot. The CEL transition rules (self == oldSelf) run only when the old value exists, which is what "immutable after it is set" means: a cluster that starts without a field can still get the field, and from then on the field holds. These rules compare with oldSelf. The Machine's grow-only storage rules compare with status instead, so that a declared size the machine cannot satisfy can be edited back down to the real size. These fields need no such escape, because their real values never change, so there is never a failed declaration to reverse. |
| <span id="spec--time"></span>`time` | [object](#spectime) | no | The cluster's time hierarchy: where the leaders get their time. Only the leaders read this section. The followers sync from the leaders themselves, so time moves from the upstreams to the leaders, and then to every other machine. |
| <span id="spec--disruption"></span>`disruption` | [object](#specdisruption) | no | How much of the fleet can be down at the same time when the cluster sequences reboots. This is the machine-level equivalent of a workload's PodDisruptionBudget, as one number. A staged change that needs a reboot waits for the cluster to grant the machine a turn, which is a RebootApproved condition written onto the Machine. This budget sets how many turns can be open at once. The count includes the machines that are down for any other reason, so a fleet that already has machines down pauses its own rollout. |
| <span id="spec--features"></span>`features` | map[string]map[string]string | no | The cluster's opt-ins from liken's feature vocabulary: optional capabilities that the fleet as a whole offers, beyond the minimum cluster. The map is keyed by feature slug. The presence of a key is the opt-in, and a feature that takes no parameters has the configuration {}. traefik is k3s's bundled ingress controller. servicelb is klipper-lb, which serves Services of type LoadBalancer on the nodes' own addresses. metrics-server is the aggregated API behind `kubectl top`. helm is the Helm controller inside the k3s server process, which turns HelmChart resources into workloads. A declared traefik enables helm automatically, because k3s deploys Traefik through such a resource. network-policy is the controller that turns NetworkPolicy resources into packet filtering. Without it, the API accepts those resources, but nothing enforces them, which is the behavior of the flannel CNI itself. iscsi is the host-side iSCSI initiator: static open-iscsi binaries, the iscsi_tcp kernel module, an initiator name made from each machine's name, and an iscsid DaemonSet. Every machine also publishes the stable name of each LUN it reaches under /dev/disk/by-path, in the form ip-address:port-iscsi-target-lun-number, which is the name a driver resolves a volume through. This is the host contract that CSI drivers such as synology-csi need. nfs is the NFSv4 client: a static mount.nfs and the nfsv4 kernel module, with no daemons, and version 4 only. flux is GitOps sync through Flux: the fleet's declared state is in a git repository, and the cluster applies what the repository holds. flux is the first feature in the vocabulary that takes parameters. Its repository parameter is required: the git URL the fleet syncs from, for example ssh://git@forge.example/fleet.git. Its path parameter (default ".") and its branch parameter (default "main") select the part of the repository that this cluster syncs. Its knownHosts parameter holds the forge's SSH host keys in known_hosts form, one line for each key. These keys are public material that lets the first clone verify the forge, so they go in the spec and not in a Secret. Its prune parameter (default "true") says whether the sync deletes an object that the repository no longer produces. Pruning is right for a cluster that liken founds, because every object in it came from the repository. A repository that carries its own copy of the flux-system Kustomization must declare prune: "false". That Kustomization appears in its own inventory, so the first build that stops producing it marks it for deletion, and the deletion removes everything the repository ever applied. The repository is expected to be private, and the cluster makes its own SSH deploy key. The private half never leaves the cluster, and status.flux.publicKey carries the half to register at the forge. If you remove flux from this object, the cluster retracts the whole feature: the sync stops, the engine and its namespace go, and the deploy key goes with them, so a new opt-in makes a new key to register. What the repository deployed stays in operation. Retraction stops the sync; it does not remove the workloads. The first three features (traefik, servicelb, and metrics-server) are components that the k3s binary already includes, and liken disables them by default: anything more than the control plane must be a declared, visible workload, and this field is that declaration. Adding a feature converges when k3s restarts in place, one leader at a time; the machine and its pods stay up. Removing network-policy, iscsi, or nfs reboots the machine instead, because each leaves kernel state that only a boot clears. Removing helm waits until no HelmChart remains, and removing servicelb waits until no LoadBalancer Service remains; until then the feature keeps running and the machine reports RetractionBlocked with the objects to remove. Removing traefik removes helm with it, in that order. |
| <span id="spec--registries"></span>`registries` | [object](#specregistries) | no | How container images arrive on the fleet's machines: the mirror endpoints that containerd pulls through, and k3s's embedded peer-to-peer registry (Spegel). These are cluster facts, because any node can be asked to pull any image. Credentials are deliberately not here. A spec is public, so credentials enter through the registry-credentials Secret in liken-system, which is the shape that `kubectl create secret docker-registry` produces. The machine operator reads that Secret and delivers it to each machine. As with the features above, the system reads the registries only when the k3s process starts, so an edit here converges when k3s restarts in place on each machine. The restart is sequenced through the same turns as a reboot, one leader at a time, but the machine, and its pods, stay up. |
| <span id="spec--runtime"></span>`runtime` | [object](#specruntime) | no | The discipline that the cluster applies to the k3s process, to the components inside it, and to containerd beside it. The section has one subsection for each thing that reads a setting: k3s for the process itself, kubelet for the configuration of the kubelet component that runs inside it, and containerd for the container runtime that k3s starts. The whole section is an opt-in. An unset field applies nothing, so the reader keeps its own default, and a cluster that names nothing here renders the same configuration it rendered before. Every value here is read only when the k3s process starts, so an edit converges when k3s restarts in place on each machine, sequenced through the same turns as a reboot, one leader at a time, with the machine and its pods up. containerd follows the same rule, because k3s starts containerd and stops it again. |

### spec.releases

Where release artifacts come from, and which releases exist. The operator reads this section live on every pass, as it reads version. The cluster document that machines stage and reboot into does not include it. A published release must move machines through downloads and sequenced reboots. It must never stage a configuration reboot of the whole fleet by itself. The fleet observer polls the channel's root document at a slow interval. To request an immediate poll, set the Cluster's liken.sh/check-releases annotation to any new value, for example a timestamp. The value itself means nothing: the edit is the request. This is an annotation and not a spec field, because it asks for one action and does not declare a standing state. kubectl uses the same shape to request a Deployment rollout.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specreleases--source"></span>`source` | string | no | The base URL that serves the releases. A release's artifacts are at `<source>/<version>/`. The first of them is release.yaml, the document that names every artifact by its digest. Pattern: `^https?://.+$`. |
| <span id="specreleases--catalog"></span>`catalog` | [\[\]object](#specreleasescatalog) | no | The releases that machines can be asked to run. Each entry's digest is the sha256 of the bytes of that release's release.yaml, and it is the start of the trust chain. This API names the document, the document names the artifacts, and the system compares every downloaded byte with one of the two. |

#### spec.releases.catalog[]

The releases that machines can be asked to run. Each entry's digest is the sha256 of the bytes of that release's release.yaml, and it is the start of the trust chain. This API names the document, the document names the artifacts, and the system compares every downloaded byte with one of the two.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specreleasescatalog--version"></span>`version` | string | yes | The release that this entry catalogs, as a calendar date and a serial. This is the name of the directory under releases.source that holds the release, and the value that spec.version names to move the fleet onto the release. Pattern: `^\d{4}\.\d{2}\.\d{2}-\d{3}$`. |
| <span id="specreleasescatalog--digest"></span>`digest` | string | yes | The sha256 of the bytes of this release's release.yaml, as sha256: and 64 hex digits. A machine compares the document it downloads with this digest. If the two do not agree, the machine stays on the release it runs and reports the phase Blocked with the reason DigestMismatch. Read the digest from the release channel when you add the entry. Pattern: `^sha256:[0-9a-f]{64}$`. |

### spec.network

The cluster's address plan: the facts that k3s requires every node to agree on, declared once here and not repeated on each machine. An unset field keeps k3s's own default. Every field here is immutable after it is set, except nodePortCIDRs. k3s cannot change these values on a live cluster, so an edit would change nothing, and the difference would appear only at the next reboot. The CEL transition rules (self == oldSelf) run only when the old value exists, which is what "immutable after it is set" means: a cluster that starts without a field can still get the field, and from then on the field holds. These rules compare with oldSelf. The Machine's grow-only storage rules compare with status instead, so that a declared size the machine cannot satisfy can be edited back down to the real size. These fields need no such escape, because their real values never change, so there is never a failed declaration to reverse.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specnetwork--nodecidr"></span>`nodeCIDR` | string | no | The subnet that the nodes use to address each other. On each machine, the interface with an address inside this subnet carries the cluster traffic, and that address becomes the machine's node IP. Pattern: `^[0-9a-fA-F:.]+/[0-9]{1,3}$`. |
| <span id="specnetwork--clustercidr"></span>`clusterCIDR` | string | no | The range that pod addresses come from. The k3s default is 10.42.0.0/16. Pattern: `^[0-9a-fA-F:.]+/[0-9]{1,3}$`. |
| <span id="specnetwork--servicecidr"></span>`serviceCIDR` | string | no | The range that service addresses come from. The k3s default is 10.43.0.0/16. Pattern: `^[0-9a-fA-F:.]+/[0-9]{1,3}$`. |
| <span id="specnetwork--clusterdns"></span>`clusterDNS` | string | no | The service address of the cluster's DNS resolver. The address is inside serviceCIDR. The k3s default is 10.43.0.10. Pattern: `^[0-9a-fA-F:.]+$`. |
| <span id="specnetwork--clusterdomain"></span>`clusterDomain` | string | no | The DNS suffix for the names inside the cluster. The k3s default is cluster.local. |
| <span id="specnetwork--nodeportcidrs"></span>`nodePortCIDRs` | []string | no | The networks that a NodePort service answers on. A NodePort is a port that the machines themselves open, and the machine addresses it answers on are a choice. If this list is unset, a NodePort answers on the node IP only: the address on nodeCIDR, where the rest of the cluster expects to find this machine. Set this list when the traffic arrives at a different address, which is the usual case for a cluster that is reached over a tunnel or a second segment. Each entry is a subnet, and a NodePort answers on every local address inside any of them. The list replaces the default and does not add to it, so the document states the whole answer: a list without nodeCIDR closes the NodePorts on the node network, also for the other machines in the cluster. The other fields here are immutable, because a change to one of them renumbers the cluster. This field is not immutable. It only increases or decreases the set of local addresses that answer, and the change reverses. k3s reads this list when its process starts, so an edit converges when k3s restarts in place on each machine, with no reboot. |

### spec.time

The cluster's time hierarchy: where the leaders get their time. Only the leaders read this section. The followers sync from the leaders themselves, so time moves from the upstreams to the leaders, and then to every other machine.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="spectime--upstreams"></span>`upstreams` | []string | no | The NTP servers that the leaders sync from, as hostnames or addresses. There is no default, and that is deliberate: the person who runs the deployment chooses whether to give the machines to a public pool. An empty list makes the fleet free-run. The machines stay consistent with each other, but they are correct only if the hardware clocks are also correct. |

### spec.disruption

How much of the fleet can be down at the same time when the cluster sequences reboots. This is the machine-level equivalent of a workload's PodDisruptionBudget, as one number. A staged change that needs a reboot waits for the cluster to grant the machine a turn, which is a RebootApproved condition written onto the Machine. This budget sets how many turns can be open at once. The count includes the machines that are down for any other reason, so a fleet that already has machines down pauses its own rollout.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specdisruption--maxunavailable"></span>`maxUnavailable` | integer | no | How many machines can be unavailable at the same time, planned and unplanned together. The default is one, the safest rollout. The leaders keep a stricter automatic floor that this number cannot raise: only one leader is down at a time. The datastore keeps quorum only while a majority of the leaders is up, and no policy setting changes that arithmetic. Default: `1`. |

### spec.registries

How container images arrive on the fleet's machines: the mirror endpoints that containerd pulls through, and k3s's embedded peer-to-peer registry (Spegel). These are cluster facts, because any node can be asked to pull any image. Credentials are deliberately not here. A spec is public, so credentials enter through the registry-credentials Secret in liken-system, which is the shape that `kubectl create secret docker-registry` produces. The machine operator reads that Secret and delivers it to each machine. As with the features above, the system reads the registries only when the k3s process starts, so an edit here converges when k3s restarts in place on each machine. The restart is sequenced through the same turns as a reboot, one leader at a time, but the machine, and its pods, stay up.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specregistries--mirrors"></span>`mirrors` | map[string][]string | no | A registry host, mapped to the endpoint URLs that containerd tries, in order of preference, before it falls back to the registry itself. The key is the host as an image reference names it (docker.io, registry.example:5000). |
| <span id="specregistries--embedded"></span>`embedded` | boolean | no | Turns on k3s's embedded registry mirror (Spegel). Each node serves the images it already holds to its peers, so the fleet pulls each image over the uplink one time, and not one time for each machine. This setting shares the images of every registry, through the wildcard mirror entry in registries.yaml. The mirrors declared above keep their own entries in both cases. |

### spec.runtime

The discipline that the cluster applies to the k3s process, to the components inside it, and to containerd beside it. The section has one subsection for each thing that reads a setting: k3s for the process itself, kubelet for the configuration of the kubelet component that runs inside it, and containerd for the container runtime that k3s starts. The whole section is an opt-in. An unset field applies nothing, so the reader keeps its own default, and a cluster that names nothing here renders the same configuration it rendered before. Every value here is read only when the k3s process starts, so an edit converges when k3s restarts in place on each machine, sequenced through the same turns as a reboot, one leader at a time, with the machine and its pods up. containerd follows the same rule, because k3s starts containerd and stops it again.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specruntime--k3s"></span>`k3s` | [object](#specruntimek3s) | no | The runtime discipline of the k3s process itself: the Go environment that init gives it, and how much it prints. An unset field applies nothing, so k3s keeps its own default for it. The two Go fields shape only the environment that init gives to the k3s process it starts, where k3s keeps Go's own defaults: no memory ceiling, and a heap that grows to twice its live data before the collector runs. Set a field to change that trade on a small machine, where k3s is the largest resident process. containerd and the shims that k3s starts inherit that environment, because k3s is their parent. No other process reads it: not init, not the operators, and not the workloads, which get their environment from their own pod specs. |
| <span id="specruntime--kubelet"></span>`kubelet` | [object](#specruntimekubelet) | no | The configuration of the kubelet. The kubelet is a component inside the k3s process, not a program of its own, and it runs on every machine of the cluster. An unset field applies nothing, so the kubelet keeps its own default for it. |
| <span id="specruntime--containerd"></span>`containerd` | [object](#specruntimecontainerd) | no | The configuration of containerd, the container runtime that runs beside the k3s process. containerd is a program of its own that k3s starts, not a component compiled into k3s, so it keeps its own configuration file and its own log level. Nothing in the k3s section reaches it. |

#### spec.runtime.k3s

The runtime discipline of the k3s process itself: the Go environment that init gives it, and how much it prints. An unset field applies nothing, so k3s keeps its own default for it. The two Go fields shape only the environment that init gives to the k3s process it starts, where k3s keeps Go's own defaults: no memory ceiling, and a heap that grows to twice its live data before the collector runs. Set a field to change that trade on a small machine, where k3s is the largest resident process. containerd and the shims that k3s starts inherit that environment, because k3s is their parent. No other process reads it: not init, not the operators, and not the workloads, which get their environment from their own pod specs.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specruntimek3s--gomemorylimit"></span>`goMemoryLimit` | string | no | The soft ceiling on everything that the k3s Go runtime manages: the heap, the stacks, and its own metadata (Go's GOMEMLIMIT). As the memory in use gets near the ceiling, the collector works harder instead of letting the heap grow. Above the ceiling, the runtime limits collection to half of the CPU and lets the heap grow, so a spike becomes slowness and not a crash. The field accepts three forms. "off" removes the ceiling. A percent such as "25%" is that share of each machine's memory, so one setting applies across a fleet of different sizes. An absolute quantity such as "448Mi" is the same ceiling on every machine. If the field is unset, k3s runs with no ceiling, the same as "off". The two wrong directions have different symptoms. With no ceiling, the kernel's OOM killer is the only protection, so the process dies under a spike. With a ceiling that is too tight, the collector runs against a limit it cannot clear and burns CPU on collection; the symptom is high user-time CPU on k3s with no matching workload. To find a good value, run k3s with no ceiling, read its resting size, then set the ceiling above that size with headroom for the node. Pattern: `^(off\|[0-9]{1,3}%\|[0-9]+(Ki\|Mi\|Gi\|Ti)?)$`. |
| <span id="specruntimek3s--gogc"></span>`goGC` | integer | no | The everyday pace of the collector, as a percent of heap growth between collections (Go's GOGC). If the field is unset, init sets no GOGC, so k3s keeps Go's own pace of one hundred percent: the heap grows to twice its live data before the collector runs. A value of 50 collects at fifty percent heap growth, and trades a little CPU all the time to keep the process resting near the size of its live data. A higher value collects less often and rests larger. A value below 1 is refused, because it would make the collector run without pause. |
| <span id="specruntimek3s--debug"></span>`debug` | boolean | no | Raises the k3s process to debug logging, the same thing that k3s's own --debug flag does. It reaches every Kubernetes component compiled into the process: the API server, the scheduler, the controllers, and the kubelet. It does not reach containerd's level, which the containerd section below sets. It does reach containerd's volume: k3s writes containerd's output to its own stream as well as to containerd's log file, and liken ships both files, so this field roughly doubles the containerd lines that leave the machine. If the field is unset, k3s logs at info. Turn it on to read a decision that the info lines do not explain, and turn it off again, because debug multiplies the volume of a stream that a small machine has to store and ship off itself. |

#### spec.runtime.kubelet

The configuration of the kubelet. The kubelet is a component inside the k3s process, not a program of its own, and it runs on every machine of the cluster. An unset field applies nothing, so the kubelet keeps its own default for it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specruntimekubelet--imagegc"></span>`imageGC` | [object](#specruntimekubeletimagegc) | no | The policy for the kubelet's image collector. containerd's image store grows with every image that a node pulls, and nothing leaves the store while a container uses it. The kubelet is the only thing that prunes the store, and it prunes on two triggers: how full the disk is, and how long an image has gone unused. Use this section when the clusterState filesystem grows more than you want, or when a node keeps images that no workload names any more. |

#### spec.runtime.kubelet.imageGC

The policy for the kubelet's image collector. containerd's image store grows with every image that a node pulls, and nothing leaves the store while a container uses it. The kubelet is the only thing that prunes the store, and it prunes on two triggers: how full the disk is, and how long an image has gone unused. Use this section when the clusterState filesystem grows more than you want, or when a node keeps images that no workload names any more.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specruntimekubeletimagegc--highthresholdpercent"></span>`highThresholdPercent` | integer | no | The disk usage that starts collection, as a percent of the filesystem that holds containerd's image store. On liken that filesystem is clusterState. The kubelet measures the filesystem every five minutes. When usage passes this threshold, the kubelet removes unused images, the least recently used first, until usage falls under lowThresholdPercent. If the field is unset, the threshold is 85. A percent rather than a byte count is what lets one setting serve machines with different disk sizes. |
| <span id="specruntimekubeletimagegc--lowthresholdpercent"></span>`lowThresholdPercent` | integer | no | The disk usage that stops collection, as a percent of the same filesystem. It must be below highThresholdPercent, so collection has a range to work in. If the field is unset, the threshold is 80. The distance between the two thresholds sets how much each round of collection removes: a narrow gap collects a little, often, and a wide gap collects a lot, rarely. |
| <span id="specruntimekubeletimagegc--maximumage"></span>`maximumAge` | string | no | How long an unused image may stay in the store before the kubelet removes it, whatever the disk usage is, as a Go duration such as "168h". This is the trigger that keeps a store from carrying years of tags that no workload names any more. If the field is unset, the kubelet does no age check at all, and only the disk thresholds prune the store. The value must be greater than minimumAge. Pattern: `^[0-9]+(\.[0-9]+)?(ns\|us\|ms\|s\|m\|h)([0-9]+(\.[0-9]+)?(ns\|us\|ms\|s\|m\|h))*$`. |
| <span id="specruntimekubeletimagegc--minimumage"></span>`minimumAge` | string | no | How long an unused image is kept before the kubelet may remove it, as a Go duration such as "5m". This is what stops a node from removing an image it just stopped using and pulling the same image again minutes later. If the field is unset, the age is two minutes. Raise it on a node that cycles through a few large images. Pattern: `^[0-9]+(\.[0-9]+)?(ns\|us\|ms\|s\|m\|h)([0-9]+(\.[0-9]+)?(ns\|us\|ms\|s\|m\|h))*$`. |

#### spec.runtime.containerd

The configuration of containerd, the container runtime that runs beside the k3s process. containerd is a program of its own that k3s starts, not a component compiled into k3s, so it keeps its own configuration file and its own log level. Nothing in the k3s section reaches it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specruntimecontainerd--loglevel"></span>`logLevel` | string | no | How much containerd prints. containerd is the loudest writer on a liken machine. At info it prints a line for each step of every pod's life, on every node, for as long as the node runs. If the field is unset, containerd keeps its own default of info. Set "warn" to keep the failures and drop the pod lifecycle lines, once the workloads on this cluster run without your attention. Set "debug" while you follow an image pull or a container start that the info lines do not explain. containerd takes three more levels that this field does not offer: trace above debug, which is a volume that no machine should write to the disk it also runs on, and fatal and panic below error, which drop the error lines written before a crash, the lines that explain it. One of: `debug`, `info`, `warn`, `error`. |

## status

What a reader can observe about the cluster as a whole. The leaders write it. Each leader's operator sweeps the Machine list on every pass and publishes what it found. Only the leaders can do this, because a follower that reaches the API is reaching a leader.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="status--phase"></span>`phase` | string | no | The conditions below, as one word. The system derives this value on every sweep, and never stores it as a truth of its own. Ready means that every machine is Ready. Updating means that the machines which are not Ready are all in a transition: rebooting into a change, waiting for one, or booting. This is the expected shape of a Cluster edit as it moves through the fleet. Degraded means that at least one machine is Lost, Blocked, or unhealthy in another way. This field can never truthfully show one state: lost quorum. The loss of a majority of the leaders takes the API server down with it, so no writer is left. When quorum is lost, the signal is that this status does not change any more. One of: `Ready`, `Updating`, `Degraded`. |
| <span id="status--observedgeneration"></span>`observedGeneration` | integer | no | The metadata.generation of the spec that this status judged. The sweep stamps it on every write. Each condition carries the same stamp, but a client that only asks whether the sweep has seen its edit reads the stamp here, at the top of status, where Kubernetes controllers conventionally publish it. |
| <span id="status--conditions"></span>`conditions` | [\[\]object](#statusconditions) | no | The sweep's observations about the fleet, one entry for each type. The sweep maintains two types. MachinesReady reports the headcount and names each machine that is not Ready. Progressing reports the reboot rollout, and is False when the rollout stops. The phase above is these conditions in one word. |
| <span id="status--machines"></span>`machines` | [object](#statusmachines) | no | How many machines are fully healthy, out of how many exist. A fully healthy machine is in phase Ready, with a heartbeat recent enough to count as current. The summary field holds the same two numbers in the form 4/5. The status stores the summary because a printer column can read one field, but cannot combine two. |
| <span id="status--flux"></span>`flux` | [object](#statusflux) | no | The observable half of the flux feature. publicKey is the fleet's deploy key: the public half, in the authorized_keys form that a forge accepts. The private half never appears where a person can read it. The cluster operator makes the pair in the flux-system Secret when the feature is declared, and publishes the public half here. Register it at the forge as a deploy key with write access, and the fleet syncs from then on. To rotate the key, delete the flux-system Secret in the flux-system namespace. The next sweep makes a new pair, and this field shows the new half to register. |
| <span id="status--releases"></span>`releases` | [object](#statusreleases) | no | What the sweep observes about the releases: the catalog it was given, and the channel it polls. |

### status.conditions[]

The sweep's observations about the fleet, one entry for each type. The sweep maintains two types. MachinesReady reports the headcount and names each machine that is not Ready. Progressing reports the reboot rollout, and is False when the rollout stops. The phase above is these conditions in one word.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusconditions--type"></span>`type` | string | yes | The name of the observation, and the key of this entry in the list. The sweep writes MachinesReady, the fleet headcount, and Progressing, the state of the reboot rollout. Pattern: `^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$`. |
| <span id="statusconditions--status"></span>`status` | string | yes | The verdict for this observation: True, False, or Unknown. A True MachinesReady means that every machine is Ready. A False Progressing means that the rollout no longer makes progress. Unknown means that the sweep cannot tell. One of: `True`, `False`, `Unknown`. |
| <span id="statusconditions--observedgeneration"></span>`observedGeneration` | integer | no | The metadata.generation of the Cluster spec that this condition judged. The generation counts spec edits, so this field lets a client tell a verdict on the current spec from a verdict on an earlier spec. That difference is important here, because an edit can wait for a reboot. |
| <span id="statusconditions--reason"></span>`reason` | string | yes | One CamelCase word for the cause of the current status, for a program to match on. MachinesReady uses AllMachinesReady, MachinesUpdating, and MachinesDegraded. Progressing uses RolloutComplete, RollingOut, and RolloutStalled. Pattern: `^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$`. |
| <span id="statusconditions--message"></span>`message` | string | no | The detail behind the reason, for a person to read. The MachinesReady message gives the tally and names each machine that is not Ready. The Progressing message names each machine that holds a reboot turn, and each machine that waits for one. |
| <span id="statusconditions--lasttransitiontime"></span>`lastTransitionTime` | string | yes | When the status field last changed value. The sweep writes this condition on every pass, but moves this time only when the status changes, so the value gives the age of the current verdict. |

### status.machines

How many machines are fully healthy, out of how many exist. A fully healthy machine is in phase Ready, with a heartbeat recent enough to count as current. The summary field holds the same two numbers in the form 4/5. The status stores the summary because a printer column can read one field, but cannot combine two.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusmachines--ready"></span>`ready` | integer | no | How many machines are fully healthy: in phase Ready, with a heartbeat recent enough to count as current. |
| <span id="statusmachines--total"></span>`total` | integer | no | How many Machine resources the sweep found. A machine counts here from the moment its Machine resource exists, whether or not the machine is up. |
| <span id="statusmachines--summary"></span>`summary` | string | no | The two counts above in the form 4/5, ready first. The MACHINES column of `kubectl get clusters` reads this field. |

### status.flux

The observable half of the flux feature. publicKey is the fleet's deploy key: the public half, in the authorized_keys form that a forge accepts. The private half never appears where a person can read it. The cluster operator makes the pair in the flux-system Secret when the feature is declared, and publishes the public half here. Register it at the forge as a deploy key with write access, and the fleet syncs from then on. To rotate the key, delete the flux-system Secret in the flux-system namespace. The next sweep makes a new pair, and this field shows the new half to register.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusflux--publickey"></span>`publicKey` | string | no | The public half of the fleet's deploy key, on one line, in the authorized_keys form that a forge accepts. Register this value at the forge as a deploy key with write access. An empty value means that the spec does not declare the flux feature, or that the cluster operator has not made the key pair yet. |

### status.releases

What the sweep observes about the releases: the catalog it was given, and the channel it polls.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusreleases--newest"></span>`newest` | string | no | The highest version in the catalog. The sweep derives it, so the NEWEST printer column can sit next to VERSION and nobody compares versions at the terminal. |
| <span id="statusreleases--available"></span>`available` | string | no | The latest version that the release channel itself announces: the channel.yaml document at spec.releases.source, which the fleet observer polls at a slow interval. The liken.sh/check-releases annotation forces the next poll. This value is advisory. It can name a version that the catalog does not hold yet, and to adopt that version you still commit a catalog entry with its digest. |


<!-- Generated from machine/manifests/machines-crd.yaml by docs/crdref. Do not edit. -->

A Machine declares one machine: its network interfaces, its disks,
and the kernel modules it loads. The machine writes what it
observes and what it does into `status`. [Install a
cluster](/docs/guides/install/) writes your first Machine
manifests, and the hardware report in that guide proposes one from
the hardware it finds. [Kernel settings](/docs/reference/sysctls/)
and [Resource limits](/docs/reference/rlimits/) list the values the
OS applies before `spec.sysctls` and `spec.rlimits` override them.

A Machine is one computer that runs liken. It holds the declared state of the operating system below this cluster, and the observed facts about the hardware that the operating system runs on.

## spec

The declared state of this machine. The fields take effect at different times. The liken operator applies sysctls and nodeLabels at boot, and it also reconciles them while the machine runs. The operator stages a change to network, storage, modules, or rlimits, and the change takes effect at the next boot.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="spec--network"></span>`network` | [object](#specnetwork) | no | The machine's network configuration. An empty field selects the default: DHCP on the first physical interface. Init applies this configuration at boot. The cluster cannot apply it while the machine runs, because the cluster reaches the machine through the addresses that an edit changes. The operator stages an edit, as it does a storage edit, and the next boot applies it. rebootPolicy sets who starts that boot, and status.boot.network reports the network that the running boot started with. |
| <span id="spec--sysctls"></span>`sysctls` | map[string]string | no | Kernel tuning values, from a parameter name to its declared value, for example vm.overcommit_memory: "1". Init applies these values at boot, and the liken operator applies them again on every pass, so an edit here takes effect without a reboot. liken sets its own values for about twenty parameters before it applies this field, so a name here also overrides liken's value for that parameter. The manual's Kernel settings page lists every one of them and why liken sets it. status.sysctls reports both sets together, read back from the kernel. |
| <span id="spec--rlimits"></span>`rlimits` | map[string]string | no | Resource limits, from a resource name to its declared limit, for example nofile: "1048576". These are the per-process ceilings that a systemd distribution sets in each service's Limit directives. liken has no systemd, so init applies them to itself before it starts k3s, and every process on the machine inherits them: k3s, containerd, and the containers below them. liken sets its own limits before it applies this field, so a name here overrides liken's limit for that resource. status.rlimits reports what the kernel holds, read back after both passes. Unlike sysctls, an edit here needs a reboot. The kernel fixes a process's limits when it forks, so nothing can raise the ceiling of a k3s that is already running. rebootPolicy says who starts that reboot. |
| <span id="spec--modules"></span>`modules` | []string | no | Extra kernel modules that this machine loads at boot, in addition to the fixed list that the operating system needs. These are the drivers for the hardware that this machine's workloads use. The image carries the kernel build's whole module tree, so any name that kernel has is loadable here, whatever manifests existed when the image was built. status.modules reports the result for each name, including a name this kernel has no module for. Adding a name takes effect without a reboot, because loading a driver is live-capable. Removing one needs a reboot. Admission checks the shape of the name only, because only the boot, or the load that follows a later edit, can determine whether the name exists. |
| <span id="spec--nodelabels"></span>`nodeLabels` | map[string]string | no | The labels that this machine's Node object carries. Workloads select on these labels, for example to find the machine that has the GPU, or the machine on battery-backed power. The system writes these labels into the k3s boot configuration, so the node has them when it registers. The liken operator then reconciles them while the machine runs. This includes the removal of a label that this spec no longer declares, which the kubelet does not do. The system never changes a label applied outside this spec. |
| <span id="spec--nodetaints"></span>`nodeTaints` | [\[\]object](#specnodetaints) | no | The taints that this machine's Node object carries. A taint repels every pod that does not tolerate it, which is how a machine that is dedicated to one job keeps other work off it. The system writes these taints into the k3s boot configuration, so a node that registers for the first time carries them from its first moment. The kubelet applies those registration taints only when it creates the Node object, on a first boot or after a reinstall. On every later boot the Node object exists already and the boot configuration changes nothing, so the liken operator is the only mechanism from then on: it reconciles this list while the machine runs. This includes the removal of a taint that this spec no longer declares. The system never changes a taint applied outside this spec. |
| <span id="spec--rebootpolicy"></span>`rebootPolicy` | string | no | What the operator can do when the spec needs a reboot to take effect. A change to storage, network, or modules is a change of this kind, because the operator measures drift in those three sections and the boot is what applies them. Manual, the default, stages the change to the machineState filesystem and reports RebootPending, and the next boot applies the change. Auto permits the operator to reboot the machine. This field also gates a reboot that a person asked for through the liken.sh/request-reboot annotation, which is the one reboot that applies no change: Manual reports RebootPending until the reboot is approved, and Auto takes the reboot turn on its own. Whichever policy is in force, the machine waits for the cluster to grant the turn, and drains before it goes down. One of: `Auto`, `Manual`. Default: `Manual`. |
| <span id="spec--storage"></span>`storage` | [object](#specstorage) | no | Assigns storage roles to disks. Each role is one field, because a machine has one cluster state and one pod-storage pool. Each role names the device it is on and, when it shares a disk, how much of the disk to take. One role per disk can omit its size and take the remainder. Init applies this configuration at boot, before k3s. The system reads the device path only on the boot that claims a blank disk. After that, the system finds the role's partition by the GPT partition name written on it. The operator stages an edit here, and the edit takes effect at the next boot. If the machine cannot satisfy a staged spec, the system quarantines that spec and the machine boots its last proven spec. Only a machine that has no proven spec stops the boot. The machine's RAM root backs each role that the spec does not declare. |

### spec.network

The machine's network configuration. An empty field selects the default: DHCP on the first physical interface. Init applies this configuration at boot. The cluster cannot apply it while the machine runs, because the cluster reaches the machine through the addresses that an edit changes. The operator stages an edit, as it does a storage edit, and the next boot applies it. rebootPolicy sets who starts that boot, and status.boot.network reports the network that the running boot started with.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specnetwork--interfaces"></span>`interfaces` | [\[\]object](#specnetworkinterfaces) | no | The machine's interfaces, each named as the kernel names it. A machine in a cluster usually declares two interfaces: a DHCP uplink, and the cluster-facing interface with the static address that its peers use to find it. |
| <span id="specnetwork--hostentries"></span>`hostEntries` | [\[\]object](#specnetworkhostentries) | no | Static entries for /etc/hosts, each one address and the names that resolve to it. Init writes each entry as a line below the three lines that define localhost and this machine's own name, so an entry can add a name but never override those two. Init also writes /etc/nsswitch.conf with hosts: files dns on every boot, so every resolver on the machine reads the hosts file before it queries DNS. This gives a program that resolves a name from the host's own files, such as the NFS mount helper, an answer that does not depend on cluster DNS being up. Init applies these entries at boot, and the liken operator applies them again on every pass, so an edit here takes effect without a reboot. status.hostEntries reports the entries as the hosts file holds them, read back after each pass. |

#### spec.network.interfaces[]

The machine's interfaces, each named as the kernel names it. A machine in a cluster usually declares two interfaces: a DHCP uplink, and the cluster-facing interface with the static address that its peers use to find it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specnetworkinterfaces--name"></span>`name` | string | yes | The interface to configure, named as the kernel names it (for example, eth1). |
| <span id="specnetworkinterfaces--address"></span>`address` | string | no | A static address in CIDR form, for example 10.10.0.1/24. The prefix length gives the kernel the subnet. An empty field selects DHCP on this interface. Pattern: `^[0-9a-fA-F:.]+/[0-9]{1,3}$`. |
| <span id="specnetworkinterfaces--gateway"></span>`gateway` | string | no | The gateway address that makes this interface the default route. This field is optional, also for a static address, because a DHCP lease on a different interface can supply the route. Pattern: `^[0-9a-fA-F:.]+$`. |
| <span id="specnetworkinterfaces--nameservers"></span>`nameservers` | []string | no | The nameservers for this interface, in preference order. The machine uses them in addition to the nameservers from a DHCP lease. The machine writes at most three nameservers to resolv.conf, in interface order. |

#### spec.network.hostEntries[]

Static entries for /etc/hosts, each one address and the names that resolve to it. Init writes each entry as a line below the three lines that define localhost and this machine's own name, so an entry can add a name but never override those two. Init also writes /etc/nsswitch.conf with hosts: files dns on every boot, so every resolver on the machine reads the hosts file before it queries DNS. This gives a program that resolves a name from the host's own files, such as the NFS mount helper, an answer that does not depend on cluster DNS being up. Init applies these entries at boot, and the liken operator applies them again on every pass, so an edit here takes effect without a reboot. status.hostEntries reports the entries as the hosts file holds them, read back after each pass.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specnetworkhostentries--address"></span>`address` | string | yes | The address this entry resolves to: an IPv4 or IPv6 literal, with no prefix length. This is the exact text that init writes into /etc/hosts. Pattern: `^[0-9a-fA-F:.]+$`. |
| <span id="specnetworkhostentries--names"></span>`names` | []string | yes | The names that resolve to this address, at most twenty, for example [nas, nas.home.arpa]. Each name is lowercase, because DNS compares names without case and one spelling keeps the file readable. |

### spec.nodeTaints[]

The taints that this machine's Node object carries. A taint repels every pod that does not tolerate it, which is how a machine that is dedicated to one job keeps other work off it. The system writes these taints into the k3s boot configuration, so a node that registers for the first time carries them from its first moment. The kubelet applies those registration taints only when it creates the Node object, on a first boot or after a reinstall. On every later boot the Node object exists already and the boot configuration changes nothing, so the liken operator is the only mechanism from then on: it reconciles this list while the machine runs. This includes the removal of a taint that this spec no longer declares. The system never changes a taint applied outside this spec.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specnodetaints--key"></span>`key` | string | yes | The taint's key: an optional DNS-style prefix, a slash, and a name that contains letters, digits, dashes, underscores, and dots. A pod tolerates this taint by naming the key and the effect. Pattern: `^([a-z0-9]([-a-z0-9]*[a-z0-9])?([.][a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?[A-Za-z0-9]([-A-Za-z0-9_.]*[A-Za-z0-9])?$`. |
| <span id="specnodetaints--value"></span>`value` | string | no | An optional value that goes with the key. A taint often needs none, because the key and the effect alone say to stay off this machine. A toleration that names a value must name the same value. Pattern: `^(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])?$`. |
| <span id="specnodetaints--effect"></span>`effect` | string | yes | What happens to a pod that does not tolerate this taint. NoSchedule keeps a new pod off the node and leaves the pods already running there alone. PreferNoSchedule asks the scheduler to avoid the node, but the scheduler still places a pod there when no other node fits. NoExecute keeps new pods off and also evicts every running pod without a matching toleration, at the moment the taint lands. That eviction includes DaemonSet pods, because the DaemonSet controller adds tolerations only for the node lifecycle keys. One of: `NoSchedule`, `PreferNoSchedule`, `NoExecute`. |

### spec.storage

Assigns storage roles to disks. Each role is one field, because a machine has one cluster state and one pod-storage pool. Each role names the device it is on and, when it shares a disk, how much of the disk to take. One role per disk can omit its size and take the remainder. Init applies this configuration at boot, before k3s. The system reads the device path only on the boot that claims a blank disk. After that, the system finds the role's partition by the GPT partition name written on it. The operator stages an edit here, and the edit takes effect at the next boot. If the machine cannot satisfy a staged spec, the system quarantines that spec and the machine boots its last proven spec. Only a machine that has no proven spec stops the boot. The machine's RAM root backs each role that the spec does not declare.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstorage--biosboot"></span>`biosBoot` | [object](#specstoragebiosboot) | no | A machine that declares this role, together with bootHome, boots in BIOS mode, through liken's own GRUB, and not through UEFI firmware. This is a small raw partition, about 1Mi in size, with no filesystem. It holds GRUB's core image, the code that the 440 boot bytes in the MBR jump into. The size is fixed after the system claims the partition, because the MBR refers to the partition by literal sector numbers. |
| <span id="specstorage--boothome"></span>`bootHome` | [object](#specstorageboothome) | no | The second part of the BIOS boot declaration, with biosBoot. This is a small FAT32 partition, about 64Mi in size, that holds GRUB's configuration and its environment block. The environment block is the file that holds the boot variables that UEFI firmware supplies on a UEFI machine. This partition is on neither system slot, so the system can write to both slots while the boot configuration stays in its own space. |
| <span id="specstorage--systema"></span>`systemA` | [object](#specstoragesystema) | no | The first of the operating system's two boot slots. Each slot holds one complete liken version. The machine runs from one slot while the system writes an upgrade to the other. The slots are EFI system partitions that carry FAT32, because the firmware reads each slot first. Their size is fixed after the system claims them, because FAT32 cannot grow in place. |
| <span id="specstorage--systemb"></span>`systemB` | [object](#specstoragesystemb) | no | The second boot slot. The systemA description gives the operation of the two slots. |
| <span id="specstorage--machinestate"></span>`machineState` | [object](#specstoragemachinestate) | no | The machine's own durable data, mainly the staged and proven manifests. These manifests keep a spec edit through a reboot, and the next boot applies the edit. This role is the first of the data roles in the standard partition order, so a later boot can find it before it reads any spec. |
| <span id="specstorage--machineephemeral"></span>`machineEphemeral` | [object](#specstoragemachineephemeral) | no | The operating system's own scratch space (/tmp). |
| <span id="specstorage--clusterstate"></span>`clusterState` | [object](#specstorageclusterstate) | no | k3s's state: its database, its TLS material, and containerd's images. The system keeps this data, so a reboot can continue the same cluster and does not start a new one. |
| <span id="specstorage--podstorage"></span>`podStorage` | [object](#specstoragepodstorage) | no | Durable storage that pods claim by name: the PersistentVolumeClaim pool. The local-path provisioner supplies volumes from this pool. |
| <span id="specstorage--podephemeral"></span>`podEphemeral` | [object](#specstoragepodephemeral) | no | The kubelet's working space: emptyDir volumes and per-pod scratch space. Pods meter this pool with ephemeral-storage requests and limits. The system also writes the pod logs here, and they stay visible at /var/log/pods. Make this role large enough for the logs and the volumes. A machine that does not declare this role keeps all of this data in memory. |

#### spec.storage.biosBoot

A machine that declares this role, together with bootHome, boots in BIOS mode, through liken's own GRUB, and not through UEFI firmware. This is a small raw partition, about 1Mi in size, with no filesystem. It holds GRUB's core image, the code that the 440 boot bytes in the MBR jump into. The size is fixed after the system claims the partition, because the MBR refers to the partition by literal sector numbers.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragebiosboot--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragebiosboot--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.bootHome

The second part of the BIOS boot declaration, with biosBoot. This is a small FAT32 partition, about 64Mi in size, that holds GRUB's configuration and its environment block. The environment block is the file that holds the boot variables that UEFI firmware supplies on a UEFI machine. This partition is on neither system slot, so the system can write to both slots while the boot configuration stays in its own space.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstorageboothome--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstorageboothome--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.systemA

The first of the operating system's two boot slots. Each slot holds one complete liken version. The machine runs from one slot while the system writes an upgrade to the other. The slots are EFI system partitions that carry FAT32, because the firmware reads each slot first. Their size is fixed after the system claims them, because FAT32 cannot grow in place.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragesystema--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragesystema--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.systemB

The second boot slot. The systemA description gives the operation of the two slots.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragesystemb--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragesystemb--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.machineState

The machine's own durable data, mainly the staged and proven manifests. These manifests keep a spec edit through a reboot, and the next boot applies the edit. This role is the first of the data roles in the standard partition order, so a later boot can find it before it reads any spec.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragemachinestate--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragemachinestate--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.machineEphemeral

The operating system's own scratch space (/tmp).

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragemachineephemeral--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragemachineephemeral--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.clusterState

k3s's state: its database, its TLS material, and containerd's images. The system keeps this data, so a reboot can continue the same cluster and does not start a new one.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstorageclusterstate--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstorageclusterstate--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.podStorage

Durable storage that pods claim by name: the PersistentVolumeClaim pool. The local-path provisioner supplies volumes from this pool.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragepodstorage--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragepodstorage--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

#### spec.storage.podEphemeral

The kubelet's working space: emptyDir volumes and per-pod scratch space. Pods meter this pool with ephemeral-storage requests and limits. The system also writes the pod logs here, and they stay visible at /var/log/pods. Make this role large enough for the logs and the volumes. A machine that does not declare this role keeps all of this data in memory.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="specstoragepodephemeral--device"></span>`device` | string | yes | The disk that this role is on: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. The system reads this field only on the boot that claims a blank disk. The by-id name belongs to the disk and survives a port move; the by-path name belongs to the port. After the claim, the system finds the role's partition by the GPT partition name written on it. Pattern: `^/dev/[a-zA-Z0-9/_.:=+#@-]+$`. |
| <span id="specstoragepodephemeral--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). This is an exact allocation, not a request. An empty value means that the role takes the rest of its disk, and only one role per disk can omit it. Pattern: `^[0-9]+(Ki\|Mi\|Gi\|Ti)?$`. |

## status

What the machine observed about itself. Init writes these facts through the facts tree, and the liken operator publishes them. The system derives every field here again on each reconcile pass and keeps no earlier value. Status reports the current state only.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="status--phase"></span>`phase` | string | no | The machine's state in one word. The system derives this value from the conditions on each reconcile pass, so the phase always agrees with them. Ready means that every condition holds. UpdatePending and Updating mean that a staged change waits for a reboot (Manual), waits for this machine's next boot, or is in progress (Auto). Blocked means that drift exists which liken refuses to stage or cannot stage. This state needs a different edit, and time does not clear it. Degraded covers all other faults. Booting means that init has not yet published this boot. Unknown means that the operator cannot read the machine's facts at all. Lost is the one value that the machine does not report about itself: a leader writes it when this machine's heartbeat lease (in liken-system) stops, because a machine that is down cannot report its own failure. One of: `Unknown`, `Booting`, `Lost`, `Blocked`, `Updating`, `UpdatePending`, `Degraded`, `Ready`. |
| <span id="status--observedgeneration"></span>`observedGeneration` | integer | no | The metadata.generation of the spec that this status judged. The operator writes this value on each pass. Each condition carries the same value, but a client that only needs to know whether the operator has seen its edit reads the value here, at the top of status, where Kubernetes controllers usually publish it. |
| <span id="status--role"></span>`role` | string | no | This machine's part in its cluster: a leader, which runs a control plane, or a follower, which runs workloads. The system derives this value at boot from the leaders list in the Cluster manifest. No one declares this field directly. |
| <span id="status--version"></span>`version` | [object](#statusversion) | no | The full inventory of what this machine runs: liken's own version and every third-party component that the operating system carries. The system reads the kernel and xtables versions from the running machine. The other values come from the components record that the image build wrote beside the image, the same pins that the release document publishes. k3s appears here and also on the Node object, because the two values are different: the Node reports the running kubelet, and this field reports what the operating system image carried. |
| <span id="status--network"></span>`network` | [object](#statusnetwork) | no | The network result of this boot, for DHCP leases and for static addresses. The top-level fields give the values of the primary interface, which is the cluster-facing interface when the Cluster's nodeCIDR identifies it. The interfaces field gives the full detail for each interface. |
| <span id="status--time"></span>`time` | [object](#statustime) | no | The state of this machine's clock. This field changes through the machine's whole life, unlike most of status, because the system corrects the clock continuously and not only at boot. |
| <span id="status--hardware"></span>`hardware` | [object](#statushardware) | no | What this machine is made of, as the machine observed itself. The processor count and the memory come from the boot. The disks and the unclaimed devices stay current while the machine runs, because init watches the kernel's uevents and writes both lists again on every change. |
| <span id="status--firmware"></span>`firmware` | [object](#statusfirmware) | no | The firmware state that the machine keeps: its boot mode and the boot menu from its non-volatile store. In UEFI mode, this field reports the firmware's variables as text. BIOS covers every environment that has no firmware variables to read, for example a legacy server or a direct-kernel boot under a hypervisor. This field describes the machine, not the boot: every value here stays through a power cycle, and the system writes status.boot again on every boot. Each entry shows the firmware's variable name and the entry's own description, for example "Boot0001 (liken slot A)". |
| <span id="status--storage"></span>`storage` | [object](#statusstorage) | no | What backs every storage role on this boot, whether the spec declares the role or not. The spec gives what the machine asked for, and hardware.blockDevices gives what is attached. This field connects the two. A memory-backed role reports no capacity on purpose, because all of these roles share the one RAM root, and a value for each role would count the same capacity more than once. |
| <span id="status--boot"></span>`boot` | [object](#statusboot) | no | The record of this boot. The system writes this record again on every boot, and the operator compares the spec against it. Compare this field with firmware, the state that stays through reboots. The manifest sources give the documents that the boot used: the staged copy that waits for its proving boot, the proven last-known-good copy, or, on the first boot only, the seed in the image. The system records these sources separately for the Machine manifest and the Cluster document, because the two stage and prove independently. The hashes identify the exact bytes, and the storage, network, and modules blocks show the spec as actuated. Together they are the reference for drift between what the cluster asks for and what the machine did. A rejection reports a staged document that failed and that the system quarantined. The rejection stands until a different edit proves out. |
| <span id="status--sysctls"></span>`sysctls` | map[string]string | no | The observed value of every parameter liken sets, read back from the kernel: the values liken sets on every machine, and the values spec.sysctls declares. This puts what was asked for and what the kernel holds side by side in one place. A parameter liken could not write is absent, because a failed write is never read back, so this field lists the parameters that currently hold rather than the parameters somebody wanted. |
| <span id="status--hostentries"></span>`hostEntries` | [\[\]object](#statushostentries) | no | The entries /etc/hosts actually holds, observed on the pass that published them: the live view of spec.network.hostEntries, the same way sysctls above is the live view of spec.sysctls. Init writes the file at boot, and the liken operator reconciles it live afterward, so this field changes within one reconcile pass of an edit, with no reboot. |
| <span id="status--rlimits"></span>`rlimits` | map[string]string | no | The resource limits init holds, read back from the kernel: the limits liken sets on every machine, and the limits spec.rlimits declares. Init is the first process, so these are the limits that every process on the machine inherits, including every container. Reading /proc/1/limits on the machine gives the same answer in the kernel's own layout. Each value is in the same form spec.rlimits uses, so the two can be compared without translating either. A limit liken could not set is absent, because a failed write is never read back. |
| <span id="status--modules"></span>`modules` | [\[\]object](#statusmodules) | no | The result for every module named in spec.modules, as the machine resolved it: at boot, or at the load that followed a later edit. Loaded and Builtin are good states. Missing means that this kernel has no module by that name, which is usually a misspelling, because the image carries the kernel's whole module tree. Rebuilding the image changes nothing; correct the name, or take a release whose kernel has the module. Failed means that the kernel refused a module it does have. The message field gives the detail and names the correction. |
| <span id="status--features"></span>`features` | [\[\]object](#statusfeatures) | no | The state on this machine of every feature that the Cluster's spec.features enables. The Cluster declares features for the whole fleet, but the state is per machine, because a feature needs support in the booted image, and machines run different releases during a rollout. Active means that this boot applied everything that the feature asks of this machine. Missing means that the booted image does not carry the feature, and the correction is a release whose image carries it. Failed means that the image carries the feature, but that the machine could not apply it. The message field gives the detail. |
| <span id="status--registries"></span>`registries` | [object](#statusregistries) | no | What this machine wrote into k3s's registries.yaml: the registries that it mirrors, the registries that have credentials, and whether the embedded registry is on. This field carries host names and flags only. The credential material never appears in status. |
| <span id="status--runtime"></span>`runtime` | [object](#statusruntime) | no | The runtime discipline that init imposed on this boot: the Go environment and log level it gave the k3s process, the configuration it wrote for the kubelet inside it, and the level it gave containerd beside it. These are the resolved values, not the strings from the Cluster's spec.runtime. An operator who changes a setting reads back here what each machine actually runs. |
| <span id="status--lastcrash"></span>`lastCrash` | [object](#statuslastcrash) | no | The most recent kernel crash that this machine still holds records for. When the kernel panics, it writes the end of its log to the firmware's storage, and the next boot moves that record onto the machine's own disk. Each boot derives this summary from the records again. The crash is not always the previous boot's crash. The summary stays until the records leave the machine's retention window, and the time field gives the age of the crash. An empty field means that the machine holds no crash records. |
| <span id="status--lastfailstop"></span>`lastFailStop` | [object](#statuslastfailstop) | no | The last boot that this machine refused to run. Init stops a boot for two failures: it cannot tell which configuration is its own, or it cannot satisfy a declared storage role. A machine that runs with the wrong configuration can do damage that a reboot does not undo, so init powers the machine off instead. The refusal writes its reason to the machine's own disk, and the next boot reports it here. The refusal is not always the previous boot, and the time field gives the age of the news. Nothing clears this field. The next refusal replaces it. An empty field means that this machine has never refused a boot, or that the role it could not satisfy was the one that holds the record. |
| <span id="status--conditions"></span>`conditions` | [\[\]object](#statusconditions) | no | The typed observations that the operators keep on this machine, in the standard Kubernetes form. Each pass writes every condition again. Ready is the roll-up, and it is True only when every other condition is True. The other conditions each report one check: FactsPublished for the facts that init writes, SysctlsApplied, StorageReady, ModulesLoaded, FeaturesReady, and NodeLabelsApplied for the parts of the spec that the operator applies, NodeHealthy for the Node's own Ready condition, NodeCurrent for the cleanup after a demotion, and SpecConverged, ClusterConverged, VersionConverged, CredentialsConverged, and ImportsConverged for the documents and the release that this boot ran under. RebootApproved is the one condition that another program writes: the cluster operator adds it to grant this machine a reboot turn, and removes it to take the grant back. Its absence carries the meaning, so it has no False state, and Ready does not count it. |
| <span id="status--pending"></span>`pending` | [\[\]object](#statuspending) | no | What this machine waits to apply: one entry for each staged document that needs a disruption. Each entry names the convergence condition that reported it, the kind of disruption that applies it, the staged document's hash, and a one-line summary. To grant one change its disruption on a machine whose rebootPolicy is Manual, annotate the machine with liken.sh/approve-disruption set to the entry's hash (the liken CLI's approve-reboot command does this). The grant is one-shot: once the change applies, its hash is no longer pending, and the next change hashes differently, so a stale annotation approves nothing. A reboot that a person requested through the liken.sh/request-reboot annotation is an entry here too. It waits on the same policy, the same turn, and the same drain, and the one thing it lacks is a document, so its hash is the running boot's identity instead. |

### status.version

The full inventory of what this machine runs: liken's own version and every third-party component that the operating system carries. The system reads the kernel and xtables versions from the running machine. The other values come from the components record that the image build wrote beside the image, the same pins that the release document publishes. k3s appears here and also on the Node object, because the two values are different: the Node reports the running kubelet, and this field reports what the operating system image carried.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusversion--liken"></span>`liken` | string | no | The liken version that this machine runs. The build stamps this value into the binary: a release build stamps the release name, and a development build stamps the git-described commit. The operator compares this value with the Cluster's spec.version to decide whether this machine needs an upgrade. |
| <span id="statusversion--kernel"></span>`kernel` | string | no | The kernel release, as uname reports it on the running machine. |
| <span id="statusversion--xtables"></span>`xtables` | string | no | The netfilter userspace, as iptables -V reports it: the version and the variant (legacy or nft). |
| <span id="statusversion--k3s"></span>`k3s` | string | no | The k3s release that this operating system image carries. liken vendors one pinned k3s binary for each release. |
| <span id="statusversion--trust"></span>`trust` | string | no | The date of the CA bundle: the day of the Mozilla root certificates that this machine trusts. |
| <span id="statusversion--e2fsprogs"></span>`e2fsprogs` | string | no | The e2fsprogs release that supplied mke2fs, the program that makes the ext4 filesystem on each storage role's partition. |
| <span id="statusversion--openiscsi"></span>`openIscsi` | string | no | The open-iscsi release that supplied iscsiadm and iscsid, the iSCSI initiator that the iscsi feature uses. |
| <span id="statusversion--nfsutils"></span>`nfsUtils` | string | no | The nfs-utils release that supplied mount.nfs, the mount helper that the nfs feature uses. |
| <span id="statusversion--systemdboot"></span>`systemdBoot` | string | no | The systemd-boot release that supplied the boot menu on liken's install media. |
| <span id="statusversion--grub"></span>`grub` | string | no | The GRUB release that supplied the boot program that liken writes into a BIOS machine's biosBoot partition. |
| <span id="statusversion--hwdata"></span>`hwdata` | string | no | The PCI database that gives names to vendor and device IDs. |
| <span id="statusversion--tzdata"></span>`tzdata` | string | no | The IANA timezone database release, for example 2026c. This is the database that resolves a CronJob's spec.timeZone on this machine. Read it when a job fires at an unexpected hour. A machine keeps the release its image carried until it upgrades. |
| <span id="statusversion--linuxfirmware"></span>`linuxFirmware` | string | no | The linux-firmware release that supplied the driver blobs under /lib/firmware. |
| <span id="statusversion--microcode"></span>`microcode` | string | no | The CPU microcode release that supplied the early cpio on this machine's boot slot. |
| <span id="statusversion--microcoderevision"></span>`microcodeRevision` | string | no | The microcode revision that the CPUs report. On physical hardware, this value shows whether the early cpio applied. On a virtual machine, this value comes from the hypervisor. |

### status.network

The network result of this boot, for DHCP leases and for static addresses. The top-level fields give the values of the primary interface, which is the cluster-facing interface when the Cluster's nodeCIDR identifies it. The interfaces field gives the full detail for each interface.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusnetwork--interface"></span>`interface` | string | no | The primary interface, as the kernel names it (for example, eth0). |
| <span id="statusnetwork--mac"></span>`mac` | string | no | The hardware address of the primary interface. |
| <span id="statusnetwork--addresses"></span>`addresses` | []string | no | The addresses of the primary interface, each in CIDR form (10.10.0.5/24). |
| <span id="statusnetwork--gateway"></span>`gateway` | string | no | The default gateway of the primary interface. This field is empty when the interface has no default route. |
| <span id="statusnetwork--nameservers"></span>`nameservers` | []string | no | The nameservers of the primary interface, in preference order: the nameservers from its DHCP lease, and the nameservers that the manifest declares for it. |
| <span id="statusnetwork--leaseexpires"></span>`leaseExpires` | string | no | The time when the DHCP lease of the primary interface expires. This field is empty when the interface has a static address. |
| <span id="statusnetwork--interfaces"></span>`interfaces` | [\[\]object](#statusnetworkinterfaces) | no | Every interface that this boot configured, with the detail for each one. The fields above repeat the values of the primary interface, which is the one that the Cluster's nodeCIDR identifies, or the first interface that came up. |

#### status.network.interfaces[]

Every interface that this boot configured, with the detail for each one. The fields above repeat the values of the primary interface, which is the one that the Cluster's nodeCIDR identifies, or the first interface that came up.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusnetworkinterfaces--name"></span>`name` | string | yes | This interface, as the kernel names it (for example, eth0). It is the key of this list. |
| <span id="statusnetworkinterfaces--mac"></span>`mac` | string | no | The hardware address of this interface. |
| <span id="statusnetworkinterfaces--address"></span>`address` | string | no | The address of this interface, in CIDR form (10.10.0.5/24). |
| <span id="statusnetworkinterfaces--method"></span>`method` | string | no | How the interface got its address: DHCP or Static. |
| <span id="statusnetworkinterfaces--gateway"></span>`gateway` | string | no | The default gateway of this interface. This field is empty when the interface has no default route. |
| <span id="statusnetworkinterfaces--nameservers"></span>`nameservers` | []string | no | The nameservers of this interface, in preference order: the nameservers from its DHCP lease, and the nameservers that the manifest declares for it. |
| <span id="statusnetworkinterfaces--leaseexpires"></span>`leaseExpires` | string | no | The time when the DHCP lease of this interface expires. This field is empty when the interface has a static address. |

### status.time

The state of this machine's clock. This field changes through the machine's whole life, unlike most of status, because the system corrects the clock continuously and not only at boot.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statustime--state"></span>`state` | string | no | Whether the system corrects the clock against a source that is itself synchronized. If it does not, this field shows whether the cause is the configuration or an outage. FreeRunning means that the spec declares no sources, and the machine runs on its hardware clock on purpose. Unsynchronized means that the spec declares sources but that none of them answers. A free-running fleet can agree with itself, but the state stays FreeRunning. Agreement in the fleet is not the same as agreement with real-world time, and certificate validation depends on that difference. One of: `Synchronized`, `FreeRunning`, `Unsynchronized`. |
| <span id="statustime--source"></span>`source` | string | no | The source that this machine synchronizes with. On a leader, this is the name of an upstream server. On a follower, this is one of the cluster's leaders. |
| <span id="statustime--stratum"></span>`stratum` | integer | no | The distance from a reference clock, in NTP's vocabulary: a source at stratum n makes this machine n+1. A server that free-runs on purpose reports 10, the local-clock value. A value of 16 means unsynchronized. |
| <span id="statustime--offset"></span>`offset` | string | no | The clock error measured at the last exchange, as a duration ("1.28ms"). The value is positive when this machine is behind its source. |
| <span id="statustime--lastsync"></span>`lastSync` | string | no | The last time that the clock agreed with its source. |

### status.hardware

What this machine is made of, as the machine observed itself. The processor count and the memory come from the boot. The disks and the unclaimed devices stay current while the machine runs, because init watches the kernel's uevents and writes both lists again on every change.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statushardware--cpus"></span>`cpus` | integer | no | The number of logical processors that the kernel makes available. Hardware that has threads counts each thread as one processor. |
| <span id="statushardware--memorybytes"></span>`memoryBytes` | integer | no | The machine's total usable memory, in bytes, as the kernel reports it. This value is less than the memory that the machine holds, because the firmware and the kernel reserve some of it. |
| <span id="statushardware--blockdevices"></span>`blockDevices` | [\[\]object](#statushardwareblockdevices) | no | The machine's storage inventory: every physical disk that the kernel found, whether the spec declares it or not. The name is the kernel's name for this boot. The kernel assigns the name in probe order, so the name identifies the device for this boot only and not permanently. The stable names identify the device across boots. The model and the serial come from the device itself. |
| <span id="statushardware--unclaimed"></span>`unclaimed` | [\[\]object](#statushardwareunclaimed) | no | Devices that the kernel found but that no driver controls, because spec.modules does not declare the module that the device needs. This list shows only those devices. It is not a full inventory, and a machine that has a driver for all of its hardware reports nothing here. Each entry gives the name of the device, the candidate modules whose alias patterns match the device's identifier, and the correction: declare a candidate module that the image carries, or upgrade to a release that carries one. |

#### status.hardware.blockDevices[]

The machine's storage inventory: every physical disk that the kernel found, whether the spec declares it or not. The name is the kernel's name for this boot. The kernel assigns the name in probe order, so the name identifies the device for this boot only and not permanently. The stable names identify the device across boots. The model and the serial come from the device itself.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statushardwareblockdevices--name"></span>`name` | string | yes | This disk, as the kernel names it this boot (vda, nvme0n1). It is the key of this list. |
| <span id="statushardwareblockdevices--sizebytes"></span>`sizeBytes` | integer | no | The capacity of this disk, in bytes. |
| <span id="statushardwareblockdevices--model"></span>`model` | string | no | The model name that the device reports. This field is empty when the device reports none. |
| <span id="statushardwareblockdevices--serial"></span>`serial` | string | no | The serial number that the device reports. This field is empty when the device reports none. |
| <span id="statushardwareblockdevices--stablenames"></span>`stableNames` | []string | no | Every name that identifies this disk across boots. spec.storage.<role>.device accepts any name in this list. The first by-id entry is the one to prefer. A by-id name follows the disk itself, so it survives a swap into a different bay. A by-path name follows the port instead, and stops naming the disk after that swap. |

#### status.hardware.unclaimed[]

Devices that the kernel found but that no driver controls, because spec.modules does not declare the module that the device needs. This list shows only those devices. It is not a full inventory, and a machine that has a driver for all of its hardware reports nothing here. Each entry gives the name of the device, the candidate modules whose alias patterns match the device's identifier, and the correction: declare a candidate module that the image carries, or upgrade to a release that carries one.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statushardwareunclaimed--modalias"></span>`modalias` | string | yes | The kernel's identifier for this device: the string that it announces in a uevent and matches driver alias patterns against. It is the key of this list, and it names the device exactly when the words below do not. |
| <span id="statushardwareunclaimed--bus"></span>`bus` | string | yes | The bus that the device is on: pci or usb. |
| <span id="statushardwareunclaimed--name"></span>`name` | string | no | The device in words: the manufacturer and product strings of a USB device, or the names from the pci.ids database for a PCI device. When no better name exists, this field gives the numeric vendor and device IDs. |
| <span id="statushardwareunclaimed--class"></span>`class` | string | no | The coarse kind of the device, decoded from the bus's class code: for example, mass-storage, display, or network. |
| <span id="statushardwareunclaimed--candidates"></span>`candidates` | []string | no | The loadable modules whose alias patterns match this device, in the kernel build's order of preference. More than one candidate is usual: USB storage matches both uas and usb_storage. The choice belongs to the person who edits spec.modules. |
| <span id="statushardwareunclaimed--message"></span>`message` | string | no | The correction, in words: declare one of the candidate modules in spec.modules when the image carries it, or upgrade to a release whose image carries one. |

### status.firmware

The firmware state that the machine keeps: its boot mode and the boot menu from its non-volatile store. In UEFI mode, this field reports the firmware's variables as text. BIOS covers every environment that has no firmware variables to read, for example a legacy server or a direct-kernel boot under a hypervisor. This field describes the machine, not the boot: every value here stays through a power cycle, and the system writes status.boot again on every boot. Each entry shows the firmware's variable name and the entry's own description, for example "Boot0001 (liken slot A)".

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusfirmware--mode"></span>`mode` | string | no | The kind of firmware that started this machine. UEFI means that the machine keeps boot variables that liken reads and writes. BIOS covers every environment that keeps no such variables, for example a legacy server or a direct-kernel boot under a hypervisor. One of: `UEFI`, `BIOS`. |
| <span id="statusfirmware--bootcurrent"></span>`bootCurrent` | string | no | The entry that this boot came from. In UEFI mode, the value is the firmware's BootCurrent variable. In BIOS mode, it is the liken.slot= parameter on the kernel command line. This field is empty when neither source names an entry, as in a direct-kernel boot. |
| <span id="statusfirmware--bootnext"></span>`bootNext` | string | no | The one-shot entry that is armed for the next boot: the firmware's BootNext variable in UEFI mode, or try_slot in GRUB's environment block in BIOS mode. The boot uses this entry one time and then clears it. A value here means that a proving boot waits and has not yet happened. This field is empty at every other time. |
| <span id="statusfirmware--bootorder"></span>`bootOrder` | []string | no | The standing list of preferences, with the first choice first: the firmware's BootOrder variable in UEFI mode, where each entry is a menu entry, or default_slot in GRUB's environment block in BIOS mode, which gives one entry. |

### status.storage

What backs every storage role on this boot, whether the spec declares the role or not. The spec gives what the machine asked for, and hardware.blockDevices gives what is attached. This field connects the two. A memory-backed role reports no capacity on purpose, because all of these roles share the one RAM root, and a value for each role would count the same capacity more than once.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstorage--biosboot"></span>`biosBoot` | [object](#statusstoragebiosboot) | no | What backs the biosBoot role now. spec.storage.biosBoot describes what the role holds. |
| <span id="statusstorage--boothome"></span>`bootHome` | [object](#statusstorageboothome) | no | What backs the bootHome role now. spec.storage.bootHome describes what the role holds. |
| <span id="statusstorage--systema"></span>`systemA` | [object](#statusstoragesystema) | no | What backs the systemA role now. spec.storage.systemA describes what the role holds. |
| <span id="statusstorage--systemb"></span>`systemB` | [object](#statusstoragesystemb) | no | What backs the systemB role now. spec.storage.systemB describes what the role holds. |
| <span id="statusstorage--machinestate"></span>`machineState` | [object](#statusstoragemachinestate) | no | What backs the machineState role now. spec.storage.machineState describes what the role holds. |
| <span id="statusstorage--machineephemeral"></span>`machineEphemeral` | [object](#statusstoragemachineephemeral) | no | What backs the machineEphemeral role now. spec.storage.machineEphemeral describes what the role holds. |
| <span id="statusstorage--clusterstate"></span>`clusterState` | [object](#statusstorageclusterstate) | no | What backs the clusterState role now. spec.storage.clusterState describes what the role holds. |
| <span id="statusstorage--podstorage"></span>`podStorage` | [object](#statusstoragepodstorage) | no | What backs the podStorage role now. spec.storage.podStorage describes what the role holds. |
| <span id="statusstorage--podephemeral"></span>`podEphemeral` | [object](#statusstoragepodephemeral) | no | What backs the podEphemeral role now. spec.storage.podEphemeral describes what the role holds. |

#### status.storage.biosBoot

What backs the biosBoot role now. spec.storage.biosBoot describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragebiosboot--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragebiosboot--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragebiosboot--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragebiosboot--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragebiosboot--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.bootHome

What backs the bootHome role now. spec.storage.bootHome describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstorageboothome--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstorageboothome--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstorageboothome--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstorageboothome--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstorageboothome--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.systemA

What backs the systemA role now. spec.storage.systemA describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragesystema--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragesystema--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragesystema--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragesystema--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragesystema--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.systemB

What backs the systemB role now. spec.storage.systemB describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragesystemb--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragesystemb--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragesystemb--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragesystemb--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragesystemb--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.machineState

What backs the machineState role now. spec.storage.machineState describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragemachinestate--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragemachinestate--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragemachinestate--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragemachinestate--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragemachinestate--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.machineEphemeral

What backs the machineEphemeral role now. spec.storage.machineEphemeral describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragemachineephemeral--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragemachineephemeral--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragemachineephemeral--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragemachineephemeral--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragemachineephemeral--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.clusterState

What backs the clusterState role now. spec.storage.clusterState describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstorageclusterstate--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstorageclusterstate--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstorageclusterstate--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstorageclusterstate--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstorageclusterstate--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.podStorage

What backs the podStorage role now. spec.storage.podStorage describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragepodstorage--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragepodstorage--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragepodstorage--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragepodstorage--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragepodstorage--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

#### status.storage.podEphemeral

What backs the podEphemeral role now. spec.storage.podEphemeral describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusstoragepodephemeral--backing"></span>`backing` | string | no | What holds this role's data. Partition means that the machine found the role's partition. Memory means that the role's directory stays on the machine's RAM root. A role that the spec does not declare has this backing. |
| <span id="statusstoragepodephemeral--device"></span>`device` | string | no | The device node of this role's partition, as the machine found it this boot (vda1). The kernel assigns this name in probe order, so the name identifies the partition for this boot only. This field is empty for a memory-backed role. |
| <span id="statusstoragepodephemeral--partition"></span>`partition` | string | no | The on-disk name of this role's partition. The name is the liken: prefix and the role's name. The claim writes this name, and every boot after that finds the partition by it, wherever the kernel enumerates the disk. This field is empty for a memory-backed role. |
| <span id="statusstoragepodephemeral--capacitybytes"></span>`capacityBytes` | integer | no | The size of this role's partition, in bytes. A memory-backed role reports no capacity, because all memory-backed roles share the one RAM root. |
| <span id="statusstoragepodephemeral--laststopunclean"></span>`lastStopUnclean` | boolean | no | Whether this boot found the role's filesystem still marked as mounted, which means the machine's previous stop did not release it. Only the FAT32 roles report this: the system slots and the boot home carry a mark for exactly this purpose, and the ext4 roles replay a journal instead. The field describes the stop before this boot, so it keeps its value until the machine boots again. |

### status.boot

The record of this boot. The system writes this record again on every boot, and the operator compares the spec against it. Compare this field with firmware, the state that stays through reboots. The manifest sources give the documents that the boot used: the staged copy that waits for its proving boot, the proven last-known-good copy, or, on the first boot only, the seed in the image. The system records these sources separately for the Machine manifest and the Cluster document, because the two stage and prove independently. The hashes identify the exact bytes, and the storage, network, and modules blocks show the spec as actuated. Together they are the reference for drift between what the cluster asks for and what the machine did. A rejection reports a staged document that failed and that the system quarantined. The rejection stands until a different edit proves out.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusboot--time"></span>`time` | string | no | The time when this machine booted. Init calculates it from the kernel's uptime counter. It belongs to the boot record because it shares the record's lifetime: a reboot changes it, and an in-place k3s restart does not. The Uptime printer column shows this timestamp as an elapsed time. |
| <span id="statusboot--manifestsource"></span>`manifestSource` | string | no | The copy of the Machine manifest that this boot used. Staged is a copy that waits for its proving boot. Proven is the last-known-good copy. Seed is the manifest in the image, and only a first boot uses it. One of: `Staged`, `Proven`, `Seed`. |
| <span id="statusboot--manifesthash"></span>`manifestHash` | string | no | The sha256 hash of the exact bytes of the Machine manifest that this boot used. The operator compares this hash with the hash of the manifest in the cluster to find drift. |
| <span id="statusboot--clustermanifestsource"></span>`clusterManifestSource` | string | no | The copy of the Cluster document that this boot used, with the same values as manifestSource. The system records the two documents separately, because each one stages and proves on its own. A machine can be current on one document and drifted on the other. One of: `Staged`, `Proven`, `Seed`. |
| <span id="statusboot--clustermanifesthash"></span>`clusterManifestHash` | string | no | The sha256 hash of the exact bytes of the Cluster document that this boot used. |
| <span id="statusboot--credentialssource"></span>`credentialsSource` | string | no | The copy of the registry-credentials document that this boot, or the most recent k3s restart, wrote into registries.yaml. This field is absent on a machine that never had credentials. That is a usual state, and the machine pulls images anonymously. One of: `Staged`, `Proven`. |
| <span id="statusboot--credentialshash"></span>`credentialsHash` | string | no | The sha256 hash of the exact bytes of the registry-credentials document that this boot, or the most recent k3s restart, wrote into registries.yaml. This field is absent on a machine that never had credentials. |
| <span id="statusboot--importssource"></span>`importsSource` | string | no | The imported-images record that this boot used. Staged means that the tarballs are on trial, and the operator has not yet proven that the container store serves them. Proven is the usual boot, where all of the tarballs match the record. This field is absent when the import lifecycle does not run (an ephemeral machineState or an ephemeral container store). One of: `Staged`, `Proven`. |
| <span id="statusboot--importshash"></span>`importsHash` | string | no | The sha256 hash of the exact bytes of the imported-images record that this boot used. This field is absent when the import lifecycle does not run. |
| <span id="statusboot--importsdiscarded"></span>`importsDiscarded` | boolean | no | True when this boot found an imports trial from an earlier boot that stopped before it proved the trial. This boot then discarded the container store instead of trusting it. The operating system images unpack again from this boot's tarballs, and workload images pull again. |
| <span id="statusboot--restarts"></span>`restarts` | integer | no | The number of in-place k3s restarts in this boot. A restart applies a restart-class change, for example a feature toggle or the registry configuration. This field shares the boot record's lifetime on purpose. A change that a restart applied increases this field and does not change the boot record's time. A change that a reboot applied changes the time and sets this field to zero. |
| <span id="statusboot--slot"></span>`slot` | string | no | The system slot that this boot came from, read from the liken.slot= parameter in the boot entry that started it. This field is absent when the boot did not come from a slot (a direct-kernel boot or install media). A release downloads to the slot that the machine does not run from, which is the purpose of the two slots. One of: `A`, `B`. |
| <span id="statusboot--commandline"></span>`commandLine` | string | no | The kernel command line that this boot ran with, whole and unparsed. The slot above is one parameter out of it. No field sets this line: liken writes each machine's boot entries itself, naming the console, the machine, and the slot. The line is reported because it is the only input a machine reads before it reads a file, so it is the first thing to check when one machine behaves unlike the others that share its release. |
| <span id="statusboot--storage"></span>`storage` | [object](#statusbootstorage) | no | The storage that the manifest of this boot declared. The system records it as actuated, whatever the result for each role was. A boot with the same manifest claims the same disks, so this field is the drift reference, and status.storage reports what backs each role now. An edit to spec.storage is converged when the two agree. |
| <span id="statusboot--network"></span>`network` | [object](#statusbootnetwork) | no | The network that the manifest of this boot declared. The system records it as actuated, whatever the result for each interface was. A boot with the same manifest asks for the same network, so this field is the drift reference, and status.network reports what each interface received. An edit to spec.network is converged when the two agree. This field is absent, and not empty, on a boot that reported no network at all. An empty field means that the machine declared no interface and used the zero-configuration default. |
| <span id="statusboot--modules"></span>`modules` | []string | no | The module list that the manifest of this boot declared. The system records it as actuated, whatever the result of each load was. A boot with the same image asks for the same modules, so this field is the drift reference, and status.modules gives the state of each module. |
| <span id="statusboot--rlimits"></span>`rlimits` | map[string]string | no | The resource limits that the manifest of this boot declared. The system records them as actuated, whatever the result of each one was. A boot under the same manifest asks for the same limits, so this field is the drift reference, and status.rlimits gives what the kernel actually holds. The limits liken sets on every machine are not part of this record, because they ship with the release rather than with the spec. |
| <span id="statusboot--rejection"></span>`rejection` | [object](#statusbootrejection) | no | The standing quarantine record for the Machine manifest: the staged manifest that the machine refused, and the reason. Every boot writes this record again until a promotion clears it, so the record stays through a power cycle. |
| <span id="statusboot--clusterrejection"></span>`clusterRejection` | [object](#statusbootclusterrejection) | no | The standing quarantine record for the Cluster document. It is the same record as rejection, for the other document. |
| <span id="statusboot--systemrejection"></span>`systemRejection` | [object](#statusbootsystemrejection) | no | The verdict on a system release whose proving boot did not complete: the machine booted the staged slot, did not prove it, and the firmware returned the machine to the proven slot. This verdict stands until the catalog names a different release. |
| <span id="statusboot--credentialsrejection"></span>`credentialsRejection` | [object](#statusbootcredentialsrejection) | no | The verdict on a staged registry-credentials document that the machine could not parse. This verdict stands until a different rendering, which an edit to the Secret makes, replaces it. |

#### status.boot.storage

The storage that the manifest of this boot declared. The system records it as actuated, whatever the result for each role was. A boot with the same manifest claims the same disks, so this field is the drift reference, and status.storage reports what backs each role now. An edit to spec.storage is converged when the two agree.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstorage--biosboot"></span>`biosBoot` | [object](#statusbootstoragebiosboot) | no | The biosBoot role, as this boot actuated it. spec.storage.biosBoot describes what the role holds. |
| <span id="statusbootstorage--boothome"></span>`bootHome` | [object](#statusbootstorageboothome) | no | The bootHome role, as this boot actuated it. spec.storage.bootHome describes what the role holds. |
| <span id="statusbootstorage--systema"></span>`systemA` | [object](#statusbootstoragesystema) | no | The systemA role, as this boot actuated it. spec.storage.systemA describes what the role holds. |
| <span id="statusbootstorage--systemb"></span>`systemB` | [object](#statusbootstoragesystemb) | no | The systemB role, as this boot actuated it. spec.storage.systemB describes what the role holds. |
| <span id="statusbootstorage--machinestate"></span>`machineState` | [object](#statusbootstoragemachinestate) | no | The machineState role, as this boot actuated it. spec.storage.machineState describes what the role holds. |
| <span id="statusbootstorage--machineephemeral"></span>`machineEphemeral` | [object](#statusbootstoragemachineephemeral) | no | The machineEphemeral role, as this boot actuated it. spec.storage.machineEphemeral describes what the role holds. |
| <span id="statusbootstorage--clusterstate"></span>`clusterState` | [object](#statusbootstorageclusterstate) | no | The clusterState role, as this boot actuated it. spec.storage.clusterState describes what the role holds. |
| <span id="statusbootstorage--podstorage"></span>`podStorage` | [object](#statusbootstoragepodstorage) | no | The podStorage role, as this boot actuated it. spec.storage.podStorage describes what the role holds. |
| <span id="statusbootstorage--podephemeral"></span>`podEphemeral` | [object](#statusbootstoragepodephemeral) | no | The podEphemeral role, as this boot actuated it. spec.storage.podEphemeral describes what the role holds. |

#### status.boot.storage.biosBoot

The biosBoot role, as this boot actuated it. spec.storage.biosBoot describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragebiosboot--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragebiosboot--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.bootHome

The bootHome role, as this boot actuated it. spec.storage.bootHome describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstorageboothome--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstorageboothome--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.systemA

The systemA role, as this boot actuated it. spec.storage.systemA describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragesystema--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragesystema--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.systemB

The systemB role, as this boot actuated it. spec.storage.systemB describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragesystemb--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragesystemb--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.machineState

The machineState role, as this boot actuated it. spec.storage.machineState describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragemachinestate--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragemachinestate--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.machineEphemeral

The machineEphemeral role, as this boot actuated it. spec.storage.machineEphemeral describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragemachineephemeral--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragemachineephemeral--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.clusterState

The clusterState role, as this boot actuated it. spec.storage.clusterState describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstorageclusterstate--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstorageclusterstate--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.podStorage

The podStorage role, as this boot actuated it. spec.storage.podStorage describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragepodstorage--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragepodstorage--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.storage.podEphemeral

The podEphemeral role, as this boot actuated it. spec.storage.podEphemeral describes what the role holds.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootstoragepodephemeral--device"></span>`device` | string | no | The disk that this role is on, exactly as the manifest of this boot declared it: a device path (/dev/vda), or a stable name under /dev/disk/by-id/ or /dev/disk/by-path/. |
| <span id="statusbootstoragepodephemeral--size"></span>`size` | string | no | How much of the disk this role takes, as a binary quantity (2Gi). An empty value means that the role takes the rest of its disk. |

#### status.boot.network

The network that the manifest of this boot declared. The system records it as actuated, whatever the result for each interface was. A boot with the same manifest asks for the same network, so this field is the drift reference, and status.network reports what each interface received. An edit to spec.network is converged when the two agree. This field is absent, and not empty, on a boot that reported no network at all. An empty field means that the machine declared no interface and used the zero-configuration default.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootnetwork--interfaces"></span>`interfaces` | [\[\]object](#statusbootnetworkinterfaces) | no | The interfaces that the manifest of this boot declared. The system records them as actuated, whatever the result for each interface was. status.network.interfaces reports what each interface received. |

#### status.boot.network.interfaces[]

The interfaces that the manifest of this boot declared. The system records them as actuated, whatever the result for each interface was. status.network.interfaces reports what each interface received.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootnetworkinterfaces--name"></span>`name` | string | no | The interface that the manifest named, as the kernel names it (for example, eth1). |
| <span id="statusbootnetworkinterfaces--address"></span>`address` | string | no | The static address in CIDR form that the manifest declared, for example 10.10.0.1/24. An empty field means DHCP on this interface. |
| <span id="statusbootnetworkinterfaces--gateway"></span>`gateway` | string | no | The gateway address that the manifest declared. An empty field means that the manifest gave this interface no default route. |
| <span id="statusbootnetworkinterfaces--nameservers"></span>`nameservers` | []string | no | The nameservers that the manifest declared for this interface, in preference order. The machine uses them in addition to the nameservers from a DHCP lease. |

#### status.boot.rejection

The standing quarantine record for the Machine manifest: the staged manifest that the machine refused, and the reason. Every boot writes this record again until a promotion clears it, so the record stays through a power cycle.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootrejection--hash"></span>`hash` | string | no | The sha256 hash of the exact bytes that the machine refused. The operator refuses to stage the same bytes again, so only a different edit clears the rejection. |
| <span id="statusbootrejection--reason"></span>`reason` | string | yes | Why the machine refused the document, in the same words that the boot prints on the console. |
| <span id="statusbootrejection--rejectedat"></span>`rejectedAt` | string | yes | The time when the machine refused the document. |

#### status.boot.clusterRejection

The standing quarantine record for the Cluster document. It is the same record as rejection, for the other document.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootclusterrejection--hash"></span>`hash` | string | no | The sha256 hash of the exact bytes that the machine refused. The operator refuses to stage the same bytes again, so only a different edit clears the rejection. |
| <span id="statusbootclusterrejection--reason"></span>`reason` | string | yes | Why the machine refused the document, in the same words that the boot prints on the console. |
| <span id="statusbootclusterrejection--rejectedat"></span>`rejectedAt` | string | yes | The time when the machine refused the document. |

#### status.boot.systemRejection

The verdict on a system release whose proving boot did not complete: the machine booted the staged slot, did not prove it, and the firmware returned the machine to the proven slot. This verdict stands until the catalog names a different release.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootsystemrejection--hash"></span>`hash` | string | no | The sha256 hash of the exact bytes of the staged release record that the machine refused. No later boot arms the same trial again. |
| <span id="statusbootsystemrejection--reason"></span>`reason` | string | yes | Why the machine refused the release, in the same words that the boot prints on the console. |
| <span id="statusbootsystemrejection--rejectedat"></span>`rejectedAt` | string | yes | The time when the machine refused the release. |

#### status.boot.credentialsRejection

The verdict on a staged registry-credentials document that the machine could not parse. This verdict stands until a different rendering, which an edit to the Secret makes, replaces it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusbootcredentialsrejection--hash"></span>`hash` | string | no | The sha256 hash of the exact bytes of the credentials document that the machine refused. Only a different rendering, which an edit to the Secret makes, clears the rejection. |
| <span id="statusbootcredentialsrejection--reason"></span>`reason` | string | yes | Why the machine refused the document, in the same words that the boot prints on the console. |
| <span id="statusbootcredentialsrejection--rejectedat"></span>`rejectedAt` | string | yes | The time when the machine refused the document. |

### status.hostEntries[]

The entries /etc/hosts actually holds, observed on the pass that published them: the live view of spec.network.hostEntries, the same way sysctls above is the live view of spec.sysctls. Init writes the file at boot, and the liken operator reconciles it live afterward, so this field changes within one reconcile pass of an edit, with no reboot.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statushostentries--address"></span>`address` | string | yes | The address, as the hosts file holds it. |
| <span id="statushostentries--names"></span>`names` | []string | yes | The names that resolve to this address, as the hosts file holds them. |

### status.modules[]

The result for every module named in spec.modules, as the machine resolved it: at boot, or at the load that followed a later edit. Loaded and Builtin are good states. Missing means that this kernel has no module by that name, which is usually a misspelling, because the image carries the kernel's whole module tree. Rebuilding the image changes nothing; correct the name, or take a release whose kernel has the module. Failed means that the kernel refused a module it does have. The message field gives the detail and names the correction.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusmodules--name"></span>`name` | string | yes | The module, by the name that spec.modules declares (for example, e1000e). It is the key of this list. |
| <span id="statusmodules--state"></span>`state` | string | yes | The result for this module on this boot. Loaded means that the kernel took the module, or already held it. Builtin means that the kernel has the module compiled in, so there was nothing to load. Missing means that the booted image does not carry the module, and a new image corrects it, not a retry. Failed means that the image carries the module and the kernel refused it, which usually points to a fault in the hardware. One of: `Loaded`, `Builtin`, `Missing`, `Failed`. |
| <span id="statusmodules--message"></span>`message` | string | no | The detail for a Missing or a Failed module, in the same words that the boot prints on the console, and named as a correction. This field is empty when the state is Loaded or Builtin. |

### status.features[]

The state on this machine of every feature that the Cluster's spec.features enables. The Cluster declares features for the whole fleet, but the state is per machine, because a feature needs support in the booted image, and machines run different releases during a rollout. Active means that this boot applied everything that the feature asks of this machine. Missing means that the booted image does not carry the feature, and the correction is a release whose image carries it. Failed means that the image carries the feature, but that the machine could not apply it. The message field gives the detail.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusfeatures--name"></span>`name` | string | yes | The feature, by the name that the Cluster's spec.features enables (for example, iscsi). It is the key of this list. |
| <span id="statusfeatures--state"></span>`state` | string | yes | The state of this feature on this machine, on this boot. Active means that the boot applied everything that the feature asks of this machine. Missing means that the booted image carries no payload for the feature, and the correction is a release whose image carries one. Failed means that the image carries the payload and the machine could not apply it, for example a module that the kernel refused, or a boot hook that returned an error. One of: `Active`, `Missing`, `Failed`. |
| <span id="statusfeatures--message"></span>`message` | string | no | The detail for a Missing or a Failed feature, in the same words that the boot prints on the console, and named as a correction. This field is empty when the state is Active. |

### status.registries

What this machine wrote into k3s's registries.yaml: the registries that it mirrors, the registries that have credentials, and whether the embedded registry is on. This field carries host names and flags only. The credential material never appears in status.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusregistries--mirrors"></span>`mirrors` | []string | no | The registry hosts that registries.yaml carries a mirror entry for, sorted by name. The list holds "*" as well when the embedded registry is on, because the wildcard entry is how it shares the images of every registry. This field is empty when the machine mirrors nothing. |
| <span id="statusregistries--credentialedhosts"></span>`credentialedHosts` | []string | no | The registry hosts that registries.yaml carries a credential for, sorted by name. This field names the hosts only. The usernames and the passwords stay in the registry-credentials Secret and never reach status. This field is empty when the machine pulls every image anonymously. |
| <span id="statusregistries--embedded"></span>`embedded` | boolean | no | True when this boot turned k3s's embedded registry mirror (Spegel) on, which the Cluster's spec.registries.embedded asks for. |

### status.runtime

The runtime discipline that init imposed on this boot: the Go environment and log level it gave the k3s process, the configuration it wrote for the kubelet inside it, and the level it gave containerd beside it. These are the resolved values, not the strings from the Cluster's spec.runtime. An operator who changes a setting reads back here what each machine actually runs.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusruntime--k3s"></span>`k3s` | [object](#statusruntimek3s) | no | The runtime discipline of the k3s process on this machine. containerd and the shims get the same Go environment. This field is absent when init set no variable and rendered no log level, so k3s runs on Go's own defaults and logs at info. |
| <span id="statusruntime--kubelet"></span>`kubelet` | [object](#statusruntimekubelet) | no | The kubelet configuration that init wrote on this machine, from the Cluster's spec.runtime.kubelet. This field is absent when the cluster names no setting, so init wrote no configuration file and the kubelet runs on its own defaults. |
| <span id="statusruntime--containerd"></span>`containerd` | [object](#statusruntimecontainerd) | no | The containerd configuration that init wrote on this machine, from the Cluster's spec.runtime.containerd. This field is absent when the cluster names no setting, so init wrote no drop-in and containerd runs on its own defaults. |

#### status.runtime.k3s

The runtime discipline of the k3s process on this machine. containerd and the shims get the same Go environment. This field is absent when init set no variable and rendered no log level, so k3s runs on Go's own defaults and logs at info.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusruntimek3s--gomemorylimit"></span>`goMemoryLimit` | string | no | The resolved memory ceiling, as an absolute quantity in MiB, for example "256Mi". It is empty when the cluster does not set the ceiling or turns it off, so an absent value means no ceiling and Go's own default. |
| <span id="statusruntimek3s--gogc"></span>`goGC` | integer | no | The resolved collector rate (Go's GOGC). It is absent when the cluster sets no value, so k3s keeps Go's own rate. |
| <span id="statusruntimek3s--debug"></span>`debug` | boolean | no | True when init told k3s to log at debug, which reaches every Kubernetes component inside the process. It is absent when the cluster sets no value, so k3s logs at info. |

#### status.runtime.kubelet

The kubelet configuration that init wrote on this machine, from the Cluster's spec.runtime.kubelet. This field is absent when the cluster names no setting, so init wrote no configuration file and the kubelet runs on its own defaults.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusruntimekubelet--imagegc"></span>`imageGC` | [object](#statusruntimekubeletimagegc) | no | The image collection policy that this machine's kubelet runs. Each field is present only when the cluster names it, so an absent field means the kubelet's own default for that field. |

#### status.runtime.kubelet.imageGC

The image collection policy that this machine's kubelet runs. Each field is present only when the cluster names it, so an absent field means the kubelet's own default for that field.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusruntimekubeletimagegc--highthresholdpercent"></span>`highThresholdPercent` | integer | no | The disk usage that starts collection, as a percent of the filesystem that holds containerd's image store. It is absent when the cluster names no value, so the kubelet uses 85. |
| <span id="statusruntimekubeletimagegc--lowthresholdpercent"></span>`lowThresholdPercent` | integer | no | The disk usage that stops collection, as a percent of the same filesystem. It is absent when the cluster names no value, so the kubelet uses 80. |
| <span id="statusruntimekubeletimagegc--maximumage"></span>`maximumAge` | string | no | How long an unused image may stay in the store before the kubelet removes it, whatever the disk usage is. It is absent when the cluster names no value, so the kubelet does no age check and only the thresholds prune the store. |
| <span id="statusruntimekubeletimagegc--minimumage"></span>`minimumAge` | string | no | How long an unused image is kept before the kubelet may remove it. It is absent when the cluster names no value, so the kubelet uses two minutes. |

#### status.runtime.containerd

The containerd configuration that init wrote on this machine, from the Cluster's spec.runtime.containerd. This field is absent when the cluster names no setting, so init wrote no drop-in and containerd runs on its own defaults.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusruntimecontainerd--loglevel"></span>`logLevel` | string | no | The log level that this machine's containerd runs at. It is absent when the cluster names no level, so containerd uses info. |

### status.lastCrash

The most recent kernel crash that this machine still holds records for. When the kernel panics, it writes the end of its log to the firmware's storage, and the next boot moves that record onto the machine's own disk. Each boot derives this summary from the records again. The crash is not always the previous boot's crash. The summary stays until the records leave the machine's retention window, and the time field gives the age of the crash. An empty field means that the machine holds no crash records.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statuslastcrash--time"></span>`time` | string | no | The machine's own clock at the time of the crash. A crash usually occurs before the boot's first clock synchronization, so this time comes from the hardware clock. |
| <span id="statuslastcrash--reason"></span>`reason` | string | no | The kernel's word for the crash. The shipped kernel records two words: Panic and Oops. The vocabulary belongs to the kernel, so this field is a free string, not a fixed list. |
| <span id="statuslastcrash--message"></span>`message` | string | no | The kernel's first description of the failure, in the kernel's own words: the "Kernel panic - not syncing:" line, or the "BUG:" line of an oops. |
| <span id="statuslastcrash--records"></span>`records` | string | no | The directory on the machine that holds the full end of the kernel log: a directory in the crash store under machineState, or /sys/fs/pstore on a machine whose machineState runs in memory. |

### status.lastFailStop

The last boot that this machine refused to run. Init stops a boot for two failures: it cannot tell which configuration is its own, or it cannot satisfy a declared storage role. A machine that runs with the wrong configuration can do damage that a reboot does not undo, so init powers the machine off instead. The refusal writes its reason to the machine's own disk, and the next boot reports it here. The refusal is not always the previous boot, and the time field gives the age of the news. Nothing clears this field. The next refusal replaces it. An empty field means that this machine has never refused a boot, or that the role it could not satisfy was the one that holds the record.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statuslastfailstop--time"></span>`time` | string | no | The machine's own clock at the refusal. A boot stops before its first clock synchronization, so this time comes from the hardware clock. |
| <span id="statuslastfailstop--reason"></span>`reason` | string | no | Init's own words for the refusal, the same text that the console printed before the machine powered off. |

### status.conditions[]

The typed observations that the operators keep on this machine, in the standard Kubernetes form. Each pass writes every condition again. Ready is the roll-up, and it is True only when every other condition is True. The other conditions each report one check: FactsPublished for the facts that init writes, SysctlsApplied, StorageReady, ModulesLoaded, FeaturesReady, and NodeLabelsApplied for the parts of the spec that the operator applies, NodeHealthy for the Node's own Ready condition, NodeCurrent for the cleanup after a demotion, and SpecConverged, ClusterConverged, VersionConverged, CredentialsConverged, and ImportsConverged for the documents and the release that this boot ran under. RebootApproved is the one condition that another program writes: the cluster operator adds it to grant this machine a reboot turn, and removes it to take the grant back. Its absence carries the meaning, so it has no False state, and Ready does not count it.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statusconditions--type"></span>`type` | string | yes | The check that this entry reports, in CamelCase. It is the key of this list, so a machine carries one entry for each type. The description of the conditions field lists the types that liken publishes. Pattern: `^([a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*/)?(([A-Za-z0-9][-A-Za-z0-9_.]*)?[A-Za-z0-9])$`. |
| <span id="statusconditions--status"></span>`status` | string | yes | The verdict. Every condition type on a Machine states a healthy fact, so True is the good verdict for all of them. Unknown means that the operator cannot tell yet, for example when the machine's facts carry no boot record. One of: `True`, `False`, `Unknown`. |
| <span id="statusconditions--observedgeneration"></span>`observedGeneration` | integer | no | The metadata.generation that this condition judged. The generation counts spec edits, so this field lets a client tell a verdict on the current spec from a verdict on an earlier spec. That difference is important here, because an edit can wait for a reboot. |
| <span id="statusconditions--reason"></span>`reason` | string | yes | Why the condition holds this status, in one CamelCase word. Programs read this field, so it changes less than the message does. Ready reports Reconciled when it is True. When Ready is False, its reason is the phase word that the conditions produce, so the roll-up and the phase agree: a machine that waits on a reboot turn reads UpdatePending, and Degraded appears exactly when the phase says Degraded. NothingDeclared is the reason that a check with nothing to do reports. The convergence conditions share one set of reasons: Converged, FactsIncomplete, MachineStateEphemeral, StagingRejected, RejectedLastBoot, AwaitingTurn, RebootPending, RebootRequested, RestartPending, RestartRequested, and StagedForNextBoot. The last one means that the machine holds the change and applies it at its next boot. Only a boot reads the fields that changed, so no reboot and no restart is scheduled. Pattern: `^[A-Za-z]([A-Za-z0-9_,:]*[A-Za-z0-9_])?$`. |
| <span id="statusconditions--message"></span>`message` | string | no | The detail, for a person to read. A message names the correction where one exists, for example the roles that memory still backs, or the exact difference between the spec and this boot. |
| <span id="statusconditions--lasttransitiontime"></span>`lastTransitionTime` | string | yes | The time when the status of this condition last changed. A pass that writes the same status again does not move this time, so the field gives the length of time that this verdict has held. |

### status.pending[]

What this machine waits to apply: one entry for each staged document that needs a disruption. Each entry names the convergence condition that reported it, the kind of disruption that applies it, the staged document's hash, and a one-line summary. To grant one change its disruption on a machine whose rebootPolicy is Manual, annotate the machine with liken.sh/approve-disruption set to the entry's hash (the liken CLI's approve-reboot command does this). The grant is one-shot: once the change applies, its hash is no longer pending, and the next change hashes differently, so a stale annotation approves nothing. A reboot that a person requested through the liken.sh/request-reboot annotation is an entry here too. It waits on the same policy, the same turn, and the same drain, and the one thing it lacks is a document, so its hash is the running boot's identity instead.

| Field | Type | Required | Description |
| --- | --- | --- | --- |
| <span id="statuspending--condition"></span>`condition` | string | yes | The condition that reported this entry, for example CredentialsConverged, or RebootRequestHonored for a reboot that a person requested. |
| <span id="statuspending--kind"></span>`kind` | string | yes | What applies the staged document. A reboot applies every staged document at once. A restart applies only the documents that k3s reads when its process starts, and the machine does not reboot. One of: `Reboot`, `Restart`. |
| <span id="statuspending--hash"></span>`hash` | string | yes | The staged document's identity: the value that an approve-disruption annotation names to grant this change its disruption. A requested reboot has no document, so its entry carries the running boot's identity, which spends itself the same way. |
| <span id="statuspending--summary"></span>`summary` | string | yes | One line saying what the document changes. |


# The release channel

A release channel is a directory that any web server can share. The
public channel is at [releases.liken.sh](https://releases.liken.sh/).
Machines download upgrades from it, and
[`liken fetch`](/docs/reference/cli/#liken-fetch) downloads releases
from it to your workstation. [Upgrade the
fleet](/docs/guides/upgrade/) gives the steps that use it.

## Layout

    channel.yaml               the channel document: the latest version
    versions.yaml              the versions document: every release, by digest
    <version>/                 one directory per release
      release.yaml             the release document: every artifact, by digest
      vmlinuz                  the Linux kernel
      liken.sqfs               the operating system: a read-only squashfs
                               image a machine mounts as its root
      boot.cpio                the small initramfs: init and the early
                               boot's kernel modules
      microcode.cpio           CPU microcode, loaded ahead of everything else
      liken                    the toolkit
      systemd-bootx64.efi      the install stick's boot menu, for UEFI
      grub-boot.img            the BIOS boot loader's first stage
      grub-core.img            the BIOS boot loader's second stage
      LICENSES.md              third-party license notices
      notes.md                 what changed in this release
    sources/                   source mirrors for GPL and LGPL components,
      <component>/<version>/   keyed by the component's own version
    index.html                 the index pages, one at the root, one in
                               each release, and one in sources/
    favicon.ico                the icon browsers request

## Versions

A version is a calendar date and a serial number: `2026.07.20-001`.
Every field is zero-padded to a fixed width, so a plain string
comparison puts versions in the correct order. The serial starts at `001` and
increases during the day. No published release uses serial `000`.

The version gives the date only. The `components` section of the
release document records what is in the release.

A version always refers to the same bytes. If a release is bad,
nobody builds it again or publishes it again with the same name.
Publish the next serial number instead.

## The release document

`release.yaml` is a `Release` document (`apiVersion:
liken.sh/v1alpha1`). It has two lists:

* `artifacts`: each file in the release, with its `name`, its
  `sha256`, and its `size`.
* `components`: the upstream projects in the release, each with its
  `name` and `version`. These are the kernel, k3s, and the other
  components.

## The channel document

`channel.yaml` is a `Channel` document. Its `latest` field records
the newest published version. The cluster polls this document to fill the
AVAILABLE column of `kubectl get clusters`, and `liken fetch
... latest ...` reads it to resolve `latest`.

The channel document is the only object in the channel that changes.
Every other object is published one time and never changes, which is
why a machine can verify a release byte for byte against a pinned
digest.

## The versions document

`versions.yaml` is a `Versions` document. It lists every release that
the channel holds, newest first, each with the digest of its release
document:

```yaml
apiVersion: liken.sh/v1alpha1
kind: Versions
metadata:
  name: liken
latest: 2026.07.25-002
releases:
  - version: 2026.07.25-002
    digest: sha256:f5a46b8b08405d6b79c4792c896089e6b3cbbb4ce1951e84a6a36656520cd616
```

Each entry has the shape of a catalog entry, so you can copy one
straight into your cluster's
[`spec.releases.catalog`](/docs/reference/cluster/#specreleasescatalog).

Read this document when you want the whole list in one request. The
storage refuses to list itself, so this file is the only way to learn
which versions exist without opening the front page.

No machine reads this document. A cluster polls `channel.yaml`, which
stays one small file however many releases exist. The digests here are
a convenience, not an authority: a digest that the channel served
vouches for nothing by itself, which is why you pin the digest in your
own Cluster.

## The notes

`<version>/notes.md` lists what changed in the release: the commit
subjects since the release before it. The release's page on the
channel shows the list, and the body of the release's page on GitHub
wraps the same text. No digest pins the notes, `release.yaml` does
not list them, and no machine reads them. They are announcement
prose, in the same trust class as the index pages. That is also what
lets a release published before notes existed gain them later.

## The index pages

Open [releases.liken.sh](https://releases.liken.sh/) in a browser to
read the channel. The front page lists every release, newest first,
and marks the one that `channel.yaml` records as the latest. Each
release has a page
at `https://releases.liken.sh/<version>/`. It gives the catalog entry
to copy, every artifact with its digest and its size, and the
component versions in the release. The mirror at
[sources/](https://releases.liken.sh/sources/) has a page too.

These pages carry no information of their own. Each page is a view of
the documents above, and no machine reads a page. A machine reads
`channel.yaml` and `release.yaml`, and it verifies both.

## The release page

Every release also has a page on GitHub, under
[liken-sh/liken/releases](https://github.com/liken-sh/liken/releases).
CI makes the page after it publishes the release to the channel. The
page is the announcement, not the distribution. It gives the digest,
the catalog entry that you can copy, and the changes after the last
release. The binaries stay on the channel, with the license notices
and the source mirror. To hear about new releases, use the page's
feed.

## The trust chain

The chain has three links:

1. The Cluster's catalog entry pins the sha256 of the exact bytes of
   `release.yaml`. The release's own page on the channel, and its page
   on GitHub, publish the same digest.
2. `release.yaml` pins the sha256 of every artifact.
3. Each machine, and `liken fetch`, verifies every downloaded byte
   against these digests before it uses the bytes.

## Sources

`liken` redistributes binaries with the GPL license and the LGPL
license, so the channel also serves the source of each such
component, at `sources/<component>/<version>/`. The `LICENSES.md`
file in every release gives the notices. The paths use the
component's own version, not the release's version, because one
component version can be in many releases.


# Devices

liken gives hardware to workloads with dynamic resource allocation,
the Kubernetes API for devices. A pod asks for a device by its
properties. The scheduler selects a node that has such a device, and
the node gives that pod the device's `/dev` entries. The pod needs no
privilege, no host mounts, and no knowledge of the machine it runs
on.

[Give a workload a device](/docs/guides/devices/) gives the steps.
This page describes what liken publishes, and why.

## The driver

The driver's name is `liken.sh`. It is not a separate program. It is
two jobs in the machine operator, which runs on every node. One job
reads sysfs and publishes what it finds. The other job answers the
kubelet when a pod's claim comes to the node.

Usually a cluster administrator installs a DRA driver as a DaemonSet.
liken's driver is part of the operating system, because the operating
system is the only thing that identifies the hardware before other
software starts.

The [hardware operators](/docs/concepts/hardware-operators/) are
separate DRA drivers that you install as workloads. Each one
publishes a kind of device this driver does not, such as the
controllers paired to a Bluetooth radio.

## What a node publishes

Each node publishes one `ResourceSlice`, with the name
`<node>-liken.sh`. The slice lists the node's claimable devices:

    kubectl get resourceslices
    kubectl get resourceslice <node>-liken.sh -o yaml

A device on the PCI bus or the USB bus appears in the slice when
these three conditions are true:

1. **A driver is bound to it.** Hardware with no driver can supply
   nothing, so it goes in the machine's unclaimed report instead. To
   move a device from one report to the other, declare its module in
   the Machine's `spec.modules`.
2. **A claim on it supplies something.** The device's sysfs subtree
   must contain device nodes. A network card is hardware, but it has
   nothing to give a pod, so it never appears. A Bluetooth adapter is
   the one exception. Its driver, `btusb`, puts it in the slice,
   because the kernel gives a radio no device node at all. See
   [Bluetooth adapters](#bluetooth-adapters).
3. **The machine does not depend on it.** A device whose subtree holds a
   storage role belongs to the machine, and never to a workload. A
   disk belongs to the machine, through a storage role, or to the
   workloads, through a claim, but never to both.

The slice is an offer, and not a full record of the machine's
hardware. The
scheduler can allocate only what a slice lists, so the contents of
the slice are the control.

The slice does not list the bus structure. Hubs, PCIe ports, and the
USB core's own devices are the structure that peripherals attach to,
not peripherals.

## Device names

A device's name is its bus and its address, with dashes in place of
the punctuation: `pci-0000-00-02-0`, `usb-2-1-1-0`.

The address gives the slot, not the unit. If you replace a dongle
with an identical dongle in the same port, the device name does not
change, which is what a claim on the adapter on that port needs. To
select one physical unit instead, select on its `serial` attribute.

A constraint pairs the requests of a claim by an attribute, and never
by name. The address is also an attribute, `address`, so a claim can
constrain two requests to one physical device.

## Attributes

Every attribute belongs to the driver's domain, so a selector reads
it as `device.attributes["liken.sh"].<name>`. If the hardware does
not have an attribute, the attribute is absent, not empty. Thus
`has(device.attributes["liken.sh"].serial)` gives a correct result.

| Attribute | Type | What it is |
|---|---|---|
| `bus` | string | `pci` or `usb` |
| `address` | string | the device's address on that bus: `0000:00:02.0` on PCI, the port path on USB. Every device liken publishes for one physical device carries the same address |
| `driver` | string | the name of the bound driver, such as `i915` |
| `class` | string | the type of device, in one word: `display`, `multimedia`, `serial-bus` |
| `classCode` | string | the full class code that the bus published: six hex digits on PCI, two on USB |
| `subsystem` | string | the kind of device that liken published: `drm`, `sound`, `tty`. It is absent when the delivery is a mix that liken does not know, and when the device delivers no node of its own, as a Bluetooth adapter does |
| `renderNode` | bool | the device supplies a DRM render node |
| `displayNode` | bool | the device supplies a DRM card node, which carries modesetting |
| `name` | string | the name of the device in words, from its own strings or from the PCI database |
| `modalias` | string | the identifier that the kernel uses to match drivers |
| `serial` | string | the serial number of the hardware, when it has one |
| `vendor` | string | the vendor ID, lowercase hex with no prefix |
| `product` | string | the product ID, lowercase hex with no prefix |

Use the attribute that describes what you need. `renderNode` and
`classCode` describe a capability of the hardware, and they stay
correct across a fleet of different machines. `vendor` and `product`
give one model, and a DeviceClass that uses them stops working on the
next machine that you buy.

`address` is the attribute that pairs devices. Two cards in one
machine have different addresses, and every device liken publishes for
one card carries that card's address. So a claim with a request for a
render node and a request for a card node constrains the two with
`matchAttribute: liken.sh/address`, and both requests get halves of
the same card. The
[guide](/docs/guides/devices/#two-requests-one-card) has the claim.

liken publishes no capability facts that need a driver stack to
measure, for example the codecs that a GPU can encode. To read those
facts, you must run libva and a vendor driver, which the image does
not contain. A pod that holds a claim on the render node can measure
them for itself, so the operating system does not publish them.

## Sharing

liken allocates a device to one claim, unless liken publishes
`allowMultipleAllocations` for that device. A device's subtree can
deliver nodes of more than one kernel subsystem. When it does, liken
can publish more than one device for it, one for each kind of node. A
GPU splits again, because its render node and its card node give
different authority over the same silicon. The name of each extra
device is the primary device's name plus a suffix. A claim receives
the nodes of the one published device it allocated, and no others.

liken publishes `allowMultipleAllocations` for one kind of device:
the graphics half of a GPU. A graphics device is one that delivers a
DRM render node, and the published graphics device delivers that
render node, `/dev/dri/renderD*`. It carries `renderNode: true`. The
driver arbitrates between concurrent clients on a render node, so
more than one claim can hold it.

An audio controller is a device that delivers ALSA's nodes, and no
nodes except the ones a sound card holds. It carries
`subsystem: sound`, and it is exclusive. In practice one sound server
owns every PCM on a card and mixes its clients' streams through them,
so the card belongs to one claim. A second claimant waits in the
scheduler, where a person can see it wait, instead of meeting ALSA's
`EBUSY` at play time.

A claim on an audio controller delivers the card's whole subtree,
which is more than the `/dev/snd` nodes. ALSA registers an input
device for each jack that it can sense, and an HDA controller with
HDMI outputs has one for each display pin, so the claim also delivers
those `/dev/input/event*` nodes. A jack reports the state of an
output that the same claim plays through, so it is not a separate
device.

A GPU publishes its card node, `/dev/dri/card*`, as a separate
device. The name of that device is the primary name plus `-display`,
and it carries `displayNode: true`. This device is exclusive, because
DRM master is one for each card: the kernel gives modesetting to one
open card node, and a second display program on the same card fails
when it starts. An exclusive device makes the second claim wait in
the scheduler instead, where a person can see it wait. A workload
that modesets and also renders claims both devices, with one request
for each. On a machine with two cards, a constraint of
`matchAttribute: liken.sh/address` on those two requests keeps both on
one card.

A GPU driver also registers i2c monitor-control buses. These publish
as their own device, with `subsystem: i2c-dev`. That device stays
exclusive. An i2c node passes raw transfers to every device on its
wire, and two writers on one wire have no arbitration contract. A
DisplayPort output also registers a DisplayPort AUX channel, with
`subsystem: drm_dp_aux_dev`. This node publishes the same way, as its
own exclusive device, and it exists only while a display is
connected. The legacy framebuffer node is not delivered at all:
holding it grants display takeover, and no workload claims a bare
framebuffer.

A device that delivers one kind of node publishes as one device,
unless it is a GPU. A device that delivers a mix that liken does not
know publishes whole and exclusive, and names no `subsystem`. A
Bluetooth adapter publishes the same way, whole and exclusive with no
`subsystem`, because the only node it delivers is its usbfs node.

A DRM render node has a multiplexing contract in the kernel: the
driver arbitrates between concurrent clients. A drill measured this
on an integrated GPU. Twelve concurrent encoders divided the GPU
equally, and two pods each got approximately half of the throughput
that one pod got alone. A serial port has no such contract, and a
dongle's control endpoint has none.

Only the driver can state this, because only the driver writes a
`ResourceSlice`. Thus the rule is narrow. If a device is incorrectly
marked as shareable, two workloads get the same hardware while each
one operates as the only user, and no DeviceClass, claim, or workload
can correct it. If a device is incorrectly marked as exclusive, a
claim waits, and a person can see that it waits.

### Sharing a claim is not the same thing

`allowMultipleAllocations` controls how many **claims** can allocate
a device. It does not control how many **pods** can use one claim.

Every pod that names a `ResourceClaim` shares that claim, and all of
these pods receive the device. Kubernetes does this on purpose: it is
how two pods share one allocation. liken does not refuse the second
pod. A refusal would be a race with no owner. Also, the delivery of a
device node was never the mechanism that gave exclusive access. The
kernel gives exclusive access, through `O_EXCL` and the driver's own
open path.

To give a device to one pod only, use a `ResourceClaimTemplate`,
which makes a separate claim for each pod. If you use one
`ResourceClaim` for more than one pod, you share the device on
purpose.

## What a claim delivers

A claim delivers device nodes only. The node writes a CDI
specification that names the `/dev` entries for that claim, and the
container runtime injects them into the containers that requested the
claim. The claim grants no privilege, mounts no host path, and loads
no kernel module for the pod.

The workload supplies everything else that it needs: the userspace
library that communicates with the device, and the group membership
that its image gives its user.

### USB devices

A claim on a USB device also delivers that device's usbfs node,
`/dev/bus/usb/<busnum>/<devnum>`. A program that uses libusb, for
example Network UPS Tools, reads sysfs to find the hardware and then
opens this node to communicate with it. A node that a kernel driver
registers, such as `hidraw`, carries that driver's protocol only, so
it cannot take the place of the usbfs node.

A program that uses libusb cannot share an interface with a kernel
driver, so it detaches the kernel driver while it runs. liken
publishes only devices that have a driver, so the device leaves the
node's slice for as long as the pod runs. When the pod stops and its
claim ends, the node binds a kernel driver to the interface again, and
the device returns to the slice at the next reconcile pass.

The kernel gives the device a new device number at each enumeration.
The node changes when you unplug the device and plug it in again.
Each reconcile pass writes the current node into the specification of
every claim that the kubelet prepared, so the next pod receives the
current node. A container that already runs keeps the node that it
received, and it receives the current node when the pod restarts.

usbfs has no interface boundary. If a device has more than one
interface with a driver, each interface publishes as its own device,
and each one delivers the same usbfs node. A pod that holds one of
these claims can communicate with the whole device.

### Bluetooth adapters

A claim on a Bluetooth adapter delivers the adapter's usbfs node, and
nothing else. The adapter has no node of its own. A program reaches a
radio through an `AF_BLUETOOTH` socket, which it binds to an adapter
by index, so the kernel registers no `/dev` entry anywhere the adapter
owns. The claim states which workload owns the radio, and it holds
that workload on the machine the radio is in. A stack that drives the
radio through the kernel opens its socket with the capabilities of its
own container, because a claim grants no privilege.

liken stops the delivery walk at a `bluetooth` subtree. The kernel
puts the HID device of a connected peripheral under the adapter's own
USB interface, so a game controller's `/dev/input/event*` and
`/dev/hidraw*` nodes appear in the adapter's part of sysfs. Those
nodes belong to the controller. A claim on the adapter does not
deliver them, and a controller that connects later adds no nodes to
the claim.

The same nodes move when BlueZ changes how it drives the kernel. With
`/dev/uhid` present, BlueZ 5.73 and later create the HID device under
`/sys/devices/virtual/misc/uhid`, where nothing connects it to the
adapter. A claim that delivered these nodes would deliver them only
under one of the two arrangements.

## Limits

* One slice for each node, with a maximum of 128 devices. If a node
  has more devices, it prints the count of the devices it dropped on
  its console, and no claim can reach those devices. It does not
  divide the pool.
* liken supplies no DeviceClasses. A DeviceClass states what a
  deployment needs, so each deployment writes its own. The
  [guide](/docs/guides/devices/) has examples to start from.
* Devices have no capacity and no taints at this time. If a device
  fails, it disappears from the slice at the next pass. This stops
  new allocations, but it does not change an allocation that is in
  use.
* liken does not keep a claim across a reboot. The claim is a
  Kubernetes object, and Kubernetes reschedules the pod that holds it
  together with the claim.

## Hardware that is not published

You cannot claim hardware that has no driver bound to it. liken shows
this hardware in two other places:

* The Machine's status lists it as unclaimed hardware, with the
  modules that can drive it.
* The [hardware report](/docs/guides/install/#first-run-the-hardware-report)
  lists it before the first install, as commented lines under
  `spec.modules`.

To publish the device, declare the module in `spec.modules`. This is
the only necessary step. The device appears in the node's slice at
the next reconcile pass.


# Kernel settings

A Linux kernel has about 1500 tunable settings, called sysctls. Each one
is a file under `/proc/sys`, and writing the file changes the setting.
The kernel ships a default for every one of them.

Most Linux distributions do not run on those defaults. They ship a set
of files under `/usr/lib/sysctl.d`, and systemd applies them at boot.
liken runs no systemd. It applies its own set instead, listed below.

## How the values are applied

Two programs apply kernel settings, and they apply them in the same
order.

1. liken's own values, listed on this page.
2. The values in the machine's
   [`spec.sysctls`](/docs/reference/machine/#spec--sysctls).

Init applies both at boot, before k3s starts. The liken operator
applies both again on every pass, about every ten seconds. Each pass
writes the value and reads it back, so a setting that something else on
the machine changed returns within one pass, without a reboot.

Because `spec.sysctls` is applied second, a name there overrides
liken's value for that parameter:

```yaml
apiVersion: liken.sh/v1alpha1
kind: Machine
metadata:
  name: node-1
spec:
  sysctls:
    vm.max_map_count: "524288"
```

`status.sysctls` reports both sets together, read back from the kernel:

```console
$ kubectl get machine node-1 -o jsonpath='{.status.sysctls}' | jq
{
  "kernel.pid_max": "4194304",
  "net.core.default_qdisc": "fq_codel",
  "vm.max_map_count": "524288",
  ...
}
```

A parameter liken could not write is missing from that map, because a
failed write is never read back. So the map lists the parameters that
currently hold, not the parameters somebody asked for.

If a value in `spec.sysctls` fails to apply, the machine reports
`SysctlsApplied` as false and becomes Degraded. If one of liken's own
values fails, the machine stays Ready and the condition reports
`DefaultsIncomplete`. The reason for the difference is that liken's
values ship with the release, so every machine running that release
would report the same fault in the same pass, and degrading a whole
fleet at once hides the one machine that has a real problem.

## The values liken sets

This list holds only the parameters where liken differs from the
kernel. Every other parameter keeps the kernel's own default.

### Memory

| Parameter | Value | Why |
| --- | --- | --- |
| `vm.watermark_scale_factor` | `100` | Reclaims memory steadily in the background. At the kernel's default, reclaim starts only when allocation is close to failing, and then many allocations stall at once. |
| `vm.max_map_count` | `262144` | The kernel allows 65530 memory mappings per process, which a container runtime exhausts. The failure is confusing: a request for memory fails on a machine that has memory free. |

### Watches

| Parameter | Value | Why |
| --- | --- | --- |
| `fs.inotify.max_user_instances` | `8192` | Every program that watches files on the machine runs as root and draws on one quota: kubelet, containerd, k3s, init, and the liken operator. The kernel allows 128. |
| `fs.inotify.max_user_watches` | `524288` | The same quota, counted in watched files rather than watchers. |

Both are ceilings. An unused ceiling costs no memory.

### Processes

| Parameter | Value | Why |
| --- | --- | --- |
| `kernel.pid_max` | `4194304` | The kernel allows 32768 process IDs, a limit that predates the container. A machine running many short-lived containers reaches it, and then no program can start. This is the largest value a 64-bit kernel accepts. |

### Networking

| Parameter | Value | Why |
| --- | --- | --- |
| `net.ipv4.ip_forward` | `1` | A node that cannot forward packets cannot route pod traffic. |
| `net.core.default_qdisc` | `fq_codel` | The queueing discipline a network interface gets when it comes up. The kernel's default puts every packet in one queue, so a large transfer delays every small request behind it. `fq_codel` gives each flow a queue of its own. |
| `net.ipv4.tcp_slow_start_after_idle` | `0` | Stops a connection restarting slowly after every quiet moment. An API server watch stays open for hours and carries a burst, a silence, and another burst, which is the worst case for the kernel's default. |
| `net.ipv4.tcp_mtu_probing` | `1` | Recovers from a network path that drops oversized packets without saying so. Pod-to-pod traffic inside VXLAN encapsulation travels such a path. |
| `net.unix.max_dgram_qlen` | `512` | The queue depth of a local datagram socket. The kernel allows 10, and a program that fills the queue blocks. |

### Reliability

| Parameter | Value | Why |
| --- | --- | --- |
| `kernel.panic_on_oops` | `1` | A kernel bug that does not kill the machine outright leaves it running in a state it does not understand. liken has somewhere better to go: the machine reboots into the system slot it already proved, and the crash journal carries the log across the reboot, so the next boot reports the crash. |

### Hardening

| Parameter | Value | Why |
| --- | --- | --- |
| `kernel.kptr_restrict` | `1` | Hides real kernel addresses, which are what an attack needs to work out where the kernel loaded. |
| `fs.protected_symlinks` | `1` | Refuses to follow another user's symbolic link in a directory that anyone may write to. |
| `fs.protected_hardlinks` | `1` | Refuses a hard link to a file the person making the link cannot read. |
| `fs.protected_regular` | `2` | Refuses to create a file where another user already left one, in a directory that anyone may write to. |
| `fs.protected_fifos` | `1` | The same protection for named pipes. |

### Interface settings

A network setting under `net.ipv4.conf` exists three times: once under
`all`, once under `default`, and once for each interface. The kernel
copies `default` into an interface when that interface appears, and the
machine's own network card appears before these values are applied. So
`default` reaches the interfaces that Kubernetes creates later, and
`all` is what reaches the network card. liken sets both.

| Parameter | Value | Why |
| --- | --- | --- |
| `net.ipv4.conf.all.rp_filter`, `net.ipv4.conf.default.rp_filter` | `2` | Drops a packet whose source address no interface would route a reply to. This is the loose form of the check. Do not set the strict form, `1`: pod traffic routinely arrives on a different interface from the one that would reply, and the strict form drops it. |
| `net.ipv4.conf.all.accept_source_route`, `net.ipv4.conf.default.accept_source_route` | `0` | Refuses a packet that carries its own return path. Nothing on a server uses this, and it is a way to reach an address that ordinary routing protects. |
| `net.ipv4.conf.all.promote_secondaries`, `net.ipv4.conf.default.promote_secondaries` | `1` | Keeps an interface's other addresses when its primary address is removed. The kernel's default deletes them all. |

## Settings that are not sysctls

Not every kernel setting is a sysctl. Four other kinds matter.

**Resource limits** are ceilings the kernel puts on one process, such
as how many files it may hold open. They are not files under
`/proc/sys`, and no sysctl changes one. liken sets its own, and
`spec.rlimits` overrides them.
[Resource limits](/docs/reference/rlimits/) describes both sets.

**The kernel command line** is read once, at boot. It carries the
settings the kernel needs before any program runs. liken builds this
line itself, and there is no field to add to it.
[`status.boot.commandLine`](/docs/reference/machine/#statusboot--commandline)
reports the line the machine booted with.

**Kernel modules** are drivers and other kernel parts that load on
demand. Some sysctls need one: `net.core.default_qdisc` cannot name
`fq_codel` unless the `sch_fq_codel` module is loaded. liken loads a
fixed list at boot, and
[`spec.modules`](/docs/reference/machine/#spec--modules) adds to it.

**Compiled settings** are fixed when the kernel is built, and liken
does not build kernels. The build's own configuration ships in the
image at `/lib/modules/<release>/config`. Read it to find out whether
some kernel feature is built in, available as a module, or absent:

```console
$ grep CONFIG_NET_SCH_FQ_CODEL /lib/modules/$(uname -r)/config
CONFIG_NET_SCH_FQ_CODEL=m
```


# Resource limits

A resource limit is a ceiling the kernel puts on one process: how many
files it may hold open, how many processes one user may run, how large
a core dump may be. Every limit has two halves. The soft limit is the
one in force. The hard limit is the ceiling on the soft limit. A
process may lower either half, and may raise its soft limit up to its
hard limit, but it can never raise its hard limit.

Two rules make these limits different from
[sysctls](/docs/reference/sysctls/). The kernel fixes a
process's limits when the process starts, and it copies them from the
parent. So a limit reaches a program only if whoever started that
program held the limit first, and nothing can change a program that is
already running.

Most Linux distributions set these limits for each service, in the
`Limit` directives of its systemd unit, and systemd applies them when
it starts the service. liken runs no systemd. It applies its own set
instead, listed below.

## How the values are applied

Init applies two sets of limits to itself, in this order.

1. liken's own values, listed on this page.
2. The values in the machine's
   [`spec.rlimits`](/docs/reference/machine/#spec--rlimits).

Both run at boot, before k3s starts. Init is the first process on the
machine, so every process started after this point inherits the
result: k3s, containerd, the containerd shims, and every container.
Inheritance is the whole mechanism. There is no file to write and no
per-container setting.

Because `spec.rlimits` is applied second, a name there overrides
liken's value for that resource:

```yaml
apiVersion: liken.sh/v1alpha1
kind: Machine
metadata:
  name: node-1
spec:
  rlimits:
    nofile: "524288"
    memlock: "67108864"
```

An edit to `spec.rlimits` takes effect at the next boot, not on the
next reconcile pass. This is the one way resource limits differ from
sysctls in practice. The kernel fixes a process's limits when it
forks, so nothing can raise the ceiling of a k3s that is already
running. The operator stages the edit and reports it, and
[`rebootPolicy`](/docs/reference/machine/#spec--rebootpolicy) says
who starts the boot that applies it. A container
that is already running keeps its old limits until its pod restarts,
which the reboot does anyway.

## Writing a value

A value uses the same form a systemd unit file uses.

| Form | Meaning |
| --- | --- |
| `"1048576"` | Sets both halves to 1048576. |
| `"infinity"` | Sets both halves to no limit. |
| `"1024:1048576"` | Sets the soft limit to 1024 and the hard limit to 1048576. |

A soft limit above the hard limit is refused, because no process may
exceed its hard limit.

These resource names are accepted: `nofile`, `nproc`, `core`,
`memlock`, `stack`, `fsize`, and `nice`. Any other name is refused,
so a misspelling reports itself instead of applying nothing.

## The values liken sets

This list holds only the resources where liken differs from the
kernel. Every other resource keeps the kernel's own default. Both
values match k3s's own systemd unit.

| Resource | Value | Why |
| --- | --- | --- |
| `nofile` | `1048576` | The kernel allows 1024 open files, with a ceiling of 4096. A container runtime and its workloads exhaust this quickly, and a program that runs out reports `Too many open files` on a machine with no shortage of memory or disk. 1048576 is also the kernel's `fs.nr_open` default, which is the largest value any process may ask for. |
| `nproc` | `infinity` | The kernel derives this ceiling from the machine's memory, so a small machine gets a small one. Root is exempt from the check, which hides the problem until a workload runs as an ordinary user. Kubernetes counts processes per pod, in cgroups, which is where a limit on a workload belongs. |

## The value liken does not set

k3s's unit also sets `LimitCORE=infinity`, and liken does not.

The kernel already grants an unlimited hard limit for core dumps, so
the only difference is the soft limit, which the kernel leaves at 0.
On a systemd machine, raising it is safe because `kernel.core_pattern`
sends each dump to `systemd-coredump`, which bounds its size and
expires it. liken has no such collector, so `core_pattern` keeps the
kernel's own value and a crashing process writes a file the size of
its memory into its working directory. For a container that directory
is inside its own writable layer, which lives on the same filesystem
as the cluster's datastore.

Nothing is blocked by this. A program that wants a core can raise its
own soft limit, because the hard limit is already unlimited. A
deployment that wants core dumps from every process can set `core` in
`spec.rlimits`.

## Reading the result

`status.rlimits` reports what the kernel holds, read back after both
sets are applied:

```console
$ kubectl get machine node-1 -o jsonpath='{.status.rlimits}' | jq
{
  "memlock": "67108864",
  "nofile": "524288",
  "nproc": "infinity"
}
```

A resource liken could not set is missing from that map, because a
failed write is never read back. So the map lists the limits that
currently hold, not the limits somebody asked for.

`status.boot.rlimits` reports what the manifest of that boot declared.
It carries the request rather than the result, so it is the record the
operator compares a later edit against.

On the machine itself, the kernel reports the same limits in its own
layout. Read them for the first process, and for any process below it:

```console
$ cat /proc/1/limits
Limit                     Soft Limit   Hard Limit   Units
...
Max open files            1048576      1048576      files
```

Read the same file for a process inside a container to confirm that
the limits reached the workload. `/proc/sys` is the wrong place to
look: resource limits belong to a process, so no file under `/proc/sys`
reports or changes them.


# The `liken` command

`liken` is the toolkit that you use to make and to operate a
deployment. It runs on your workstation, and every release includes
it. You do not need the repository or a build to make a cluster. To
print the full usage, run `liken` with no arguments.
[Install a cluster](/docs/guides/install/) runs the common commands
in order.

Three terms occur on this page:

* An **identity directory** holds the certificates and the join token
  that all the machines in one cluster share. Some of the files are
  private keys. Keep the directory out of version control.
* A **deployment layer** is a small archive that holds the parts of
  the operating system that are yours and not `liken`'s: your
  manifests and your identity. A machine boots the generic image and
  your layer together.
* A **release channel** is a directory that any web server can share.
  [The release channel](/docs/reference/release-channel/) describes
  its layout.

## liken new

    liken new <directory>

Starts a deployment. The command asks a few questions and writes a
directory of manifests: `cluster.yaml` and one file for each machine.
The comments in the files describe every field. The other commands
use this directory.

## liken mint

    liken mint <identity-dir>

Makes a new cluster identity: the certificate authorities and the
join token that all the machines in one cluster share.

## liken adopt

    liken adopt <harvest-dir> <identity-dir>

Takes identity files that you copied from the server of an existing
cluster, and arranges them as an identity directory. You can adopt
the identity of any k3s cluster. [Adopt an existing k3s
cluster](/docs/guides/adopt/) gives the steps.

## liken kubeconfig

    liken kubeconfig [-server URL] <deployment-dir>

Writes an administrator kubeconfig to
`<deployment-dir>/identity/kubeconfig`: the credential that `kubectl`
uses to administer the cluster. The server address comes from the
`endpoint:` in the deployment's `cluster.yaml`. Pass `-server` when
your machine reaches the cluster at a different address.

## liken approve-reboot

    liken approve-reboot [-server URL] <deployment-dir> <machine>

Reports what a machine waits for, and grants it one disruption. A
machine whose
[`rebootPolicy`](/docs/reference/machine/#spec--rebootpolicy) is
`Manual` stages each change and waits. This command reads the
machine's `status.pending`, prints each waiting change, and writes
the `liken.sh/approve-disruption` annotation with the staged
change's hash. The machine then takes the same path an `Auto`
machine takes: it waits for the cluster's turn, drains, and applies
the change with the smallest disruption it needs, which for a
credentials change is a k3s restart, not a reboot.

The grant is one-shot. Once the change applies, its hash is no
longer pending, and the next change hashes differently, so a stale
annotation approves nothing. Running the command twice writes the
same annotation. When two changes are pending, the command approves
the reboot-class one, because a reboot applies every staged change.
When nothing is pending, it reports that and writes nothing.

## liken request-reboot

    liken request-reboot [-server URL] <deployment-dir> <machine>

Asks a machine to reboot when no change asks it to. Every other
reboot that `liken` performs applies a staged document, so a machine
that agrees with every document it was given has no way to reboot,
and it has no shell to be told from. Two cases need one anyway: a
kernel driver that bound the wrong device, which releases it only at
boot, and a machine you are experimenting on. The command writes the
`liken.sh/request-reboot` annotation, valued with the identity of the
boot that is running now.

The request skips none of the cluster's coordination. The machine
waits for the cluster to grant it a reboot turn under
[`spec.disruption`](/docs/reference/cluster/#specdisruption), cordons
its node, and drains its workloads, the same as a machine applying a
staged change. What the two policies decide is only the approval:

* `rebootPolicy: Auto` needs nothing more. The machine takes its
  turn, drains, and reboots.
* `rebootPolicy: Manual`, the default, reports `RebootPending` and
  waits, the same as it does for a staged change.
  [`liken approve-reboot`](#liken-approve-reboot) releases it,
  through the same annotation.

Nothing is staged, so the machine comes back on the documents it
already runs. The reboot promotes no system slot and proves no
release.

The request is one-shot, and nothing has to clear it. It names the
boot it was written for, so the boot that comes back is a boot the
annotation does not name, and the `RebootRequestHonored` condition
reads `True` again. Run the command a second time and it names the
new boot.

The annotation is the whole interface, so `kubectl` alone can write
it. The machine reports the value to use in the same
`RebootRequestHonored` condition:

    kubectl describe machine <name>
    kubectl annotate machine <name> liken.sh/request-reboot=<identity>

## liken kubectl

    liken kubectl [-server URL] <deployment-dir> [args...]

Runs the `kubectl` from your `PATH` against the deployment's
cluster. The command writes the admin kubeconfig (see
[`liken kubeconfig`](#liken-kubeconfig)), sets `KUBECONFIG`, and
hands the terminal to `kubectl`. Everything after the deployment
directory goes to `kubectl` unchanged.

## liken stern

    liken stern [-server URL] <deployment-dir> [args...]

Runs the `stern` from your `PATH` against the deployment's cluster,
the same way `liken kubectl` runs `kubectl`. `stern` tails the logs
of many pods at once.

## liken flux

    liken flux [-server URL] <deployment-dir> [args...]

Runs the `flux` from your `PATH` against the deployment's cluster,
the same way `liken kubectl` runs `kubectl`. liken plants the Flux
engine when the cluster declares the `flux` feature, so this is the
command that inspects it.

## liken layer

    liken layer <manifests-dir> <identity-dir> <output.cpio>

Packs your cluster's part of the operating system into one small
archive: your cluster manifest, your machine manifests, and your
identity.

## liken fetch

    liken fetch [-digest sha256:<hex>] <source-url> <version|latest> <channel-dir>

Downloads a published release from a channel into a local channel
directory, and verifies every artifact against the release document.
To take the version that the channel names as the newest, give
`latest`. `-digest` pins the release document to a known digest,
which completes the trust chain.

## liken media

    liken media <release-dir> <deployment.cpio> <output.cpio>

Builds a bootable install image from a downloaded release and your
deployment layer. Machines install themselves from it. Use this form
for direct-kernel boots, for example QEMU or PXE.

## liken stick

    liken stick [-console ttyS0] <release-dir> <deployment.cpio> <output.img>

Builds the disk image for the USB install stick: one stick for the
full deployment. Boot the stick, select an entry, and obey the
console.

The menu holds two entries for each machine in the deployment, in the
order of the machine names, and one entry for the stick itself:

    install as big
    wipe and reinstall as big
    install as little
    wipe and reinstall as little
    liken hardware report

`install as <name>` uses blank disks only. `wipe and reinstall as
<name>` erases every disk that the manifest of that machine declares,
then installs. The report entry is last in the list because it
applies to no machine: it describes the hardware of the machine in
front of it and changes no disk. The menu has no time limit, because
every entry writes to a disk or asks for a person. A machine that
stays at the menu does nothing until a person selects an entry.

`-console` is repeatable. It adds a `console=` argument that the
machines keep permanently. Use it to install a machine that has no
screen: the menu and all the messages also go to that port.

## liken bundle

    liken bundle [-slot-size 1Gi] <vmlinuz> <liken.sqfs> <boot.cpio> <microcode.cpio> <liken-cli> <systemd-boot.efi> <grub-boot.img> <grub-core.img> <licenses.md> <channel-dir> <version> [component=version ...]

Lays out a release: it copies the artifacts into the channel and
writes the `release.yaml` that names each one by its digest. The
project's own release workflow runs this command. A deployment does
not need it.

## liken serve

    liken serve <channel-dir> [address]

Shares a release channel over plain HTTP, and records each request in
a log. The address defaults to `:8017`.

## liken index

    liken index -source <url> <output-dir> < keys

Renders the index of a channel: a front page that lists every release,
a page for each release, a page for the source mirror, and the
`versions.yaml` document. Give the channel's object keys on standard
input, one key per line.
The command reads each release document from the channel at `-source`,
and writes the pages into the output directory. The contents of that
directory belong at the root of the channel, because the pages link
from the root.

The project's own release workflow runs this command. A deployment
does not need it.

The pages hold no information of their own. Each one is a view of a
document that the channel already serves, and no machine reads a page.
To repair a page, run the command again over the same channel.

## liken version

    liken version

Prints the version of the toolkit.