Metrics

Three processes serve Prometheus metrics at /metrics, under liken’s shared contract . The display-operator container serves plain HTTP on port 9200, one per node. The display-capture container serves them on port 9201, the same TLS port as the captures, and the route needs no token. The display-api Deployment serves plain HTTP on port 9200, once for the cluster. The base applies with no Prometheus in the cluster. An owner who runs the prometheus-operator adds the deploy/monitoring component beside the base, which holds a PodMonitor for each port.

Each process also sets liken_build_info{component, version} to 1. The component is display-operator, display-capture, or display-api.

Component Metric Type Why
display-operator display_output_connected{output} gauge a monitor that went away
display-operator display_output_claimed{output} gauge the hardware triple
display-operator display_observation_valid{source}, display_observation_last_success_timestamp_seconds{source} gauge the card or the compositor socket stopped answering
display-operator display_output_mode_info{output, mode, refresh} gauge, info what is actuated on each panel
display-operator display_compositor_restarts_total{reason} counter the restarts the operator orders, by reason; the unbounded-restart problem as a rate
display-operator display_compositor_serving gauge the compositor answers the handshake, refuses the connect, or answers nothing
display-operator display_compositor_container_restarts_total counter the kubelet’s count of the compositor’s crashes: the exits the operator did not order
display-operator display_surfaces{output} gauge what the compositor holds; a stuck surface
display-operator display_panel_power{output}, display_panel_brightness{output} gauge the DDC state the idle screen drives
display-operator display_reconcile_duration_seconds{kind} histogram how long one pass of the reconcile loop took
display-operator display_reconcile_errors_total{kind} counter passes of the reconcile loop that returned an error
display-operator display_watch_restarts_total{kind} counter watches the API server accepted after the first one, for each kind
display-capture display_capture_ready gauge the sidecar holds the certificate the API verifies it under
display-capture display_capture_conversion{graph} gauge which of the two frame-conversion graphs the node uses, vaapi or software
display-capture display_captures_active{aspect} gauge captures running on the node now
display-capture display_capture_bytes_total{aspect,format}, display_capture_seconds_total{aspect,format} counter how much the sidecar encoded and sent, and how long it streamed
display-capture display_capture_frames_total{format} counter frames taken from the compositor
display-capture display_capture_failures_total{reason} counter captures that failed
display-api display_api_requests_total{route,method,status} counter requests answered
display-api display_api_request_seconds{route} histogram time to the response headers
display-api display_api_streams_active{aspect} gauge capture responses the API is streaming now
display-api display_api_certificate_expiry_seconds gauge when the serving certificate expires, in seconds since the epoch 
resources:
  - https://github.com/liken-sh/liken//display-operator/deploy?ref=<version>
components:
  - https://github.com/liken-sh/liken//display-operator/deploy/monitoring?ref=<version>

The card observation

display_observation_valid{source="card"} reports whether the last read of the card succeeded. The operator reads the card only while it holds a connection to a compositor. While it holds none, the operator records no card observation: the gauge keeps the value of the last read, and display_observation_last_success_timestamp_seconds{source="card"} keeps the time of the last successful read, so its age grows.

Compositor restart reasons

display_compositor_restarts_total counts each restart of the compositor that the operator orders. The reason label says why:

reason Why the operator restarts the compositor
mode A claim’s mode parameter or a Display’s spec.mode changes the mode that a screen runs. Weston reads its config only at startup.
heal The compositor re-created an output with another monitor or another mode, so the canvases on the other screens have the wrong size.
hung The compositor accepted connections on its socket and answered nothing for 10 seconds. The operator sends SIGKILL.
masterless A read of the card found that the compositor holds no DRM master, so it cannot show a frame. The operator restarts each compositor process for this reason at most once.

The compositor’s container starts weston again at once after each of these restarts, so the kubelet does not count them. A compositor that exits on its own has no reason, and this counter does not count it. Its container exits with it, and display_compositor_container_restarts_total counts it, from the kubelet’s restart count.