MetadataProvider

A MetadataProvider is one account with one metadata provider: the Secret that holds its key, and the facts it may serve. It lives in the namespace of the libraries that name it, because a pod mounts a Secret only from its own namespace. A Library names providers by name in spec.sources, in the order they are asked, and for each fact the first provider in that list that serves it is the one asked. The trailer and marks facts take the union of their providers, so they ask every provider in the list that serves them.

A MetadataProvider name is at most 32 characters, because library-operator builds the names of the objects it creates from it, and Kubernetes limits some of those names to 63 characters.

A MetadataProvider names exactly one provider block: tmdb, omdb, fanart, tvmaze, peertube, archive, theintrodb, introdb, or imdb. TMDb, OMDb, and Fanart.tv take a key from a Secret; TVmaze, the Internet Archive, IntroDB, and IMDb take none, so their blocks are empty. TheIntroDB takes a key where its block names a Secret and asks with none where it names none. PeerTube takes no key either, but it is software that many people run, so its block names the address of one instance. The PROVIDER column shows the block.

The operator paces every provider: one request at a time per block, with a fixed gap between requests that fits each provider’s stated limits, and a quarter second for the Internet Archive and IntroDB.

spec.facts is optional. A provider that names none serves every fact the operator can request, and status.facts, shown in the FACTS column, lists what it serves right now. That list is empty while the provider is not Ready.

apiVersion: library.liken.sh/v1alpha1
kind: MetadataProvider
metadata:
  name: tmdb
  namespace: media
spec:
  tmdb:
    secretRef:
      name: tmdb-key
      key: token
  facts: [identity]   # optional; omit to serve the whole table
---
apiVersion: library.liken.sh/v1alpha1
kind: MetadataProvider
metadata:
  name: omdb
  namespace: media
spec:
  omdb:
    secretRef:
      name: omdb-key
---
apiVersion: library.liken.sh/v1alpha1
kind: MetadataProvider
metadata:
  name: tvmaze
  namespace: media
spec:
  tvmaze: {}
---
apiVersion: library.liken.sh/v1alpha1
kind: MetadataProvider
metadata:
  name: imdb
  namespace: media
spec:
  imdb: {}
---
apiVersion: library.liken.sh/v1alpha1
kind: Library
metadata:
  name: movies
  namespace: media
spec:
  sources: [tmdb, imdb, omdb, tvmaze]
  # the rest as before

The operator checks each provider with one call to the provider, and reports the answer in the Ready condition: Reachable, NoSecret, Refused, LimitReached, Unreachable, or Unavailable. LimitReached is a key the provider answered with its request limit. OMDb answers a key it does not accept and a key that has spent its calls for the day with the same 401, and names which one in its answer, so the check reads the answer. A Refused or LimitReached message carries the provider’s own words. Unreachable is a check that got no answer at all and carries the error as its message. Unavailable is a check the provider answered with a status that says nothing about the account, and its message names that status code. The key reaches each phase container of a Library’s Job through a secretKeyRef that the kubelet resolves. It never passes through a status, a log, or the catalog.

The check calls a provider when the operator starts and when the provider’s metadata.generation changes. Otherwise it calls a provider whose last answer was Reachable or LimitReached once an hour, and every other provider every five minutes. Every call before the limit resets counts against it, and OMDb does not publish when it resets, so a key past its limit is asked once an hour. The operator reads the Secret only for a check call, because the Job takes the key through its secretKeyRef. An edit of the Secret alone therefore shows at the next call: within five minutes for a Refused or NoSecret provider, and within the hour for a Reachable one. An account with a daily allowance, such as OMDb’s thousand calls, spends at most twenty-four of them a day on the check while its key works.

The imdb block has no API to call. IMDb publishes its datasets as files, so the check sends one HEAD request for each file that the provider’s served facts read: title.ratings and title.episode for rating.imdb, and title.principals and name.basics for credits. Every file must answer 200 for Reachable. Any other status is Unavailable, and the message names the file. The check keeps the same schedule as every other provider, so while IMDb answers it sends one HEAD request for each file an hour. A HEAD request transfers no file.

status.imdb.datasets lists what IMDb returned for each file: its lastModified, etag, and size. A failed check leaves the entries of the last check that read the files. The UPDATED column shows the oldest lastModified. IMDb replaces each file every day, so a file older than three days writes the Stale condition with status True, and its message names the file and its date. Stale does not change Ready, because a file from last week still gives correct ratings and credits.

$ kubectl -n media get metadataprovider imdb
NAME   PROVIDER   READY   REASON      UPDATED   AGE
imdb   imdb       True    Reachable   9h        3d

The operator caches the files on the claim <provider>-datasets, 3Gi, in the provider’s namespace. The provider owns the claim, so deleting the provider deletes the cache. The claim is on the cluster’s StorageClass whose provisioner is per-node.liken.sh, whatever class the libraries use, and each node keeps its own copy of each file. The Cached condition is True with the reason PerNodeClass when the claim exists. A cluster with no per-node class gets no claim, and Cached is False with the reason NoPerNodeClass. Each enricher run then reads the files from IMDb directly. Cached does not change Ready either.

One account with one metadata provider, named by the Libraries of its namespace in spec.sources.

spec

The provider this account is with, and the facts it may serve. A spec that names no facts serves every fact the operator can request from this provider.

spec.tmdb

The account is with The Movie Database, which serves movies, series, and people.

spec.tmdb.secretRef

The Secret in this namespace that holds the credential, and the key inside it. Either credential TMDb issues works: a v3 API key of 32 hex characters, or a v4 read access token.

spec.omdb

The account is with OMDb. OMDb answers on an IMDb id, and it serves the plot, the US certification, and the ratings of IMDb, Rotten Tomatoes, and Metacritic.

spec.omdb.secretRef

The Secret in this namespace that holds the OMDb key, and the key inside it. The free tier of a key is a thousand calls a day.

spec.fanart

The account is with Fanart.tv, which provides art only. It is the only provider of clearart, banner, landscape, discart, and season-banner files.

spec.fanart.secretRef

The Secret in this namespace that holds the Fanart.tv project key, and the key inside it.

spec.peertube

The account is with one PeerTube instance, which provides only the trailer fact and needs no account. Many people run PeerTube, so this block identifies the instance by its address.

spec.theintrodb

The account is with TheIntroDB, a community database of the intro, recap, credits, and preview spans of movies and episodes. It provides only the marks fact, and it finds a work by its TMDb id. A key is optional. Without one, TheIntroDB answers 500 asks a day for each address and serves accepted submissions alone. With one, it answers 1000 asks a day for the account and adds the account’s own pending submissions. The operator checks it at /health, which spends none of the daily allowance and cannot test the key.

spec.theintrodb.secretRef

The Secret in this namespace that holds the TheIntroDB API key, and the key inside it. Omit it to ask with no key.

status

What the operator’s own check found, written only by the library operator.

status.imdb

What the check of an imdb block read from IMDb. A failed check leaves the entries of the last check that read the files, because the last version IMDb published is still a fact.

status.imdb.datasets[]

One entry for each dataset file the served facts read, with the headers IMDb returned for the check’s HEAD request. rating.imdb reads title.ratings and title.episode, and credits reads title.principals and name.basics.

status.conditions[]

Ready is True with the reason Reachable when the provider answered the operator’s check, and False with the reason NoSecret, Refused, LimitReached, Unreachable, or Unavailable. LimitReached is a key the provider answered with its request limit, as OMDb does when a key has spent its calls for the day. The key is good, and the check asks again in an hour. Refused and LimitReached carry the provider’s own words in the message. Unreachable is a check that got no answer at all, and its message is the error the check read. Unavailable is a check the provider answered with a status that says nothing about the account, and its message names that status code. For an imdb block, the check sends one HEAD request for each file, and Unavailable names the file. An imdb block also carries two conditions that do not change Ready. Stale is True with the reason NotReplaced when IMDb has not replaced a file for more than three days, and its message names the file and its date. Cached is True with the reason PerNodeClass when the claim -datasets holds the files, and False with the reason NoPerNodeClass when the cluster has no per-node class, so each run reads the files from IMDb.