anycloud cluster

An installation has one Kubernetes cluster: a management VM in your cloud running a pinned K3s server, the Anycloud API released onto it as a Helm chart, and the node pools that join it. anycloud cluster create builds it, anycloud api use points the CLI at the API it hosts, and anycloud cluster delete releases it.

Create the cluster and install the API

anycloud cluster create <name> \
  [--credentials <name>] \
  [--region <region>] \
  [--zone <zone>] \
  [--vm-type <type>] \
  [--checkpoint-bucket <name>]

In a terminal, create asks for any of --credentials, --vm-type and --region you leave out, and offers only regions that have the chosen VM type; --zone needs --region. Without a terminal, or with --json, it names the missing flags and stops.

Create and delete run on the installer API: the local API that holds the saved compute credential and whose database is the cluster's infrastructure record. They never depend on the hosted API they install, so a cluster can be resumed or deleted while that API is down. A hosted API refuses create with CLUSTER_UNAVAILABLE. Create records the cluster before it creates anything: the exact target (credential, region, zone, VM type), the pinned K3s version and binary checksum, the digest-pinned Helm Job image, the API image resolved from this CLI's version to its immutable digest, the digest of the chart that image names, and the checkpoint bucket for the hosted API's backups: --checkpoint-bucket names a bucket you own, reached with the compute credential, retained and never deleted by Anycloud (create checks it exists; one holding a backup makes create a restore). Without the flag, create makes an Anycloud-managed bucket named after the cluster id, enables object versioning on it, and delete removes it with every version. It then

1. creates one management VM in the customer's cloud, recorded with the cluster as its owner before the provider call;
2. starts a pinned K3s server on it with the management taint, disabled public components, and the VM's public address as a certificate SAN — the server configuration, with its join token, is written over the SSH channel's stdin, never as part of a remote command line;
3. creates a dedicated installer ServiceAccount (kube-system/anycloud-installer, bound to cluster-admin in this release) with a long-lived token and builds the installer's connection from that token, the server CA and the public endpoint https://<address>:6443 — the host's admin certificate never leaves the VM;
4. observes the control plane's kube-system namespace UID and binds it, with the connection encrypted, to the installation's one cluster record (step connection);
5. seeds release intent (revision 1) once, then applies it as one HelmChart through the Kubernetes API. The release values carry the cluster's name, identity, public endpoint and CA for the hosted API. Nothing is written into K3s's watched manifests directory, so a host restart replays nothing; and
6. publishes the API's DNS record <cluster-id>.anycloud.sh → the management VM's public address (proxied through Cloudflare, with your GitHub token) and installs the API's own Traefik release on the management Node as its ingress (K3s's bundled one stays disabled: its install Job cannot schedule onto the tainted Node); then
7. waits until the API Pod for that exact revision runs the pinned digest, is Ready and serves the expected version at https://<cluster-id>.anycloud.sh, which the command prints as URL: with the next step, anycloud api use <url>.

The command returns once the cluster is recorded, like submitting a Job; anycloud cluster status <name> --watch follows it through the Workload states provisioning (the management VM), initializing (Kubernetes and the installer's connection — nothing is registered or exported, see Cluster authority) and starting (the API release) to running or failed, then prints the next step. Re-running create with the same name resumes a failed cluster where it stopped: it adopts the VM attempt, connection and release intent already recorded for that cluster rather than creating new ones, and it refuses to continue when the control plane reports a different identity than the one it recorded. A request with a different credential or target is refused, a second cluster is refused while one exists, and a cluster whose delete did not finish (failed after a delete) can only be deleted, not created again. The exception is a restore from its backup, which replaces the record once the cloud shows its management VM is gone.

The API's URL is https://<cluster-id>.anycloud.sh. Cloudflare terminates TLS and reaches the management VM on port 443 or 80, admitted from Cloudflare's ranges only, where Traefik routes the hostname to the API. On 443 Traefik serves a Cloudflare Origin CA certificate for the hostname: create generates its key and writes it only into the cluster, the DNS Worker signs it after checking that your GitHub account holds the record, and delete and forget revoke it. Once 443 serves the certificate, Cloudflare connects to it in Full (strict) mode, so the hop to the VM is encrypted and verified. The management VM otherwise accepts TCP 22 and 6443 from any source (6443 is the pool join endpoint). The record is created once at bootstrap and never re-pointed — the VM's address is stable for its life — and deleted by delete. The API on the cluster starts with an empty database; the cluster, VM and credential records stay on the installer API. On its first start the hosted API records the cluster it runs in from the release values: it reads the kube-system UID through its own ServiceAccount and refuses to start when that UID differs from the one its release values name. A restored database instead adopts the new cluster (see recovery). Its record lists under anycloud cluster list against the hosted API with the cluster's id and no creation facts; no kubeconfig or token is stored for it — the API's authority is its chart-bound ServiceAccount (see Cluster authority). Before creating node pools through the hosted API, add the compute credential there (anycloud credentials new against the hosted API): credentials never leave the installer.

Inspect and delete

anycloud cluster list
anycloud cluster status <id-or-name> [--watch] [--interval <seconds>]
anycloud cluster delete <id-or-name> [--no-wait] [--timeout <seconds>]
anycloud cluster forget <id-or-name>

list shows the clusters this API knows with their location, VM type, age, address, installed API version and state; status shows one in the Workload status layout (state, cloud, VM, address, Kubernetes and API versions, URL, node pools, age and last error), and --watch refreshes it until the cluster is running, failed or fully terminated. Against a hosted API, its own cluster shows the address, API version and URL it knows of itself; the cloud, region and VM type are recorded only by the installer that created it. list also accepts ls and get, and status accepts describe. delete (aliases terminate, rm) releases the management VM through the cloud gateway, deletes the managed checkpoint bucket and the DNS record, clears the stored connection and marks the cluster terminated. Node pools are never deleted from here. Delete first asks the hosted API, with your account, which node pools it still manages: a pool with live Deployments or Services is refused and the answer names the steps — anycloud deployment delete <id> / anycloud terminate <id> against the hosted API, then anycloud node-pool delete <name>, then anycloud cluster delete <name> again; a pool without them is refused with the node-pool delete step. When the hosted API is unreachable, delete is refused too. It prints the restore command, anycloud cluster create <name> --credentials <credential> --region <region> [--zone <zone>] --vm-type <type> --checkpoint-bucket <bucket> (see recovery); if the management VM still runs but the cluster is broken, terminate that VM first. Then delete normally. If the bucket holds no restorable backup, or it cannot be read, the refusal names the VMs to terminate by hand: the management VM and the node pool VMs (tag anycloud_owner_api=<identity>). Then anycloud cluster forget <name> deletes this installer's record of the cluster and its <id>.anycloud.sh DNS record. It is allowed only while the hosted API is unreachable and the cloud shows the management VM gone. It never touches the cloud or the checkpoint bucket and its backups, and names what is left: the management VM's network, the node pool VMs, networks and disks by tag, and a managed checkpoint bucket, kept on purpose. Deletion runs in the installer API; the command waits until the cluster is terminated (or failed, exiting non-zero with the reason), printing terminated · releasing VM (still billing) meanwhile, so the API is not stopped while the VM is still running — --no-wait returns once deletion is recorded. If any of those steps fails, the cluster is left failed; running delete again resumes from the steps still recorded, and create refuses it unless it is a restore and the cloud shows the management VM gone. delete is refused while the installer API is still creating the cluster; once an installer API restart has interrupted that creation, delete tears down whatever it made. Each command accepts --json.

--json reports these fields:

Field	Meaning
id, name, state, releasing	Record identity, the Workload state, and whether the management VM is still being released
cloudProvider, region, zone, vmType	Where the management VM runs (null on the hosted API's own record)
k3sVersion, apiVersion, apiImage, chartVersion	The pinned control plane, the installed API and its chart
connected, kubernetesIdentity, connectionRevision	Whether the installer can reach the cluster, the observed kube-system UID, the connection revision
activeNodePoolCount	Node pools the record references; delete asks the hosted API for its pools instead
address	The management VM's public address
managementApiUrl	https://<cluster-id>.anycloud.sh, the anycloud api use target; an installer's once released
release	The installed API release: apiVersion, apiImage, chartVersion; null until it is installed
lastError	The last reconciliation error, or null

See Clusters for the end-to-end path from create to a running Service.

Back up and upgrade the hosted API

The API on the cluster backs its SQLite database up hourly and before every self-update: an online SQLite backup (writes continue; the copy is consistent) uploaded whole to anycloud-api/database.sqlite in the cluster's checkpoint bucket. History is the bucket's own object versioning — on for a managed bucket; on a bucket you supplied the hourly timer refuses to start — and says so in the API log — until versioning is enabled (aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled, gcloud storage buckets update gs://<name> --versioning, or az storage account blob-service-properties update --account-name <account> --enable-versioning true), then restart the API. The pre-release backup still runs without versioning and replaces the previous object, so until versioning is on exactly one copy exists — the latest pre-release one. Each backup also uploads the API's cipher key to anycloud-api/cipher.key, then writes anycloud-api/backup.json last, naming the installation, the backup time, the schema version and the database's SHA-256. Anyone who can read the bucket can decrypt every stored cloud credential and secret: keep it private, readable only by you, the installer and the API. The compute credential travels in the release values as a chart Secret on the cluster.

anycloud api upgrade [--image ghcr.io/anycloud-sh/api:<version>@sha256:<digest>]

With the hosted API active, anycloud api upgrade (alias update) asks it for this CLI's release (POST /v1/system-release) — any GitHub account the API admits, no installer, Docker or cloud credentials involved. The API pins the release tag to its published digest (or refuses with the version named when none is published), refuses an image that is not from ghcr.io/anycloud-sh/api, and applies the rule it applies to itself at startup: a target older than the version its database was last used by is refused, because that image would refuse to start. The serving API stamped that version when it started, so every target older than it is refused; going back means restoring the pre-upgrade backup. The API then backs its database up, records the upgrade durably, and patches its own HelmChart — the target image's chart, and image and revision in the release values — through its own in-cluster ServiceAccount, against the cluster record it wrote on start (the HelmChart must name this cluster, or the upgrade is refused). helm-controller applies the change: the chart's pre-upgrade guard refuses a revision that is not newer than the running one, and the Deployment is recreated with the new image while the API stays running. The command returns once the upgrade is recorded, printing the verdict as a plain line — Hosted API upgrade N started: <old> → <version>, already matches anycloud <version> or already runs <image>; nothing to do (unchanged), or Hosted API upgrade refused: <reason> on stderr — so scripts read it without a terminal. anycloud api status (GET /v1/system-release) shows the served version, the schema version and Upgrade N · <version> with one ✓/◌/✗ line: starting until the release's ready check Job succeeds, then running, or failed with the reason. The API answers from inside the cluster, so right after an upgrade the public URL may answer 503 for a few seconds while the edge reconnects; retry. These two calls accept a CLI one minor release apart; everything else needs an exact match. api upgrade upgrades the local API first, as it does without a hosted API.

Recover after losing the management VM

Restore is cluster create pointed at the lost cluster's checkpoint bucket:

anycloud cluster create prod --credentials aws-prod --region us-east-1 \
  --vm-type t3.large --checkpoint-bucket my-api-state

When the bucket holds a backup (anycloud-api/backup.json), create reuses the backup's cluster ID, so https://<cluster-id>.anycloud.sh keeps working. It refuses a backup written by a newer API; run anycloud update and retry. Before the new API first starts, it fills the API's empty state volume with the backup's cipher key and database. A database that does not match backup.json is refused because the backup was interrupted: in the bucket's version history, restore the anycloud-api/database.sqlite version whose SHA-256 the error names, then create again.

For a bucket Anycloud created, pass --checkpoint-bucket <cluster-id>. A restore asks the cloud for the cluster's management VMs by the cluster's tag (on AWS in every enabled region; on GCP and Azure anywhere in the project or subscription) and refuses while one exists, stopped or not. Restore with the lost cluster's cloud credential (the same account, project or subscription): the lookup cannot see another account. An installer that still holds the lost cluster replaces its record, but only once the cloud shows the old management VM is gone; while it still runs, re-running create with the cluster's own name is an ordinary resume. cluster delete refuses the lost cluster while its hosted API is unreachable, so it cannot delete the backup first; cluster forget never touches the bucket.

On its first start the restored API adopts the new cluster. It terminates the old node pool VMs by their native IDs and provisions replacements, then re-applies every Deployment and node pool Service from its database. Jobs that were running at the last backup are retried; a Job that finished after it runs again. Kubernetes state itself is not backed up.

Up to one backup interval of changes is lost, since backups run hourly and before every update. Make sure the old management VM is gone before you restore: two APIs must never write the same bucket.

Cluster authority

The hosted API manages node pools, Deployments, Services and its own release through the anycloud-api ServiceAccount the chart binds to a ClusterRole (charts/anycloud-api/templates/rbac.yaml): get, create, patch and delete Namespaces; get, list, watch, patch and delete Nodes; Pods, with get on pods/log for anycloud logs; Secrets, Services and ServiceAccounts; apps Deployments and DaemonSets (the pool loop watches Nodes, Pods, Secrets, ServiceAccounts, Deployments and NetworkPolicies); list, watch, create and patch networking.k8s.io NetworkPolicies (each node pool namespace's default egress policy, restored if it is deleted or edited); batch Jobs; node.k8s.io RuntimeClasses; helm.cattle.io HelmCharts and HelmChartConfigs (cluster add-ons and the API's own release); tokenreviews and selfsubjectaccessreviews. No credential for this authority is stored or exported — no kubeconfig, no token row; the only stored cluster credential is the installer's bootstrap-time ServiceAccount token, encrypted on the installer's cluster record and used by cluster create|status|delete alone. Every connection re-checks the recorded kube-system UID, verifies each permission the node pool and Service controllers need with a SelfSubjectAccessReview, and reads the server version — at least hourly and always when a node pool is created; a node pool is refused with a diagnostic naming the missing permission or the unsupported version rather than provisioning against a cluster the API cannot manage.

The cluster's management Node carries the standard control-plane taint node-role.kubernetes.io/control-plane=true:NoSchedule and the label anycloud.sh/cluster-management=true (create sets them). The API release and K3s's own add-ons tolerate it; node pool work never does, so node pool Pods cannot land on management capacity.

There is no command to register an existing Kubernetes control plane: the installation's cluster is the one anycloud cluster create creates, and the hosted API's record of it is written by the API itself.

Lifecycle states

The Workload states: provisioning, initializing and starting while it is created, failed (a failed create or delete; a failed delete's Last error: starts with Delete failed:), and terminated (with the "releasing" label while the management VM is still being released). Node pools are accepted only while the cluster is running. The record is removed only by anycloud cluster delete, which asks the hosted API what it still manages and is refused while any node pool exists — naming anycloud deployment delete <id> / anycloud terminate <id> for a pool that still has Deployments or Services, then anycloud node-pool delete <name>, then anycloud cluster delete <name> again, or by anycloud cluster forget when the hosted API is unreachable (see delete above). Each command accepts --json.