Clusters

An Anycloud installation has one Kubernetes cluster: a management VM in your cloud account running a pinned K3s control plane, with the Anycloud API installed on it as a Helm-managed release. The cluster is the hosted API's home and the place where node pools — cloud VM pools joined to it — run Kubernetes Deployments and node pool Services.

This guide walks the path the live AWS runs prove, in order: create the cluster, point the CLI at the API it hosts, add a node pool, run a Service on it, update the hosted API, and take everything down. Each command's output below is what the CLI prints; ids and addresses are placeholders.

Terminology: the cluster is the installation's Kubernetes cluster; a node pool is a set of cloud VMs joined to it; a Deployment record is the Anycloud resource that keeps an application running for targeted Jobs, and the native Kubernetes Deployment is its runtime object. Workload is a Job, Service or VM.

Before you start

A cloud credential on the local API for the provider that will run the management VM: AWS, GCP or Azure. Lambda does not support cluster creation.
anycloud login, because the hosted API requires GitHub authentication.
The local API running and active — it becomes the installer: the cluster record, its ledger and the cluster's credential live there.

anycloud api start
anycloud api use local
anycloud credentials new my-aws

1. Create the cluster

anycloud cluster create prod \
  --credentials my-aws \
  --region us-east-2 \
  --vm-type t3.medium

create records the cluster and returns, like submitting a Job. The API then creates the management VM (provisioning), starts Kubernetes on it and connects (initializing), and installs the API release (starting); the cluster is running once the API answers at its URL:

✓ Cluster prod creation started.
ID: <cluster-id>
Watch it with: anycloud cluster status prod --watch

anycloud cluster status prod --watch

[12:00:00] provisioning
[12:01:10] initializing
[12:03:40] starting
[12:05:02] running

⚡  running

Name        prod
ID          <cluster-id>
Cloud       AWS / us-east-2
VM          t3.medium
Address     <management-vm-ip>
Kubernetes  K3s <k3s-version>
API         <version>
URL         https://<cluster-id>.anycloud.sh
Node pools  0
Age         5m
Next: anycloud api use https://<cluster-id>.anycloud.sh

A live AWS creation takes a few minutes. If it fails, the cluster is failed with the reason in Last error:; re-running the same create resumes where it stopped. Pass --checkpoint-bucket <name> to back the hosted API's database up to a bucket you own; without it a managed bucket is created, and cluster delete deletes it with every backup (a restore records its bucket as user-supplied, so delete then keeps it). A bucket that already holds a backup (anycloud-api/backup.json) makes create a restore that keeps the cluster id and URL. See anycloud cluster for every flag.

2. Point the CLI at the hosted API

anycloud api use https://<cluster-id>.anycloud.sh

From here every command runs against the API on the cluster. Unless it was restored from a backup, it starts with an empty database: add the credentials node pools will use, and share the API with teammates if you want to (Hosted APIs).

anycloud credentials new pool-aws

Switch back to the installer at any time with anycloud api use local; the cluster record is only visible there (anycloud cluster list).

3. Add a node pool

A node pool is a fixed-size, cloud-specific pool of VMs (--min-vms must equal --max-vms), bought with its own credential and joined to the cluster with short-lived per-VM tokens. Its VMs carry no SSH key or cloud credential and accept no inbound traffic until the pool hosts a Service; then they admit TCP 80 from Cloudflare only:

anycloud node-pool create cpu \
  --credentials pool-aws \
  --region us-east-2 \
  --vm-type t3.large \
  --min-vms 1 \
  --max-vms 1

   ✓ Node pool cpu creation started.

🏗️  provisioning

   ID        <node-pool-id>
   Name      cpu
   Cluster   <cluster-id>
   Cloud     AWS / us-east-2
   VM types  t3.large
   VMs       0/1 ready, 0 starting, 0 draining

The pool becomes running once every VM has joined as a Node (a GPU pool also waits for the NVIDIA add-on to advertise the catalog's device count):

anycloud node-pool status cpu

⚡  running

   ID        <node-pool-id>
   Name      cpu
   Cluster   <cluster-id>
   Cloud     AWS / us-east-2
   VM types  t3.large
   VMs       1/1 ready, 0 starting, 0 draining
   Live VMs  t3.large in us-east-2 (zone us-east-2a)

anycloud node-pool list shows every pool; a failed pool carries the reason in Last error:. GPU node pools, add-ons and deletion rules are in anycloud node-pool.

4. Run a Service on the node pool

A Service placed with --node-pool runs as one replica on that pool. Its Anycloud identity and configuration are the same as a VM-backed Service; the native Deployment, Service and Secrets are runtime details. It is reachable at https://<id>.anycloud.sh, which the CLI prints as URL::

anycloud service ghcr.io/example/api:latest --node-pool cpu --id web

✓ Service workload created!

ID: web
URL: https://web.anycloud.sh

anycloud status web

status shows scheduling and startup blocks while the Pod is placed, then running. Terminate it like any Service:

anycloud terminate web

Deployments that accept targeted Jobs work the same way: anycloud deployment create <name> <image> --node-pool cpu, then anycloud job … --deployment <name>; see anycloud deployment.

5. Upgrade the hosted API

When a new CLI release is installed, upgrade the API it talks to. Against the cluster-hosted API the command asks the API for this CLI's release, backs the database up to the checkpoint bucket and patches the API's own Helm release:

anycloud update
anycloud api upgrade

✓ Hosted API upgrade 1 started: <old-version> → <version>.

An API already on that release answers Hosted API <url> already matches anycloud <version>.; a refused upgrade prints Hosted API upgrade refused: <reason> and exits 1. The API stays running while it is replaced. Follow the upgrade:

anycloud api status --watch

⚡  running

   URL      https://<cluster-id>.anycloud.sh
   Source   ~/.anycloud/api-url
   Version  <version>
   Schema   <version>

   Upgrade 1 · <version>
   ✓ api  running · 1/1 ready

The upgrade line reads ◌ starting until the new release's ready check passes, then ✓ running, or ✗ failed with the reason. --watch prints each change until then; without it, api status reads once.

See Hosted APIs for backups, --image, and why an older image is refused.

6. Take it down

Teardown runs top-down against the hosted API, then the cluster is deleted from the installer. cluster delete never deletes node pools: it asks the hosted API what it still manages and refuses while anything remains, naming the next step.

anycloud api use https://<cluster-id>.anycloud.sh
anycloud terminate web
anycloud node-pool delete cpu
anycloud api use local
anycloud cluster delete prod

Run out of order and the refusal tells you what is left:

Failed to delete cluster: Node pool cpu still has Deployments or Services. Against the hosted API (https://<cluster-id>.anycloud.sh): run anycloud deployment delete <id> / anycloud terminate <id>, then anycloud node-pool delete <name>, then anycloud cluster delete prod again.

Failed to delete cluster: Node pool cpu still exists. Against the hosted API (https://<cluster-id>.anycloud.sh): run anycloud node-pool delete <name>, then anycloud cluster delete prod again.

A clean deletion releases the management VM, deletes the managed bucket and the DNS record, and ends with:

✓ Cluster prod deleted.

If the hosted API is unreachable, delete is refused: restore it (below), then delete. Without a restorable backup, delete names the VMs to terminate by hand; then anycloud cluster forget drops the record and its DNS record and names what else is left. If deletion fails part way, the cluster is failed with the reason in Last error:; running delete again finishes it.

Recover a lost cluster

If the management VM is lost, run the same anycloud cluster create with --checkpoint-bucket set to the cluster's bucket (its id for a managed bucket). The restored API keeps the cluster id and URL, replaces the node pool VMs, and re-applies Deployments and Services; up to an hour of changes is lost, and the hourly backup needs bucket versioning (see Back up and restore). Restore and cluster forget are both refused while the cloud still shows the lost management VM. See Recover after losing the management VM.

What runs where

Concern	Where it lives
Cluster record, creation ledger, cluster credential	The installer API (`anycloud cluster list` there)
Hosted API database and encryption key	The `anycloud-api-state` volume on the cluster; hourly backups in the checkpoint bucket, read only by a restore
Node pools, Deployments, Services and their credentials	The hosted API
Authority over the cluster	The hosted API's own ServiceAccount; nothing is stored or exported — see Cluster authority
Public entry	`https://<cluster-id>.anycloud.sh` (Cloudflare → management VM → Traefik → API)

Before you start​

1. Create the cluster​

2. Point the CLI at the hosted API​

3. Add a node pool​

4. Run a Service on the node pool​

5. Upgrade the hosted API​

6. Take it down​

Recover a lost cluster​

What runs where​