Skip to main content

anycloud node-pool

Node pools are cloud-specific VM pools inside the installation's cluster: each node pool owns its own compute credential, region, optional zone, and VM type, and Anycloud provisions its VMs, joins them to the cluster, and releases them. Deleting a node pool never deletes the cluster or a sibling node pool. Node pools are created through the hosted API (anycloud api use https://<cluster-id>.anycloud.sh) after the compute credential has been added there with anycloud credentials new.

Create a node pool​

anycloud node-pool create <name> \
  [--cluster <id-or-name>] \
  --credentials <name> \
  --region <region> \
  [--zone <zone>] \
  --vm-type <type> \
  --min-vms <count> \
  --max-vms <count>

Creation is refused with NODE_POOLS_UNAVAILABLE until the cluster is ready. --cluster is optional and must resolve to the cluster record when given. The credential, region, zone, and VM type are fixed for the node pool's whole life. --zone must belong to --region (us-east-1a, us-central1-a) or be a single digit on Azure; other providers accept no zone. --min-vms and --max-vms must be equal and at least one. The credential's cloud must support isolated capacity networking (AWS, Azure, and GCP today). The cluster must run Kubernetes v1.35 through v1.38, the window the pinned agent supports.

Creation returns after durable intent is stored. Anycloud then provisions each VM with the node pool's own credential in an outbound-only network scope (no public ingress, no peer rules, no SSH key, no cloud credential on the VM), and the VM downloads a checksum-pinned K3s agent and joins the cluster with a short-lived, per-VM bootstrap token pinned to the cluster's certificate authority. The token is removed as soon as the Node object exists, before it is Ready, and a retried attempt replaces its previous token rather than adding one. Only x86_64 VM types are supported: the agent refuses to start on any other architecture. Each VM is one attempt in the same ledger as Job and Service VMs, with the node pool as its only owner, and is billed under the node pool's resource scope. The cluster advertises its public address, so an agent's supervisor tunnel to it works from any cloud, and the cluster reads pool kubelets through that tunnel — no inbound port into a pool VM is ever needed. Agents register their public address as the Node external IP and the cluster's flannel uses external IPs, so Pod-to-Pod traffic between the cluster and a pool in another cloud would need UDP 8472 open between their public addresses; Jobs and node-pool Services do not send Pod traffic across nodes today, so nothing has to be opened for them. An agent that crashed before its Node registered rejoins on restart: its kubelet certificates are reissued from the retained node password on every start.

Node pool Nodes carry the labels anycloud.sh/cluster, anycloud.sh/node-pool, anycloud.sh/provider, and anycloud.sh/attempt — plus anycloud.sh/accelerator=nvidia on NVIDIA GPU VM types — and the taint anycloud.sh/node-pool=<id>:NoSchedule. Work that Anycloud schedules onto a node pool must select exactly that node pool and tolerate only its taint; a missing or deleted node pool leaves such work Pending rather than moving it to a sibling node pool or the management Node. This constrains scheduling only and does not make node pools independent security or failure domains.

The node pool becomes ready when every VM's Node is Ready and, for GPU VM types, advertises the catalog's physical GPU count. A VM whose Node never appears within ten minutes, or stays NotReady for fifteen, is replaced — at most three times in a row (a replacement only counts as recovered once the replacement VM itself is Ready) before the node pool is parked failed with a diagnostic naming the join preconditions to check. Provider failures back off exponentially (one minute doubling to thirty) and park the node pool after five consecutive failures. A parked node pool provisions nothing until it is deleted and recreated. A Node that claims the node pool — by label or by the Node name Anycloud issued — but does not match a VM the node pool provisioned marks the node pool failed and is never counted; a Node left behind by a VM the node pool already released is removed.

Cluster add-ons​

Some cluster components are installed once per cluster and shared by every node pool on it. The first is the NVIDIA stack: the nvidia RuntimeClass and the NVIDIA device plugin DaemonSet (kube-system/anycloud-nvidia-device-plugin, image pinned by digest). Anycloud installs them through the registered connection the first time a GPU node pool is reconciled, before any VM is bought, and re-checks them on every later GPU node pool pass, after that node pool's existing VMs have been observed: an object it owns is left alone while unchanged, rewritten if something else edited it, and recreated if a field the API server treats as immutable differs. A RuntimeClass that K3s installed itself with the nvidia handler and no overrides is used as is. Any other object with the same name that Anycloud does not own marks the GPU node pool failed with a diagnostic naming the object and the kubectl delete that removes it; Anycloud never adopts or replaces foreign objects, and an add-on failure never blocks releasing or replacing the node pool's existing VMs. A write the cluster's RBAC denies fails the node pool with a diagnostic naming the missing verb and resource. CPU node pools need no add-on and touch none of these objects, and a cluster with no GPU node pool is not re-checked.

Add-ons are never uninstalled. Deleting the cluster forgets its connection, so nothing could remove them, and the objects are inert without GPU Nodes (the DaemonSet selects none). When the same control plane — the same kube-system namespace UID — is registered again, Anycloud adopts the add-on objects the destroyed cluster installed and relabels them for the new record; objects labelled for any other cluster remain foreign. To uninstall by hand, delete RuntimeClass nvidia and DaemonSet kube-system/anycloud-nvidia-device-plugin after the last GPU node pool is deleted.

The DaemonSet selects only Nodes labelled anycloud.sh/cluster=<cluster-id> and anycloud.sh/accelerator=nvidia, tolerates node pool taints, and does not tolerate the management taint, so it never runs on the management Node or on CPU node pools. GPU node pool Nodes join with the accelerator label; a GPU Node that joined before this release without it is labelled by the next reconciliation pass. The VM image must provide the NVIDIA driver and container runtime; Anycloud selects a CUDA image for NVIDIA VM types.

A GPU node pool becomes ready only when each Node advertises exactly the number of nvidia.com/gpu devices the catalog attributes to its VM type as allocatable. Allocatable is the advertised total, not free devices: Nodes whose GPUs are all held by running Jobs remain ready capacity. A Ready Node that advertises zero devices has ten minutes — from its VM's creation, or from the pass that first saw the devices missing when a plugin restart drops the count on a Node that was already ready — before the node pool is marked failed with a diagnostic that names the add-on objects and their state; it counts as starting meanwhile. A Node that advertises a different non-zero count than the catalog marks the node pool failed with a device-count mismatch diagnostic and is never counted; refresh the catalog or choose a VM type whose physical device count matches. Work scheduled onto a GPU node pool must request whole nvidia.com/gpu devices — the larger of its biggest init container and its regular containers' total, no more than one Node advertises — and run under the nvidia RuntimeClass; work on a CPU node pool must request none.

Ingress entry (proposed behavior)​

A node pool that hosts a Service elects one ingress entry: the Ready VM with the lowest attempt id, labelled anycloud.sh/ingress-entry=<pool-id> on its Node and shown as Ingress entry: <public IP> in node-pool status. The entry keeps running ordinary workloads; it is a role, not a dedicated VM. If its Node stays NotReady for two minutes and another VM's Node is Ready, that VM takes over and the label moves; with no Ready replacement the entry stays until its VM is replaced. On the first Service, the pool's network additionally admits TCP 80 from Cloudflare's published edge ranges only — the same rule VM-host Services carry. The rule is applied to the pool's network as a whole, so every VM in the pool accepts that traffic from Cloudflare, while only the entry Node runs anything on the port; it is never removed. AnyCloud then installs the pool's ingress release — a pinned Traefik HelmChart installed through the same cluster add-on mechanism as the NVIDIA stack (its install Job runs on the management Node like the API release's) — as a DaemonSet on that pool's entry Node only, on host port 80, tolerating that pool's taint and never the management Node's, watching only routes labelled anycloud.sh/ingress-pool=<pool-id>, so a request to one pool's entry is never served by another pool's Service. Deleting the node pool removes its release. Publishing a Service's https://<id>.anycloud.sh record at the entry IP is the next step and is not yet available. Jobs-only node pools elect no entry, open no port and install nothing.

node-pool status shows the release's progress on two lines — Ingress entry: <IP> — release installing|installed|failed (helm job <phase>[: <exit>]) and Ingress Traefik: <ready>/<desired> ready, entry Node Ready|not Ready — last event: … — read from the cluster at request time. With --json the same view is ingress: { entryIp, release }, where release is null for a pool that hosts no Service and otherwise { state, helmJob, daemonSet, lastEvent }: state is absent (no HelmChart on the cluster), installing, installed or failed; helmJob is { phase, message } with the install container's non-zero exit as message, or null before the HelmChart exists; daemonSet is { desired, ready, entryNodeReady } once Helm created it, else null; lastEvent is the newest event on the install Job and its Pods (or, once the DaemonSet exists, in the release namespace) as reason: message, or a cluster unavailable: … note when the cluster could not be read. ingressEntryIp remains as a shortcut for ingress.entryIp. The generated Python SDK models these as nested objects of the node pool summary.

Inspect and delete a node pool​

anycloud node-pool list [--cluster <id-or-name>]
anycloud node-pool status <id-or-name>
anycloud node-pool delete <id-or-name>

status and delete accept the node pool ID or its name. Status reports the lifecycle state, the last reconciliation error, and ready, starting, and draining VM counts. It does not expose Kubernetes credentials, endpoints, Pods, or raw Node objects.

Deletion is rejected while Deployments or Workloads target the node pool (see anycloud deployment create --node-pool), including a terminated Service whose native objects have not yet been confirmed removed. That cleanup needs the cluster connection: if the cluster is unreachable or its identity no longer holds the permissions above, the Service stays uncleaned, the node pool cannot begin deletion, and the cluster cannot be deleted either. Restore the connection (re-register the cluster's kubeconfig with the required permissions) and cleanup resumes on its own; there is no forced cleanup path in this release. The reference check runs inside the deletion request itself, so a Deployment or Service admitted at the same moment either lands before the request and blocks it or finds the node pool already deleting and is refused. Once nothing references the node pool, deletion also removes the namespace the node pool owns on the cluster (Deployment handlers run there; each Service owns its own namespace), re-checking references immediately before; a namespace of that name Anycloud does not own is left in place and recorded in the node pool's last error. Otherwise deletion is durable and asynchronous: the node pool moves to deleting, Anycloud cordons and removes its Nodes, terminates its VMs, waits for the provider to confirm each release, removes its isolated network, and then marks the node pool deleted. An uncertain provider observation keeps the VM attempt open and the node pool deleting. The cluster and sibling node pools stay usable throughout. terminate and rm are aliases for delete. Each command accepts --json.