Skip to main content

anycloud node-pool

Node pools are cloud-specific VM pools inside the installation's cluster: each node pool owns its own compute credential, region, optional zone, and VM type, and Anycloud provisions its VMs, joins them to the cluster, and releases them. Deleting a node pool never deletes the cluster or a sibling node pool. Node pools are created through the hosted API (anycloud api use https://<cluster-id>.anycloud.sh) after the compute credential has been added there with anycloud credentials new. The end-to-end path from anycloud cluster create to a Service running on a node pool is in Clusters.

Create a node pool​

anycloud node-pool create <name> \
  [--cluster <id-or-name>] \
  --credentials <name> \
  --region <region> \
  [--zone <zone>] \
  --vm-type <type> \
  --min-vms <count> \
  --max-vms <count>

Pass --wait to wait until the node pool is Ready, with the exit codes of anycloud wait. A pool that reports failed ends the wait with exit 1. With --json, the first line is the node pool, then the wait records.

Creation is refused with NODE_POOLS_UNAVAILABLE until the cluster is ready. --cluster is optional and must resolve to the cluster record when given. The credential, region, zone, and VM type are fixed for the node pool's whole life. --zone must belong to --region (us-east-1a, us-central1-a) or be a single digit on Azure; other providers accept no zone. --min-vms and --max-vms are both required, must be at least one, and must be equal. Node pools are fixed-size; pool scale-out is not supported yet. The credential's cloud must support isolated capacity networking: AWS, Azure, GCP, and Lambda. On Lambda, a node pool VM's own firewall ruleset opens nothing, but TCP 22 and ICMP stay reachable through the account's global firewall rules, which apply to every instance. In us-south-1, which has no firewall rulesets, Lambda rejects the ruleset, so no VM launches and the node pool parks with that error. The cluster must run Kubernetes v1.35 through v1.38, the window the pinned agent supports.

Creation returns after durable intent is stored. Anycloud then provisions each VM with the node pool's own credential in an outbound-only network scope (no public ingress, no peer rules, no SSH key, no cloud credential on the VM), and the VM downloads a checksum-pinned K3s agent and joins the cluster with a short-lived, per-VM bootstrap token pinned to the cluster's certificate authority. The token is removed as soon as the Node object exists, before it is Ready, and a retried attempt replaces its previous token rather than adding one. Only x86_64 VM types are supported: the agent refuses to start on any other architecture, and a Lambda node pool on an arm64 type (such as gpu_1x_gh200) launches no VM and parks with that error. Each VM is recorded like a Job or Service VM, with the node pool as its only owner, and is billed under the node pool's resource scope. The cluster advertises its public address, so an agent's supervisor tunnel to it works from any cloud, and the cluster reads pool kubelets through that tunnel — no inbound port into a pool VM is ever needed. Agents register their public address as the Node external IP and the cluster's flannel uses external IPs, so Pod-to-Pod traffic between the cluster and a pool in another cloud would need UDP 8472 open between their public addresses; Jobs send no Pod traffic across nodes, and a node pool Service's replica runs on the pool's ingress entry Node so its Traefik reaches it locally, so nothing has to be opened for them. An agent that crashed before its Node registered rejoins on restart: its kubelet certificates are reissued from the retained node password on every start.

Node pool Nodes carry the labels anycloud.sh/cluster, anycloud.sh/node-pool, anycloud.sh/provider, and anycloud.sh/attempt — plus anycloud.sh/accelerator=nvidia on NVIDIA GPU VM types — and the taint anycloud.sh/node-pool=<id>:NoSchedule. Work that Anycloud schedules onto a node pool must select exactly that node pool and tolerate only its taint; a missing or deleted node pool leaves such work Pending rather than moving it to a sibling node pool or the management Node. This constrains scheduling only and does not make node pools independent security or failure domains.

A node pool uses Workload states: it is provisioning until it becomes running, when every VM's Node is Ready and, for GPU VM types, advertises the catalog's physical GPU count. A VM whose Node never appears within ten minutes, or stays NotReady for fifteen, is replaced — at most three times in a row (a replacement only counts as recovered once the replacement VM itself is Ready) before the node pool is parked failed with a diagnostic naming the join preconditions to check. Provider failures back off exponentially (one minute doubling to thirty) and park the node pool after five consecutive failures. While it backs off, the node pool keeps its state and reports the failure in its last error, which clears once a new VM launches. Any other unexpected error in a pass, such as a cluster API timeout, also keeps the node pool's state and is reported in its last error until the node pool is next ready. A parked node pool is failed and provisions nothing until it is deleted and recreated. A Node that claims the node pool — by label or by the Node name Anycloud issued — but does not match a VM the node pool provisioned marks the node pool failed and is never counted; a Node left behind by a VM the node pool already released is removed.

Cluster add-ons​

Some cluster components are installed once per cluster and shared by every node pool on it. The first is the NVIDIA stack: the nvidia RuntimeClass and the NVIDIA device plugin DaemonSet (kube-system/anycloud-nvidia-device-plugin, image pinned by digest). The hosted API installs them through its own cluster authority the first time a GPU node pool is reconciled, before any VM is bought, and re-checks them on every later GPU node pool pass, after that node pool's existing VMs have been observed: an object it owns is left alone while unchanged, rewritten if something else edited it, and recreated if a field the API server treats as immutable differs. A RuntimeClass that K3s installed itself with the nvidia handler and no overrides is used as is. Any other object with the same name that Anycloud does not own marks the GPU node pool failed with a diagnostic naming the object and the kubectl delete that removes it; Anycloud never adopts or replaces foreign objects, and an add-on failure never blocks releasing or replacing the node pool's existing VMs. A write the cluster's RBAC denies fails the node pool with a diagnostic naming the missing verb and resource. CPU node pools need no add-on and touch none of these objects, and a cluster with no GPU node pool is not re-checked.

Add-ons are never uninstalled. They are inert without GPU Nodes (the DaemonSet selects none), and anycloud cluster delete removes the control plane they live on. To uninstall by hand before that, delete RuntimeClass nvidia and DaemonSet kube-system/anycloud-nvidia-device-plugin after the last GPU node pool is deleted.

The DaemonSet selects only Nodes labelled anycloud.sh/cluster=<cluster-id> and anycloud.sh/accelerator=nvidia, tolerates node pool taints, and does not tolerate the management taint, so it never runs on the management Node or on CPU node pools. GPU node pool Nodes join with the accelerator label; a GPU Node that joined before this release without it is labelled by the next reconciliation pass. The VM image must provide the NVIDIA driver and container runtime; Anycloud selects a CUDA image for NVIDIA VM types.

A GPU node pool becomes running only when each Node advertises exactly the number of nvidia.com/gpu devices the catalog attributes to its VM type as allocatable. Allocatable is the advertised total, not free devices: Nodes whose GPUs are all held by running Jobs remain ready capacity. A Ready Node that advertises zero devices has ten minutes — from its VM's creation, or from the pass that first saw the devices missing when a plugin restart drops the count on a Node that was already ready — before the node pool is marked failed with a diagnostic that names the add-on objects and their state; it counts as starting meanwhile. A Node that advertises a different non-zero count than the catalog marks the node pool failed with a device-count mismatch diagnostic and is never counted; refresh the catalog or choose a VM type whose physical device count matches. Work scheduled onto a GPU node pool — a Deployment Pod or a Service replica — must request whole nvidia.com/gpu devices — the larger of its biggest init container and its regular containers' total, no more than one Node advertises — and run under the nvidia RuntimeClass; work on a CPU node pool must request none.

Ingress entry​

A node pool that hosts a Service elects one ingress entry: the Ready VM with the lowest attempt id, labelled anycloud.sh/ingress-entry=<pool-id> on its Node and shown as ✓ ingress <public IP> in node-pool status. The entry keeps running ordinary workloads; it is a role, not a dedicated VM. If its Node stays NotReady for two minutes and another VM's Node is Ready, that VM takes over and the label moves; with no Ready replacement the entry stays until its VM is replaced. On the first Service, the pool's network additionally admits TCP 80 from Cloudflare's published edge ranges only — the same rule VM-host Services carry. The rule is applied to the pool's network as a whole, so every VM in the pool accepts that traffic from Cloudflare, while only the entry Node runs anything on the port; it is never removed. AnyCloud then installs the pool's ingress release — a pinned Traefik HelmChart installed through the same cluster add-on mechanism as the NVIDIA stack (its install Job runs on the management Node like the API release's) — as a DaemonSet on that pool's entry Node only, on host port 80, tolerating that pool's taint and never the management Node's, watching only routes labelled anycloud.sh/ingress-pool=<pool-id>, so a request to one pool's entry is never served by another pool's Service. Deleting the node pool removes its release. A Service on the pool is published at https://<id>.anycloud.sh pointing at the entry IP (see Public URL). Jobs-only node pools elect no entry, open no port and install nothing.

node-pool status shows the release, read from the cluster at request time, on one line: ✓ ingress <IP> once the release is installed and running on a Ready entry VM, ✗ ingress failed: <error>, or ◌ ingress with the first thing it is waiting for: waiting for an entry VM (none elected yet), pending (no ingress release on the cluster yet, or why the cluster could not be read), installing (the release is installing), entry VM not ready (the entry Node is not Ready), or starting (ingress replicas not all ready). With --json the full view is ingress: { entryIp, release }, where release is null for a pool that hosts no Service and otherwise { state, helmJob, daemonSet, lastEvent }: state is absent (no HelmChart on the cluster), installing, installed or failed; helmJob is { phase, message } with the install container's non-zero exit as message, or null before the HelmChart exists; daemonSet is { desired, ready, entryNodeReady } once Helm created it, else null; routes counts the Services the pool's Traefik serves; lastEvent is the newest event on the install Job and its Pods (or, once the DaemonSet exists, in the release namespace) as reason: message, or a cluster unavailable: … note when the cluster could not be read. ingressEntryIp remains as a shortcut for ingress.entryIp. The generated Python SDK models these as nested objects of the node pool summary.

Inspect and delete a node pool​

anycloud node-pool list [--cluster <id-or-name>]
anycloud node-pool status <id-or-name>
anycloud node-pool delete <id-or-name>

status and delete accept the node pool ID or its name. Status reports the state, the last reconciliation error, and ready, starting, and draining VM counts. It does not expose Kubernetes credentials, endpoints, Pods, or raw Node objects.

Deletion is rejected while Deployments or Workloads target the node pool (see anycloud deployment create --node-pool), including a terminated Service whose native objects have not yet been confirmed removed. That cleanup runs through the hosted API's own cluster authority: if the hosted API is down, or its chart-bound ClusterRole no longer holds the permissions above, the Service stays uncleaned, the node pool cannot begin deletion, and the cluster cannot be destroyed either. Restore the hosted API (anycloud api status shows its latest upgrade) or the chart's RBAC and cleanup resumes on its own; there is no forced cleanup path in this release. The reference check runs inside the deletion request itself, so a Deployment or Service admitted at the same moment either lands before the request and blocks it or finds the node pool already terminated and is refused. Once nothing references the node pool, deletion also removes the namespace the node pool owns on the cluster (Deployment Pods run there; each Service owns its own namespace), re-checking references immediately before; a namespace of that name Anycloud does not own is left in place and recorded in the node pool's last error. Otherwise deletion is durable and asynchronous: the node pool moves to terminated · releasing VM (still billing) (releasing: true in --json), Anycloud cordons and removes its Nodes, terminates its VMs, waits for the provider to confirm each release, removes its isolated network, and then drops the releasing label. An uncertain provider observation keeps the VM attempt open and the node pool releasing. The cluster and sibling node pools stay usable throughout. terminate and rm are aliases for delete; list accepts ls and get, and status accepts describe. Each command accepts --json.