Skip to main content

Jobs

A Job is a finite container workload. Use one for training, batch inference, evaluation, data processing, builds, and any other task that should exit when its work is complete. On VM-backed providers, Anycloud provisions a VM, runs one container, records the result, and tears the VM down.

Use a Service instead when the process should stay online and receive HTTP traffic. Use a VM when you need an interactive shell and direct control of the provisioned host.

Jobs on a node pool​

A running node pool runs Jobs from an image, as a VM host does, without starting a VM for each one. The pool sets the cloud, region, VM type, disk and spot mode; a Job asks for its share of a Node:

anycloud job ghcr.io/example/train:latest --node-pool gpu --gpus 1 -- python train.py

--gpus is required on a GPU pool. --cpus and --memory size the Job's replica, --shm-size and --ipc host give it a memory-backed /dev/shm counted against --memory, and --secret puts a named Secret in its environment. Options the pool owns (--credentials, --region, --zone, --vm-type, --gpu-type, --spot, --disk-size, --disk-tier), --bind, --runtime and any other --ipc are refused with a message that names the pool.

Buckets work as on a VM host, on Azure storage, S3 or Cloud Storage: the whole --input-bucket is synced read-only into /mnt/input, and /mnt/output and /mnt/checkpoint are uploaded about every minute and when the command ends, to the Job's own <job-id>/ prefix in --output-bucket and --checkpoint-bucket. A retry restores /mnt/checkpoint from that prefix.

Each bucket's access is signed by its own credential: a saved Azure, AWS or GCP credential named with --input-storage-credentials, --output-storage-credentials or --checkpoint-storage-credentials, with its --*-storage-region, or else the node pool's credential when the pool is on Azure, AWS or GCP. So a Job on a Lambda pool can keep its input on Cloud Storage, its output on S3 and its checkpoints in Azure. Admission refuses a bucket on any other storage, such as R2, a bucket with no credential that can sign for it, a missing input or checkpoint bucket, an AWS credential with a session token, a GCP project without the Security Token Service API and an Azure account with Shared Key access disabled, and creates a missing output bucket. Each attempt gets short-lived credentials for its own prefixes only, never stored in the cluster, from the credentials admission selected, even after the pool is deleted.

Azure and S3 credentials last 36 hours, so a Job keeps running through an API outage of up to 18 hours. A Cloud Storage credential lasts an hour, so a Job with any bucket there stops and retries after an outage of about 30 minutes. Cloud Storage can't overwrite an object without deleting it, so its credential can also delete inside the Job's own prefix, but nowhere else.

A bucket outside the pool's cloud or region pays that storage's egress on every stage and every retry's re-stage: at list prices about $0.09/GB from S3, $0.087/GB from Azure and $0.12/GB from Cloud Storage, so a 500 GB input costs about $45–60 each time. Keep a large input in the pool's cloud and region. New Cloud Storage buckets keep each replaced object for 7 days as billed soft-deleted data, which multiplies a checkpoint bucket's cost when the checkpoint is rewritten every few minutes. Turn it off on an explicit checkpoint bucket with gcloud storage buckets update gs://BUCKET --clear-soft-delete.

On a spot node pool, a Job with no --checkpoint-bucket gets its own checkpoint bucket named after it, as a spot Job on a VM host does: anycloud status names it, a retry restores from it, and it is deleted with the Job, even after the pool is. It lives in the account of --checkpoint-storage-credentials, or of an Azure, AWS or GCP pool's credential, and on Cloud Storage keeps no soft-deleted copies; a Job on any other spot pool needs that option, and is refused without it. S3 and Cloud Storage bucket names are global, so a Job ID that another account or project already uses as a bucket name is refused: pass --checkpoint-bucket or choose a different --id. A bucket with the Job's ID that your own account already holds is reused, and deleted with the Job.

Anycloud runs each Job on a replica of a Deployment it manages for the pool, the image's digest and the Job's shape (--gpus, --cpus, --memory, --shm-size, --ipc, --secret), so Jobs with the same image and shape share replicas. A replica runs one Job at a time: it starts the image's entrypoint with the Job's command and environment in the image's working directory, as a VM host does. Replicas scale from zero with the queued Jobs and back to zero when idle. anycloud deployment list shows these Deployments with a (managed) label; they can't be updated or deleted, and each is removed after 24 hours with no Job.

A queued Job waits for a replica with no deadline: anycloud status <id> says why (no replica yet, a spend cap, or the replica's Pending reason, such as the pool's capacity). --timeout 30m fails the Job if no replica starts it within 30 minutes. A replica whose image the registry refuses (not found or unauthorized), or whose image the runner can't execute in, fails the queued Jobs with invalid; a pull stuck in back-off holds them until --timeout or terminate. anycloud node-pool status <pool> counts the pool's busy, idle and Pending replicas.

anycloud logs <id> shows the Job's output from when its replica started it; logs -f waits while the Job is queued, and the last 1 MiB is kept once the Job ends. anycloud exec <id> "<command>" runs a command in the Job's replica while the Job runs (see anycloud exec). wait, status, terminate, resubmit and cost work as for any Job; ssh doesn't.

Between Jobs, the replica resets: it stops every process the Job left, empties /mnt/output, /mnt/checkpoint, /tmp and /dev/shm, and restores the image's working directory to how it was when the replica started. The reset has limits:

  • Files a Job changes outside the image's working directory aren't restored for the next Job.
  • Files the restore copies back belong to the runner's user, not their original owners.
  • If the image's working directory has paths the runner's user can't read, the replica restarts after every Job instead, and Kubernetes' restart back-off (10 s, doubling up to 5 minutes) can leave it not ready for up to 5 minutes between Jobs.

anycloud api upgrade waits for the pool's running Jobs before it replaces the API, and starts no new ones meanwhile; --force stops them instead, and each retries after the upgrade.

Jobs on Kubernetes Deployments​

A running node pool runs persistent Kubernetes Deployments whose applications accept Jobs. Submit a Job that carries only its ID to an existing Deployment, which supplies image and placement:

anycloud job --deployment model-deployments --id request-123

In Python:

from anycloud import Client

with Client() as client:
    job = client.submit(deployment="model-deployments", workload_id="request-123")
    print(job.id, job.status().workload.state)

job.status().jobs_ahead reports the API-calculated number of queued Jobs ahead on that Deployment: 0 is first, and None applies outside a queued targeted Job. Use client.get_deployment("model-deployments") for readiness and the Jobs holding its slots; job.status().workload.claimed records whether this submission was ever picked up, including after a fast completion or automatic retry. See the Python queue-status example for the capacity comparison, lifecycle details, and refresh guidance.

The long-running application calls anycloud_workflows.current_deployment().next_job() when it can accept work. That call atomically starts one queued Job and returns a CurrentJob handle. Use job.id to load invocation data and job.complete(), job.error(message), job.invalid(message), or job.retry(message) to report the outcome. The original generated SelectedJob remains available as job.selected.

next_job(timeout=20) waits up to 20 seconds to discover a candidate; timeout=0 performs one immediate lookup. An attempted claim may resolve later within the separate request budget. None confirms that this call has no unresolved claim that could assign work later. Keep one Deployment context open across errors so it can recover an uncertain claim before looking for other work. See the Python SDK workflow example for a loop with a one-second delay and cooperative cancellation methods.

Targeted Jobs use the normal status, termination, and resubmission commands. In Python, reattach with client.get(job_id), then call job.status(), job.terminate(), or job.resubmit(). Resubmission requires completed cleanup. Pass timeout=30, wait=True to client.submit(deployment="model-deployments", ...) to wait for a durable first claim within 30 seconds of admission. Waiting for the Job to finish remains a separate job.wait() operation. With wait=False, the same deadline continues after the call returns; job.wait_for_claim() observes its result. Confirmed expiry means the submission was never claimed and cannot start later. Resubmission keeps the Job ID, advances its submission revision, and replaces or clears its deadline. Automatic retries retain the first-claim evidence. Their image, command, environment, resources, and Docker options come from the Deployment. Per-Job buckets, logs, exec, and SSH are unavailable because the Jobs share a long-running application process.

Submit a job​

anycloud job ghcr.io/acme/train:latest \
  --credentials my-aws \
  --gpu-type h100:8 \
  --spot \
  --gpus all \
  -- python train.py

The Python example waits up to one hour. A WorkloadWaitTimeout stops waiting without terminating the Job; see Wait and terminate.

Choose hardware by GPU name when the accelerator matters, or by exact VM type when the instance shape matters. See GPUs & VM Types for that decision and job reference for every option.

Package code in the image​

Build the application and its dependencies into a published image before submitting it:

anycloud job ghcr.io/acme/trainer:sha-4d82f7a --gpu-type h100:8

Anycloud resolves the image to a registry digest before pulling it, so a mutable tag always resolves to the content available for that workload. Tag images with a commit SHA or another immutable release identifier when reproducibility matters. See Container Images for build, cache, and registry guidance.

Lifecycle​

Job lifecycleHappy path
Wait
queued
Waiting for capacity or for a spend control to clear
Setup
provisioning
Create the VM
initializing
Install the host runtime
downloading
Pull the container image
syncingif used
Restore input or checkpoint data
starting
Start the container
Execute
running
The workload is executing
Preserve
finalizingif used
Save final artifacts
Result
completed
Exit code 0
or
errored
Non-zero exit
Return paths
Setup failureretrying⏳ queued
Spot preemptionrecovering⏳ queued
Terminal exits
failedRequired retries exhausted
invalidConfiguration rejected
terminatedStopped by a user

Infrastructure setup failures are retried up to 100 setup attempts. Failures during image download, data sync, or container startup can reuse an existing VM. When replacement is required, retries wait for the provider to confirm release of the old VM; waiting for cleanup does not consume setup attempts. Invalid configuration is not retried. A local API can have 50 workloads in the setup pipeline at once; that cap does not limit the number already running.

A job publishes its terminal outcome as soon as it is known; completedAt is that time. Resource teardown continues without adding another lifecycle event, and cleanedAt stays null until teardown succeeds; resubmission is blocked in between. When a teardown attempt fails, status names the reason as Cleanup pending: … (cleanupError in JSON) until an attempt succeeds. Compute spend follows each VM's own lifetime — from when it is created until its endedAt cutoff — independently of both workload timestamps.

Use status to see the current phase and its events, logs --follow to stream output from submission onward, and wait to continue only after the Job succeeds:

anycloud status <job-id> --verbose
anycloud logs <job-id> --follow
anycloud wait <job-id> && echo "training finished"

wait exits 0 when the Job completes and 1 when it fails, and prints one line per state change, so scripts and agents never parse status output. See anycloud wait. While a spot Job is retrying, status and wait show the attempt, when the next attempt is due, and whether capacity or quota is the cause.

Data and artifacts​

Attach object storage when the job needs datasets, durable results, or recoverable state:

Container pathAccessBehavior
/mnt/inputRead-onlyDownloaded before the container starts
/mnt/outputRead-writeUploaded about every 60 seconds and at exit
/mnt/checkpointRead-writeRestored at startup; uploaded during the run

See Buckets for configuration and sync behavior. An explicit checkpoint bucket works on both on-demand and spot Jobs. Spot Jobs without one keep the automatic workload-named checkpoint described below.

Spot recovery​

Spot capacity costs less but may be reclaimed at any point. With --spot, Anycloud detects preemption and first tries to restart the VM in place; otherwise it cleans up the lost VM, provisions a replacement, restores /mnt/checkpoint, and restarts the container.

from pathlib import Path
import json

checkpoint = Path("/mnt/checkpoint/state.json")
start_epoch = json.loads(checkpoint.read_text())["epoch"] if checkpoint.exists() else 0

for epoch in range(start_epoch, 100):
    train_one_epoch(epoch)
    checkpoint.write_text(json.dumps({"epoch": epoch + 1}))

Checkpoint changes sync about every 60 seconds, so checkpoint often and make startup idempotent. The automatic checkpoint bucket is deleted when a spot Job finishes. To retain checkpoint state across separate Jobs, create a bucket and pass it with --checkpoint-bucket.

anycloud status shows the checkpoint bucket, its mount, whether it is kept after the Job, and what has reached the bucket so far:

Checkpoint: 52 objects, 101.0 MB in the bucket, newest 40s ago

Compose jobs​

Sequence Jobs with ordinary Python. Wait for a preparation step to finish, then use its output bucket as the training Job's input:

from anycloud import Client

with Client() as client:
    prepared = client.submit(
        "ghcr.io/acme/prepare:latest",
        credential_name="my-aws",
        gpu="h100",
        cloud_config={"output_bucket": "prepared-training-data"},
    )
    prepared.wait()
    trained = client.submit(
        "ghcr.io/acme/train:latest",
        credential_name="my-aws",
        gpu="h100:8",
        cloud_config={"input_bucket": "prepared-training-data"},
    )
    trained.wait()

The preparation container writes to /mnt/output; training reads from /mnt/input. Use separate buckets or application prefixes for concurrent runs. A failed wait raises before the next submission. To run independent Jobs in parallel, submit them before waiting and handle partial failures in your application. The SDK has no JobGroup, submit_many(), or get_or_submit() helper.

Operate and debug​

anycloud job list --status running
anycloud logs <job-id>
anycloud exec <job-id> "nvidia-smi"
anycloud terminate <job-id>
anycloud resubmit <job-id>

exec and live logs require the workload's container to still be running. When an output bucket is attached, logs are also copied under .anycloud/logs/, preserving them after teardown. Exact lifecycle and operation syntax is in Workload operations.

Agent-operated jobs​

The CLI supports coding agents that run and monitor Jobs directly. Commands that print tables also provide structured output, and non-interactive workloads from Claude Code, Codex, Cursor, and Aider are tagged with the detected agent and session.

anycloud job list --agent codex --json
anycloud job list --session <session-id> --only-ids
anycloud status <job-id> --json
anycloud cost --agent codex --json

When an agent invokes list, job list, or cost, results default to that agent's own session. This is cooperative scoping for usability, not an access boundary; an explicit --session or --agent can select another scope.

Codex submissions use the native CODEX_THREAD_ID when available. Older Codex environments that expose only CODEX_BIN_PATH retain the codex-<parent-process-id> fallback. After upgrading, new native-thread scopes do not include earlier process-based sessions. Select that history explicitly:

anycloud job list --session codex-49152
anycloud cost --agent codex

Claude Code and Codex session titles appear in human-readable session output when available. Codex titles come from session_index.jsonl under $CODEX_HOME, which defaults to ~/.codex. Titles are captured when work is submitted; renaming a session affects subsequent submissions. Missing or unreadable titles keep the usual session identifier display.

Use anycloud docs for the documentation index and anycloud docs --all for the full Markdown corpus. Pair autonomous sessions with per-session Spend Controls.