Jobs

A Job is a finite container workload. Use one for training, batch inference, evaluation, data processing, builds, and any other task that should exit when its work is complete. On VM-backed providers, Anycloud provisions a VM, runs one container, records the result, and tears the VM down.

Use a Service instead when the process should stay online and receive HTTP traffic. Use a VM when you need an interactive shell and direct control of the provisioned host.

Jobs on Workers

A Ready cluster runs persistent Anycloud Workers, not direct per-Job Pods. Submit an identity-only Job to an existing Worker:

anycloud job --worker model-workers --id request-123

In Python, use client.submit(worker="model-workers", deployment_id="request-123"). job.status().jobs_ahead reports the API-calculated number of queued Jobs ahead on that Worker: 0 is first, and None applies outside a queued targeted Job. Use client.get_worker() for readiness and unused concurrency; only Job state "running" confirms pickup. See the Python queue-status example for the capacity comparison, lifecycle details, and refresh guidance.

The long-running application calls anycloud_workflows.current_worker().next_job() when it can accept work. That call atomically starts one queued Job and returns a CurrentJob handle. Use job.id to load invocation data and job.complete(), job.error(message), job.invalid(message), or job.retry(message) to report the outcome. The original generated SelectedJob remains available as job.selected.

next_job() returns None without waiting when no Job is available. Keep one Worker context open around the polling loop and sleep between empty polls. See the Python SDK workflow example for a loop with a one-second delay and cooperative cancellation methods.

Targeted Jobs use the normal status, termination, and resubmission commands. In Python, reattach with client.get(job_id), then call job.status(), job.terminate(), or job.resubmit(). Resubmission requires completed cleanup. Their image, command, environment, resources, and Docker options come from the Worker. Per-Job buckets, logs, exec, and SSH are unavailable because the Jobs share a long-running application process.

Submit a job

CLI

anycloud job ghcr.io/acme/train:latest \
  --credentials my-aws \
  --gpu-type h100:8 \
  --spot \
  --gpus all \
  -- python train.py

Python

from anycloud import Client

with Client() as client:
    job = client.submit(
        "ghcr.io/acme/train:latest",
        gpu="h100:8",
        command=["python", "train.py"],
        credential_name="my-aws",
        cloud_config={"spot": True},
    )
    job.wait(timeout_seconds=3600)

Choose hardware by GPU name when the accelerator matters, or by exact VM type when the instance shape matters. See GPUs & VM Types for that decision and job reference for every option.

Package code in the image

Build the application and its dependencies into a published image before submitting it:

anycloud job ghcr.io/acme/trainer:sha-4d82f7a --gpu-type h100:8

Anycloud resolves the image to a registry digest before pulling it, so a mutable tag always resolves to the content available for that deployment. Tag images with a commit SHA or another immutable release identifier when reproducibility matters. See Container Images for build, cache, and registry guidance.

Lifecycle

Job lifecycleHappy path

Wait

⏳queued

Waiting for capacity or for a spend control to clear

→

Setup

🏗️provisioning

Create the VM

↓

⚙️initializing

Install the host runtime

↓

📥downloading

Pull the container image

↓

💾syncingif used

Restore input or checkpoint data

↓

🚀starting

Start the container

→

Execute

⚡running

The workload is executing

exits→

Preserve

🏁finalizingif used

Save final artifacts

→

Result

✅completed

Exit code 0

or

🪲errored

Non-zero exit

Return paths

Setup failure→🛠️retrying↩⏳ queued

Spot preemption→🔁recovering↩⏳ queued

Terminal exits

❌failedRequired retries exhausted

🚫invalidConfiguration rejected

🪦terminatedStopped by a user

Infrastructure setup failures are retried up to 100 setup attempts. Failures during image download, data sync, or container startup can reuse an existing VM. When replacement is required, retries wait for the provider to confirm release of the old VM; waiting for cleanup does not consume setup attempts. Invalid configuration is not retried. A local API can have 50 deployments in the setup pipeline at once; that cap does not limit the number already running.

A job publishes its terminal outcome as soon as it is known; completedAt is that time. Resource teardown continues without adding another lifecycle event, and cleanedAt stays null until teardown succeeds; resubmission is blocked in between. Compute spend follows each VM's own lifetime — from when it is created until its endedAt cutoff — independently of both deployment timestamps.

Use status to see the current phase and its events:

anycloud status <job-id> --watch
anycloud status <job-id> --verbose

Data and artifacts

Attach object storage when the job needs datasets, durable results, or recoverable state:

Container path	Access	Behavior
/mnt/input	Read-only	Downloaded before the container starts
/mnt/output	Read-write	Uploaded about every 60 seconds and at exit
/mnt/checkpoint	Read-write	Restored at startup; uploaded during the run

See Buckets for configuration and sync behavior. An explicit checkpoint bucket works on both on-demand and spot Jobs. Spot Jobs without one keep the automatic deployment-named checkpoint described below.

Spot recovery

Spot capacity costs less but may be reclaimed at any point. With --spot, Anycloud detects preemption, cleans up the lost VM, provisions a replacement, restores /mnt/checkpoint, and restarts the container.

from pathlib import Path
import json

checkpoint = Path("/mnt/checkpoint/state.json")
start_epoch = json.loads(checkpoint.read_text())["epoch"] if checkpoint.exists() else 0

for epoch in range(start_epoch, 100):
    train_one_epoch(epoch)
    checkpoint.write_text(json.dumps({"epoch": epoch + 1}))

Checkpoint changes sync about every 60 seconds, so checkpoint often and make startup idempotent. The automatic checkpoint bucket is deleted when a spot Job finishes. To retain checkpoint state across separate Jobs, create a bucket and pass it with --checkpoint-bucket.

Compose jobs

Application orchestration can use the optional Client and Job workflows for individual submissions and waits, or call generated operations directly. The SDK does not provide client-side chaining, batching, or fan-out helpers.

Operate and debug

anycloud list --status running
anycloud logs <job-id>
anycloud exec <job-id> "nvidia-smi"
anycloud terminate <job-id>
anycloud resubmit <job-id>

exec and live logs require the job execution environment to still exist. When an output bucket is attached, logs are also copied under .anycloud/logs/, preserving them after teardown. Exact lifecycle and operation syntax is in Deployment operations.

Agent-operated jobs

The CLI supports coding agents that run and monitor Jobs directly. Commands that print tables also provide structured output, and non-interactive deployments from Claude Code, Codex, Cursor, and Aider are tagged with the detected agent and session.

anycloud list --agent codex --json
anycloud list --session <session-id> --only-ids
anycloud status <job-id> --json
anycloud cost --agent codex --json

When an agent invokes list or cost, results default to that agent's own session. This is cooperative scoping for usability, not an access boundary; an explicit --session or --agent can select another scope.

Use anycloud docs for the documentation index and anycloud docs --all for the full Markdown corpus. Pair autonomous sessions with per-session Spend Controls.