Skip to main content

Jobs

A Job is a finite container workload. Use one for training, batch inference, evaluation, data processing, builds, and any other task that should exit when its work is complete. On VM-backed providers, Anycloud provisions a VM, runs one container, records the result, and tears the VM down.

Use a Service instead when the process should stay online and receive HTTP traffic. Use a VM when you need an interactive shell and direct control of the provisioned host.

Jobs on Workers​

A Ready cluster runs persistent Anycloud Workers, not direct per-Job Pods. Submit an identity-only Job to an existing Worker:

anycloud job --worker model-workers --id request-123

In Python:

from anycloud import Client

with Client() as client:
    job = client.submit(worker="model-workers", deployment_id="request-123")
    print(job.id, job.status().deployment.state)

job.status().jobs_ahead reports the API-calculated number of queued Jobs ahead on that Worker: 0 is first, and None applies outside a queued targeted Job. Use client.get_worker("model-workers") for readiness and unused concurrency; job.status().deployment.claimed records whether this submission was ever picked up, including after a fast completion or automatic retry. See the Python queue-status example for the capacity comparison, lifecycle details, and refresh guidance.

The long-running application calls anycloud_workflows.current_worker().next_job() when it can accept work. That call atomically starts one queued Job and returns a CurrentJob handle. Use job.id to load invocation data and job.complete(), job.error(message), job.invalid(message), or job.retry(message) to report the outcome. The original generated SelectedJob remains available as job.selected.

next_job(timeout=20) waits up to 20 seconds to discover a candidate; timeout=0 performs one immediate lookup. An attempted claim may resolve later within the separate request budget. None confirms that this call has no unresolved claim that could assign work later. Keep one Worker context open across errors so it can recover an uncertain claim before looking for other work. See the Python SDK workflow example for a loop with a one-second delay and cooperative cancellation methods.

Targeted Jobs use the normal status, termination, and resubmission commands. In Python, reattach with client.get(job_id), then call job.status(), job.terminate(), or job.resubmit(). Resubmission requires completed cleanup. Pass timeout=30, wait=True to client.submit(worker="model-workers", ...) to wait for a durable first claim within 30 seconds of admission. Handler completion remains a separate job.wait() operation. With wait=False, the same deadline continues after the call returns; job.wait_for_claim() observes its result. Confirmed expiry means the submission was never claimed and cannot start later. Resubmission keeps the Job ID, advances its submission revision, and replaces or clears its deadline. Automatic retries retain the first-claim evidence. Their image, command, environment, resources, and Docker options come from the Worker. Per-Job buckets, logs, exec, and SSH are unavailable because the Jobs share a long-running application process.

Submit a job​

anycloud job ghcr.io/acme/train:latest \
  --credentials my-aws \
  --gpu-type h100:8 \
  --spot \
  --gpus all \
  -- python train.py

The Python example waits up to one hour. A DeploymentWaitTimeout stops waiting without terminating the Job; see Wait and terminate.

Choose hardware by GPU name when the accelerator matters, or by exact VM type when the instance shape matters. See GPUs & VM Types for that decision and job reference for every option.

Package code in the image​

Build the application and its dependencies into a published image before submitting it:

anycloud job ghcr.io/acme/trainer:sha-4d82f7a --gpu-type h100:8

Anycloud resolves the image to a registry digest before pulling it, so a mutable tag always resolves to the content available for that deployment. Tag images with a commit SHA or another immutable release identifier when reproducibility matters. See Container Images for build, cache, and registry guidance.

Lifecycle​

Job lifecycleHappy path
Wait
queued
Waiting for capacity or for a spend control to clear
Setup
provisioning
Create the VM
initializing
Install the host runtime
downloading
Pull the container image
syncingif used
Restore input or checkpoint data
starting
Start the container
Execute
running
The workload is executing
Preserve
finalizingif used
Save final artifacts
Result
completed
Exit code 0
or
errored
Non-zero exit
Return paths
Setup failureretrying⏳ queued
Spot preemptionrecovering⏳ queued
Terminal exits
failedRequired retries exhausted
invalidConfiguration rejected
terminatedStopped by a user

Infrastructure setup failures are retried up to 100 setup attempts. Failures during image download, data sync, or container startup can reuse an existing VM. When replacement is required, retries wait for the provider to confirm release of the old VM; waiting for cleanup does not consume setup attempts. Invalid configuration is not retried. A local API can have 50 deployments in the setup pipeline at once; that cap does not limit the number already running.

A job publishes its terminal outcome as soon as it is known; completedAt is that time. Resource teardown continues without adding another lifecycle event, and cleanedAt stays null until teardown succeeds; resubmission is blocked in between. Compute spend follows each VM's own lifetime — from when it is created until its endedAt cutoff — independently of both deployment timestamps.

Use status to see the current phase and its events:

anycloud status <job-id> --watch
anycloud status <job-id> --verbose

Data and artifacts​

Attach object storage when the job needs datasets, durable results, or recoverable state:

Container pathAccessBehavior
/mnt/inputRead-onlyDownloaded before the container starts
/mnt/outputRead-writeUploaded about every 60 seconds and at exit
/mnt/checkpointRead-writeRestored at startup; uploaded during the run

See Buckets for configuration and sync behavior. An explicit checkpoint bucket works on both on-demand and spot Jobs. Spot Jobs without one keep the automatic deployment-named checkpoint described below.

Spot recovery​

Spot capacity costs less but may be reclaimed at any point. With --spot, Anycloud detects preemption, cleans up the lost VM, provisions a replacement, restores /mnt/checkpoint, and restarts the container.

from pathlib import Path
import json

checkpoint = Path("/mnt/checkpoint/state.json")
start_epoch = json.loads(checkpoint.read_text())["epoch"] if checkpoint.exists() else 0

for epoch in range(start_epoch, 100):
    train_one_epoch(epoch)
    checkpoint.write_text(json.dumps({"epoch": epoch + 1}))

Checkpoint changes sync about every 60 seconds, so checkpoint often and make startup idempotent. The automatic checkpoint bucket is deleted when a spot Job finishes. To retain checkpoint state across separate Jobs, create a bucket and pass it with --checkpoint-bucket.

Compose jobs​

Sequence Jobs with ordinary Python. Wait for a preparation step to finish, then use its output bucket as the training Job's input:

from anycloud import Client

with Client() as client:
    prepared = client.submit(
        "ghcr.io/acme/prepare:latest",
        credential_name="my-aws",
        gpu="h100",
        cloud_config={"output_bucket": "prepared-training-data"},
    )
    prepared.wait()
    trained = client.submit(
        "ghcr.io/acme/train:latest",
        credential_name="my-aws",
        gpu="h100:8",
        cloud_config={"input_bucket": "prepared-training-data"},
    )
    trained.wait()

The preparation container writes to /mnt/output; training reads from /mnt/input. Use separate buckets or application prefixes for concurrent runs. A failed wait raises before the next submission. To run independent Jobs in parallel, submit them before waiting and handle partial failures in your application. The SDK has no JobGroup, submit_many(), or get_or_submit() helper.

Operate and debug​

anycloud job list --status running
anycloud logs <job-id>
anycloud exec <job-id> "nvidia-smi"
anycloud terminate <job-id>
anycloud resubmit <job-id>

exec and live logs require the job execution environment to still exist. When an output bucket is attached, logs are also copied under .anycloud/logs/, preserving them after teardown. Exact lifecycle and operation syntax is in Deployment operations.

Agent-operated jobs​

The CLI supports coding agents that run and monitor Jobs directly. Commands that print tables also provide structured output, and non-interactive deployments from Claude Code, Codex, Cursor, and Aider are tagged with the detected agent and session.

anycloud job list --agent codex --json
anycloud job list --session <session-id> --only-ids
anycloud status <job-id> --json
anycloud cost --agent codex --json

When an agent invokes list, job list, or cost, results default to that agent's own session. This is cooperative scoping for usability, not an access boundary; an explicit --session or --agent can select another scope.

Codex submissions use the native CODEX_THREAD_ID when available. Older Codex environments that expose only CODEX_BIN_PATH retain the codex-<parent-process-id> fallback. After upgrading, new native-thread scopes do not include earlier process-based sessions. Select that history explicitly:

anycloud job list --session codex-49152
anycloud cost --agent codex

Claude Code and Codex session titles appear in human-readable session output when available. Codex titles come from session_index.jsonl under $CODEX_HOME, which defaults to ~/.codex. Titles are captured when work is submitted; renaming a session affects subsequent submissions. Missing or unreadable titles keep the usual session identifier display.

Use anycloud docs for the documentation index and anycloud docs --all for the full Markdown corpus. Pair autonomous sessions with per-session Spend Controls.