Skip to main content

Troubleshooting

Common errors and what to do about them. For the full state machine, see Jobs — Lifecycle.

Job states & failures​

My job is Errored — what happened?​

Errored (🪲) means your container ran but exited non-zero — your code or command failed, not the infrastructure. anycloud does not retry errored jobs. Read the output with anycloud status <id> --verbose, fix the problem, and resubmit. See Jobs — Lifecycle.

What's the difference between Failed and Errored?​

Failed (❌) is an infrastructure error — the VM never started. anycloud auto-retries these up to 100 times (through the Retrying state) before giving up and marking the job Failed. So Failed means every setup attempt failed, while Errored means your code ran and exited non-zero. See Jobs — Lifecycle.

My job went Invalid and never retried.​

Invalid (🚫) means the config is wrong — for example, an input bucket that doesn't exist or a single credential the cloud rejected. Invalid jobs are never retried; fix the config and resubmit. Run anycloud status <id> for the reason. See Jobs — Lifecycle and Cloud Credentials.

What does Terminated mean?​

Terminated (🪦) is user-initiated — someone ran anycloud terminate <id> or called job.terminate(). It's a terminal state, not a failure. Once its resources finish releasing, run it again with anycloud resubmit <id>; an early resubmit tells you to retry after cleanup finishes.

My job is stuck in queued.​

For dedicated VM Jobs, two common causes are:

  1. More than 50 deployments are setting up VMs at once (the per-API provisioning cap), so the rest wait for a slot.
  2. A spend control — throttle or budget — is at its cap.

For Worker-targeted Jobs, inspect client.get(job_id).status().jobs_ahead and client.get_worker(worker_name) while the client is open. Queue position and Worker readiness are advisory; see Python queue status.

For dedicated Jobs, the blocking reason shows in anycloud status <id> and anycloud list, and the job auto-dispatches once a slot frees or the cap clears. See Jobs — Lifecycle and Spend Controls.

My spot job keeps restarting (Recovering).​

That's expected on spot. When the cloud preempts the VM, anycloud detects it and re-provisions from scratch through the Recovering (🔁) state. Write checkpoints to /mnt/checkpoint frequently so a restart resumes with minimal lost work. See Jobs — Spot recovery.

How do I get logs from a job that already failed?​

anycloud status <id> --verbose works after the fact. With an open Python Client, use client.get(job_id).status() to inspect state and events. The SDK does not provide log streaming or download helpers; use the CLI for logs.

My job was killed (exit 137) or ran out of disk.​

Exit code 137 means the container was killed — usually out of memory. Raise --memory (and --shm-size for PyTorch DataLoaders and other shared-memory use). No space left on device means the disk filled up — raise --disk-size. See Docker runtime options.

Setup & credentials​

anycloud api start fails, or the SDK can't reach the API.​

The local API runs as a Docker container, so Docker must be running first. Check anycloud api status; if it's down, run anycloud api start. Python's Client uses an explicit api_url=, then API_URL, then the saved API target, falling back to http://localhost:8080. Check for a stale override or pass Client(api_url="http://localhost:8080"). Generated clients instead take Configuration(host="http://localhost:8080/v1"). See SDK configuration. The local API binds to 127.0.0.1 and is not intended for LAN exposure.

anycloud api start says the server is already running.​

A container is already up. Stop it first with anycloud api stop, then anycloud api start. (anycloud update restarts a running API server for you, so you normally don't need to do this after an update.)

No authentication token found.​

Run anycloud login, pass Client(token="github-access-token"), or set ANYCLOUD_TOKEN or GITHUB_TOKEN. Client checks the explicit token first, then $ANYCLOUD_DIR/.token (default ~/.anycloud/.token), then those environment variables in order. A saved nonempty token overrides environment tokens. Generated ApiClient setup requires an explicit Configuration(access_token=...); it does not discover CLI state itself.

Which credential does the SDK use?​

For an image Job, pass client.submit(..., credential_name="aws-prod") or inline cloud_config={"credentials": ...}. API token discovery does not choose a cloud compute credential. Worker-targeted Jobs use their Worker's Cluster capacity and reject per-Job credentials. See Credential selection.

A credential error sent my job to Invalid mid-run.​

A selected credential the cloud rejects makes the deployment Invalid immediately. If the secret itself is unchanged, fix its cloud-side access, re-save it to clear the invalid mark, and resubmit. Rotating or replacing the secret creates a new credential identity, so create a new deployment with that credential instead. Anycloud does not switch accounts automatically. See Cloud Credentials — Select credentials for compute.

Capacity & quota​

Capacity errors keep my job in Retrying.​

When the cloud denies capacity for the VM family, anycloud blocks that region and keeps retrying other regions — unless you pinned --region, in which case it can only retry the one region. Drop the region pin to let anycloud fail over. See GPUs & VM Types — Credentials and regions.

How do I raise my cloud quota?​

Request it from the CLI, then check status:

anycloud quota request <vmType> --credential <name>
anycloud quota status --credential <name>

Re-running against a region that already has an open case returns SKIPPED with the existing case's URL — no duplicates. See anycloud quota request.

Images​

Image pull failed with denied or 401.​

For a private image, anycloud pulls it on the VM using your GitHub token. A denied / 401 pull error usually means that token is stale — re-run anycloud login to re-authenticate. Only private images on GHCR are supported. See Container Images.

ConflictException — deployment ID already exists.​

You reused a custom deployment_id. Generated HTTP exceptions expose status 409 through ApiException.status. Use a new ID, or reattach with client.get(job_id) and call job.resubmit() once the existing Job is eligible and cleanup has finished. The CLI equivalent is anycloud resubmit <id>. See Error handling.

GPU vs VM selection​

Should I use --gpu-type or --vm-type?​

--gpu-type (Python gpu) names a GPU like h100 or a100:8 and lets anycloud pick the cheapest matching instance across clouds and regions. --vm-type (Python cloud_config["vm_type"]) pins an exact instance. They are mutually exclusive.

My container can't see the GPU.​

--gpu-type / --vm-type choose the hardware; anycloud sets up GPU access for you, so the container sees the GPUs with no extra flags. Pass --gpus to set a specific device count. See Container Images — GPU images.

How do I see what GPUs, VM types, or regions are available?​

Use the catalog commands: anycloud gpus <provider>, anycloud vm-types <provider> <region>, anycloud regions <provider>, and anycloud pricing <provider> <vm-type>. See Catalog and pricing.