Bucket Sync

anycloud can sync cloud storage (S3, Azure Blob, GCS) with your containers automatically. Configure buckets in your profile and they're mounted directly into your container — no setup code needed.

Three Bucket Types

📥 Input Bucket

Downloaded once on startup, mounted read-only. Use for training data, models, datasets. You must create the bucket before deploying.

Python

job = ac.submit(
    "ghcr.io/acme/my-app:latest",
    input=ac.bucket("my-training-data"),
)

CLI

anycloud submit ghcr.io/acme/my-app:latest \
  --credentials my-aws \
  --input-bucket my-training-data

In your container:

# Read-only at /mnt/input
model = open('/mnt/input/model.pkl', 'rb').read()

📤 Output Bucket

Uploads continuously every ~60 seconds. Use for results, logs, artifacts. Output buckets are auto-created by the server if they don't exist.

Python

job = ac.submit(
    "ghcr.io/acme/my-app:latest",
    output=ac.bucket("my-results"),
)

CLI

anycloud submit ghcr.io/acme/my-app:latest \
  --credentials my-aws \
  --output-bucket my-results

In your container:

# Write at /mnt/output — syncs to cloud every ~60s
with open('/mnt/output/output.json', 'w') as f:
    json.dump(result, f)

💾 Checkpoint Bucket

Downloaded once on startup, then uploaded continuously every ~60 seconds. Auto-created per deployment — you don't configure it. Always mounted at /mnt/checkpoint.

On startup, any existing checkpoint data is downloaded from the bucket (e.g., from a previous preempted run). Then a continuous upload syncs local changes to the bucket every ~60 seconds. This means your container can read previous checkpoints and write new ones.

Use for state that needs to survive preemption (see Spot Instances).

# Read + write at /mnt/checkpoint (always this path)
checkpoint = json.loads(open('/mnt/checkpoint/state.json').read())
checkpoint['epoch'] += 1
with open('/mnt/checkpoint/state.json', 'w') as f:
    json.dump(checkpoint, f)

Combining Buckets

Python

job = ac.submit(
    "ghcr.io/acme/my-training:latest",
    gpu="a100:all",
    input=ac.bucket("training-data"),
    output=ac.bucket("results"),
)

CLI

anycloud submit ghcr.io/acme/my-training:latest \
  --credentials my-aws \
  --gpu-type a100 \
  --spot \
  --gpus all \
  --input-bucket training-data \
  --output-bucket results

This gives your container:

Path	Access	Sync
/mnt/input	Read-only	Once on startup
/mnt/output	Read-write	Upload every ~60s
/mnt/checkpoint	Read-write	Download once on startup, upload every ~60s

Input buckets must exist before deploying. Output and checkpoint buckets are auto-created if they don't exist.

Which Bucket Type to Use

• Input — data your job reads but never writes (training data, config, models)
• Output — data your job produces (results, logs, artifacts)
• Checkpoint — state that needs to survive preemption (training progress, recovery data)

Credentials

Bucket access uses cloud-native authentication (IAM roles, managed identities, service accounts) — no keys or secrets in your container. If your storage is in a different cloud account, use per-bucket credential flags (--input-storage-credentials, --output-storage-credentials) and their corresponding region flags:

Python

from anycloud.types import CloudConfig

job = ac.submit(
    "ghcr.io/acme/my-app:latest",
    cloud_config=CloudConfig(
        credentials="compute-account",
        input_storage_credentials="storage-account",
        input_storage_region="us-east-1",
    ),
    input=ac.bucket("shared-data"),
)

CLI

anycloud submit ghcr.io/acme/my-app:latest \
  --credentials compute-account \
  --input-storage-credentials storage-account \
  --input-storage-region us-east-1 \
  --input-bucket shared-data

Input and output storage credentials are independent. If your output bucket is also on a different cloud, set --output-storage-credentials / --output-storage-region too — otherwise the output bucket falls back to the compute credentials.

Cross-cloud storage currently supports AWS S3 only. anycloud mints scoped, short-lived STS credentials for the specific bucket and writes those credentials to the VM's rclone environment file; it does not copy your long-lived AWS key to the VM. The STS token lasts up to 36 hours and is not refreshed mid-run, so cross-cloud output or checkpoint sync can fail for jobs that keep running past that limit. Resubmitting the job mints fresh credentials.

Fallback Credentials

--credentials is repeatable: --credentials aws-prod --credentials gcp-prod runs the job on AWS first and rotates to GCP on auth failure. The deployment record keeps the full curated list; each VM is stamped with the credential that actually booted it.

Submit-time validation catches the most common fan-out gotcha — an input bucket that exists on one cloud but not another. With no explicit --input-storage-credentials the API checks every listed credential's storage; first miss returns a 400 naming the credential and cloud. Output buckets follow an all-or-none rule for the same reason.

A few things still bite even with the fail-fast checks in place:

• Output bucket forks. If you keep submitting against the same name and only one cloud has the bucket today, the all-or-none rule rejects the submission. Pre-create on every listed cloud (or on none — anycloud will create on whichever cloud the VM lands).
• vmType aliasing. A vmType alias like h100 resolves to different concrete instance types per cloud. Multi-cloud fallback can silently land a different SKU than the one you priced for.
• Custom AMIs / prebake. Custom images and prebaked AMIs are per-cloud. If only AWS has the image, GCP will rebuild from scratch (slow) or fail.
• Region selection. Each credential's default region applies. If you need EU residency or a specific region, pin --region or rely on the per-credential default — there is no "stay close to the data" optimizer pass yet.

Rotation is currently triggered only by auth failures (BadCredential). Quota and capacity errors still rotate at the (cloud, vmType, region) level — they don't flip to a different credential on their own. If you want quota fallback to a different cloud, expect to wait for the AWS target to unblock or rely on multi-region fallback within AWS first.