Buckets

Buckets connect durable cloud object storage to a Job as ordinary files. Anycloud handles S3, Azure Blob Storage, or Google Cloud Storage access and bind-mounts the synchronized directories into the container.

Use Buckets for large datasets, model weights, results, logs, and state that must survive a spot interruption. They are a first-class platform primitive, not an image-building detail.

The three mounts

Role	Container path	Access	Sync
Input	/mnt/input	Read-only	Download once before startup
Output	/mnt/output	Read-write	Upload about every 60 seconds and at exit
Checkpoint	/mnt/checkpoint	Read-write	Restore at startup; upload about every 60 seconds and at exit

Input buckets must already exist. Output buckets are created when necessary. A checkpoint bucket is created automatically for a spot Job.

Attach input and output

CLI

anycloud submit ghcr.io/acme/train:latest \
  --credentials my-aws \
  --gpu-type a100:8 \
  --spot \
  --input-bucket training-data \
  --output-bucket training-results \
  -- python train.py

Python

import anycloud

ac = anycloud.Client(credentials="my-aws")
job = ac.submit(
    "ghcr.io/acme/train:latest",
    gpu="a100:8",
    input=ac.bucket("training-data"),
    output=ac.bucket("training-results"),
    command=["python", "train.py"],
)

Inside the container, application code only reads and writes paths:

from pathlib import Path

model = Path("/mnt/input/base-model.bin").read_bytes()
Path("/mnt/output/metrics.json").write_text('{"loss": 0.12}')

Sync guarantees

Input uses rclone sync once before startup. Output uses upload-only rclone copy, so two Jobs writing to one bucket do not delete one another's objects. Checkpoints restore existing objects first and then use the same upload-only behavior.

When a bucket-backed Job exits, it enters finalizing while Anycloud performs one last output and checkpoint copy. completed or errored is published only after that succeeds. If final sync exhausts its retries, the Job becomes failed because its artifacts could not be preserved.

Files exist once on the VM and are exposed through Docker bind mounts; there is no second copy inside the container. The host paths can differ by provider, but the three /mnt/... container paths are stable.

Work with objects directly

Use the CLI when you need to prepare or inspect a bucket outside a Job:

anycloud bucket create training-data --credentials my-aws
anycloud bucket upload training-data ./dataset.parquet data/dataset.parquet \
  --credentials my-aws
anycloud bucket download training-data data/dataset.parquet ./dataset.parquet \
  --credentials my-aws

The Python SDK exposes the same operations through a lazy Bucket handle:

data = ac.bucket("training-data")
data.upload("./dataset.parquet", remote_path="data/dataset.parquet")
data.download("./copy.parquet", remote_path="data/dataset.parquet")

See bucket reference for exact commands.

Authentication

Same-cloud sync uses the VM's cloud-native identity:

• AWS IAM roles for S3
• Azure managed identities for Blob Storage
• GCP service accounts for Cloud Storage

Long-lived cloud credentials are not injected into the workload container.

If storage belongs to a different account, give the Bucket its own credential:

compute = anycloud.Client(credentials="gcp-compute")
storage = anycloud.Client(credentials="aws-storage")

job = compute.submit(
    "ghcr.io/acme/train:latest",
    input=storage.bucket("shared-training-data"),
)

On the CLI, use --input-storage-credentials and --input-storage-region, or the corresponding output flags. Input and output identities are independent.

Cross-cloud storage currently supports AWS S3. Anycloud mints bucket-scoped, short-lived STS credentials rather than copying the saved AWS key to the VM. The token lasts up to 36 hours and is not refreshed during a long-running Job.

When compute credentials are unpinned, a user-supplied Bucket still needs explicit storage credentials so its identity does not change when compute lands in another cloud.