Pipeline: chain jobs through buckets

Run a multi-step pipeline — prep → train → eval — where each step reads the previous step's output from a bucket. Each job waits for the one before it; anycloud auto-creates output buckets and mounts them at /mnt/input (read-only) and /mnt/output.

import anycloud

ac = anycloud.Client()

# Output buckets are created on first write
prep_out = ac.bucket("example-prep-output")
train_out = ac.bucket("example-train-output")

prep = ac.submit("my-prep:latest", gpu="a100:1", output=prep_out)
prep.wait()

train = ac.submit("my-train:latest", gpu="a100:8", input=prep_out, output=train_out)
train.wait()

eval_job = ac.submit("my-eval:latest", gpu="a100:1", input=train_out)
eval_job.wait()

print(eval_job.logs())

Each step blocks on .wait() before the next is submitted. Data flows through the buckets: prep writes to /mnt/output (synced to example-prep-output), train reads it at /mnt/input and writes its own output, and eval reads that in turn.

See Chaining Jobs and Bucket Sync for more on passing data between jobs.