Skip to main content

Servers

A Server is a long-running container that receives HTTP traffic. Use one for model inference APIs, dashboards, development environments, and other services that should remain available instead of exiting after a finite task.

Cloud-backed Servers receive a stable public URL:

https://<deployment-id>.anycloud.sh

Use a Job for training, batch inference, or other work that should complete and automatically release its VM.

Start a server​

The process inside the container must listen on 0.0.0.0 at the port in $PORT. The default is 8088.

anycloud serve ghcr.io/acme/model-api:latest \
  --id model-api \
  --credentials my-aws \
  --gpu-type l40s \
  -- python -m myapp

The service becomes available at https://model-api.anycloud.sh.

Override the listening port with --env PORT=9000 or env={"PORT": "9000"}. The public URL still routes to the configured port.

Sync Python code from Git​

@anycloud.serve() is the server equivalent of @anycloud.function(). It clones the submitted Git commit into the container and calls a function that starts the server and blocks:

import os
import anycloud

@anycloud.serve(
    image="ghcr.io/acme/inference:latest",
    gpu="l40s:1",
    secrets=["model-api"],
)
def run():
    import uvicorn
    from myapp import app

    uvicorn.run(app, host="0.0.0.0", port=int(os.environ["PORT"]))

server = run.start(id="model-api")
server.wait_running(timeout=600)
print(server.url)

The repository and commit must be pushed, and the image must contain git.

Lifecycle​

Server lifecycleHappy path
Wait
queued
Waiting for capacity or a spend control
Setup
provisioning
Create backing capacity
initializing
Install the host runtime
downloading
Pull the container image
starting
Start and check the service
Serve
running
The service is live at its URL
Return paths
Setup failureretrying⏳ queued
Upgraderecovering⏳ queued
Terminal exits
failedRequired retries exhausted
invalidConfiguration rejected
terminatedStopped by a user

A Server stops at running instead of completing. It remains there until you terminate it or start an upgrade. An upgrade enters recovering, replaces the backing deployment, and returns through queued while preserving the Server ID and public URL.

Server behavior​

  • Servers use on-demand capacity. Spot is rejected because an HTTP endpoint cannot use the Job checkpoint-and-restart contract.
  • Job-only flags such as --persist, --persist-bucket, and bucket mounts do not apply.
  • Environment variables, named secrets, cloud selection, GPU or VM selection, disks, and Docker runtime options work the same way as on Jobs.
  • A server stays live until you terminate or upgrade it. It does not automatically shut down when a request completes.
  • On cloud substrates with public ingress, Anycloud exposes the application through its Anycloud URL. Local compute does not return a public endpoint. Your process should bind inside the container, not attempt to manage the URL.

Upgrade without changing the URL​

Upgrade replaces the backing deployment while preserving its ID and public URL:

anycloud serve upgrade model-api ghcr.io/acme/model-api:v2 \
  -- python -m myapp

Use an immutable image tag or digest when you need an auditable rollout. See serve reference for the complete upgrade syntax.

Operate a server​

Servers share the common deployment commands:

anycloud status model-api --watch
anycloud logs model-api
anycloud exec model-api "nvidia-smi"
anycloud terminate model-api

The Python Server handle provides state(), status(), logs(), exec(), and terminate(). See Python SDK: Server Methods.