Ornn Serverless
Run Python on GPUs with one decorator.
Annotate a class or function, deploy it with one command, and call it like a local object. Ornn starts replicas when calls arrive and scales them to zero when they go quiet.
How it works
From pip install to a GPU endpoint in four steps
No Dockerfiles to write, no clusters to size, and no YAML. Your code is the configuration.
- 01
Install and sign in
ornn loginsigns the CLI and the Python SDK in with your Ornn account.Terminalpip install ornn ornn login - 02
Write your app
Each
@app.cls,@app.fn, or@app.serveris its own unit with its own replicas. Ask for a GPU type and Ornn sizes CPU, memory, and disk to match.app.pyimport ornn app = ornn.App("hello-gpu") @app.cls(gpu="H100", idle_timeout="5m") class Model: @ornn.enter def load(self): self.pipe = load_pipeline() # once per replica def generate(self, prompt): return self.pipe(prompt) @app.fn(gpu="H100") def embed(texts): return encoder(texts) - 03
Deploy
One command uploads your code. Code-only deploys never rebuild images, and anything your units print shows up in
ornn serverless logs.Terminalornn serverless deploy ./app.py ornn serverless logs hello-gpu - 04
Call it from anywhere
Look the app up by name and call its units like local objects. Replicas start when calls arrive and stop when they go quiet.
client.pyimport ornn ornn.use() # your organization's namespace app = ornn.App.lookup("hello-gpu") print(app.Model.generate("Write a haiku about GPUs")) # fan out across replicas vectors = list(app.embed.map(batches))
Features
Everything a GPU workload needs
Options live on the decorator, next to the code they configure.
Scale to zero
Replicas start when calls arrive and stop after
idle_timeout. Keepmin_replicaswarm when cold starts matter.GPUs by type or by memory
Ask for
gpu="H100:2"ormin_gpu_memory=160. The scheduler picks the GPU and sizes the rest.Fan-out and background calls
.mapruns inputs in parallel with ordered results..spawnreturns at once and can be collected from any process.Shared state
ornn.Dictandornn.Queueare shared by every replica, so units can count, cache, and hand off work.Secrets and volumes
Secrets become environment variables and are never shown after you set them. Mount your organization's volumes at any path.
HTTP servers
@app.server(port=8000)serves HTTP, including streaming, at the unit's URL. Run vLLM or your own API.Image builds
Chain
apt_installandpip_install, or bring a Dockerfile. Images are cached by step, so changing one step rebuilds from there.Your own reserved nodes
Create a Serverless controller from nodes you've reserved on Ornn, and your units run only on them.
Clients in other languages
Call units with JSON from Node (
@ornn/api), Rust (ornn-api), or a header-only C++ client.
Moving from Modal
Bring your Modal app over with one command
ornn serverless migrate-modal rewrites Modal code into Ornn code in place and writes ORNN_MIGRATION.md listing anything that needs your review. Add --check in CI to fail while Modal code remains.
import modal
app = modal.App("ranker")
@app.cls(gpu="H100", scaledown_window=300)
class Embedder:
@modal.enter()
def load(self):
self.model = load_model()
@modal.method()
def encode(self, batch):
return self.model(batch)
@app.function(max_containers=10)
def rank(queries):
return sorted(Embedder().encode.remote(queries))import ornn
app = ornn.App("ranker")
@app.cls(gpu="H100", idle_timeout=300)
class Embedder:
@ornn.enter
def load(self):
self.model = load_model()
def encode(self, batch):
return self.model(batch)
@app.fn(max_replicas=10)
def rank(queries):
return sorted(Embedder().encode(queries))Get started
Ship your first GPU function today
Install the SDK, sign in with your Ornn account, and deploy.