Model deployments

A Model Deployment is the one customer-facing resource. A catalog model, private model, LoRA, serving engine and container image are composable fields, not separate deployment products.

Every deploy creates a new version behind the same endpoint. Nestor keeps the old versions so you can see what produced a result and roll back safely.

What the customer supplies

For a public Hugging Face model, create a deployment with its repository and exact commit:

{
  "model": {
    "source": "huggingface",
    "id": "Qwen/Qwen3-0.6B",
    "revision": "<hugging-face-commit-sha>"
  }
}

Nestor selects its tested vLLM version and all container configuration. serving.engine may be auto or vllm; auto is the default.

When weights alone do not reproduce the application—proprietary preprocessing, native dependencies, ComfyUI custom nodes, or weights embedded into an image—the same deployment may include a customer-supplied image.

Current availability

Today, model references support public, vLLM-compatible Hugging Face text models pinned to a commit. Deployments may also run ComfyUI or a customer JSON HTTP-serving image.

Private and gated models, LoRAs, SGLang selection, object-storage model sources, and the CLI are not yet public features.

Read-only model files

A deployment may materialize public Hugging Face repositories at exact commits before its runtime starts. Hugging Face remains the source of truth; each node keeps a reconstructable local cache and mounts only the declared directory into the container read-only:

{
  "artifacts": [{
    "name": "customer-style",
    "source": "huggingface",
    "repo": "acme/customer-style",
    "revision": "<hugging-face-commit-sha>",
    "include": ["*.safetensors", "config/*.json"],
    "mount_path": "/data/models/loras/customer-style"
  }]
}

Every mount path must be below /data/models. Newly assigned or replacement nodes download the same immutable revision automatically, while subsequent deployments on a node reuse its completed cache. /data/models is not a writable or distributed filesystem. Private/gated Hugging Face repositories and customer credentials are deliberately deferred to the next step.

include is optional. When present, only matching repository files are materialized. Patterns are part of the immutable cache identity, so a complete snapshot and a filtered snapshot never share a completion marker.

The current advanced DeploymentSpec is:

{
  "runtime": {
    "driver": "generic_http",
    "container": {
      "image": "registry.example.com/acme/model@sha256:...",
      "command": ["python", "-m", "server"]
    },
    "ports": [
      {"name": "runtime", "container_port": 8000, "protocol": "http"}
    ],
    "readiness": {
      "path": "/health",
      "interval_seconds": 5,
      "timeout_seconds": 2
    },
    "config": {
      "inference_path": "/infer",
      "method": "POST"
    }
  },
  "configuration": {
    "environment": {"MODEL_ID": "acme/model"},
    "secret_refs": {}
  },
  "resources": {
    "gpu_count": 1,
    "allowed_gpu_models": ["H100"],
    "minimum_vram_gb": 40
  },
  "execution": {
    "concurrency": 1,
    "timeout_seconds": 300,
    "startup_timeout_seconds": 900,
    "cancellation_grace_seconds": 10
  },
  "artifacts": []
}

Managed serving example

Internally, a managed vLLM choice resolves into an image, model command, readiness route and inference route. Customers should not normally construct this object:

{
  "runtime": {
    "driver": "generic_http",
    "container": {
      "image": "vllm/vllm-openai:<pinned-version>@sha256:<digest>",
      "command": [
        "--model", "Qwen/Qwen3-0.6B",
        "--host", "0.0.0.0",
        "--port", "8000"
      ]
    },
    "ports": [{"name":"runtime","container_port":8000,"protocol":"http"}],
    "readiness": {"path":"/health","interval_seconds":5,"timeout_seconds":2},
    "config": {"inference_path":"/v1/chat/completions","method":"POST"}
  },
  "configuration": {"environment": {}, "secret_refs": {}},
  "resources": {"gpu_count":1},
  "execution": {
    "concurrency":1,
    "timeout_seconds":300,
    "startup_timeout_seconds":900,
    "cancellation_grace_seconds":10
  }
}

The model-reference API generates this configuration. SGLang may follow the same pattern later.

Run chat inference

Use the standard OpenAI client with the endpoint-specific base URL:

from openai import OpenAI

client = OpenAI(
    api_key="nsk_...",
    base_url=(
        "https://api.serve.nestor.software/"
        "v1/endpoints/ep_your_model/openai"
    ),
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Hello"}],
)

Non-streaming chat completions are supported. stream=true returns a clear error until Nestor has a real token-streaming transport.

Customer serving code

If a deployment includes customer PyTorch code, the supplied image exposes a readiness route and a JSON inference route:

import os
from fastapi import FastAPI

app = FastAPI()
model = load_model(os.environ["MODEL_PATH"]).eval().cuda()

@app.get("/health")
def health():
    return {"ready": True}

@app.post("/infer")
def infer(payload: dict):
    return {"prediction": predict(model, payload)}

Package that server into an image and attach it to the same Model Deployment. The model weights may be:

• baked into the image;
• downloaded during image build;
• downloaded at container startup; or
• mounted from a persistent model volume/cache.

The first three patterns work today. Customer-declared persistent volumes, tenant-isolated caches and gated-model secret resolution are not yet customer-facing primitives.

Image models

Nestor should select a tested image server for a catalog or ordinary diffusion model. A deployment may specify ComfyUI when it includes customer-authored graphs, custom nodes, or a complex image/video pipeline. Both are still Model Deployments and use the same endpoint, capacity and job machinery.

Capacity is independent

Any runtime can be fixed dedicated, warm elastic, or scale-to-zero. Runtime selection does not decide the commercial or capacity model.