Model deployments

A Model Deployment is the one customer-facing resource. A catalog model, private model, LoRA, serving engine and container image are composable fields, not separate deployment products.

Every deploy creates a new version behind the same endpoint. Nestor keeps the old versions so you can see what produced a result and roll back safely.

What the customer supplies

For a public Hugging Face model, create a deployment with its repository and exact commit:

{
  "model": {
    "source": "huggingface",
    "id": "Qwen/Qwen3-0.6B",
    "revision": "<hugging-face-commit-sha>"
  }
}

Nestor selects its tested vLLM version and all container configuration. serving.engine may be auto or vllm; auto is the default.

When weights alone do not reproduce the application—proprietary preprocessing, native dependencies, ComfyUI custom nodes, or weights embedded into an image—the same deployment may include a customer-supplied image.

Current availability

Today, model references support public, vLLM-compatible Hugging Face text models pinned to a commit. Deployments may also run ComfyUI or a customer JSON HTTP-serving image.

Private and gated models, LoRAs, SGLang selection, object-storage model sources, and the CLI are not yet public features.

The current advanced DeploymentSpec is:

{
  "runtime": {
    "driver": "generic_http",
    "container": {
      "image": "registry.example.com/acme/model@sha256:...",
      "command": ["python", "-m", "server"]
    },
    "ports": [
      {"name": "runtime", "container_port": 8000, "protocol": "http"}
    ],
    "readiness": {
      "path": "/health",
      "interval_seconds": 5,
      "timeout_seconds": 2
    },
    "config": {
      "inference_path": "/infer",
      "method": "POST"
    }
  },
  "configuration": {
    "environment": {"MODEL_ID": "acme/model"},
    "secret_refs": {}
  },
  "resources": {
    "gpu_count": 1,
    "allowed_gpu_models": ["H100"],
    "minimum_vram_gb": 40
  },
  "execution": {
    "concurrency": 1,
    "timeout_seconds": 300,
    "startup_timeout_seconds": 900,
    "cancellation_grace_seconds": 10
  }
}

Managed serving example

Internally, a managed vLLM choice resolves into an image, model command, readiness route and inference route. Customers should not normally construct this object:

{
  "runtime": {
    "driver": "generic_http",
    "container": {
      "image": "vllm/vllm-openai:<pinned-version>@sha256:<digest>",
      "command": [
        "--model", "Qwen/Qwen3-0.6B",
        "--host", "0.0.0.0",
        "--port", "8000"
      ]
    },
    "ports": [{"name":"runtime","container_port":8000,"protocol":"http"}],
    "readiness": {"path":"/health","interval_seconds":5,"timeout_seconds":2},
    "config": {"inference_path":"/v1/chat/completions","method":"POST"}
  },
  "configuration": {"environment": {}, "secret_refs": {}},
  "resources": {"gpu_count":1},
  "execution": {
    "concurrency":1,
    "timeout_seconds":300,
    "startup_timeout_seconds":900,
    "cancellation_grace_seconds":10
  }
}

The model-reference API generates this configuration. SGLang may follow the same pattern later.

Run chat inference

Use the standard OpenAI client with the endpoint-specific base URL:

from openai import OpenAI

client = OpenAI(
    api_key="nsk_...",
    base_url=(
        "https://api.serve.nestor.software/"
        "v1/endpoints/ep_your_model/openai"
    ),
)

response = client.chat.completions.create(
    model="Qwen/Qwen3-0.6B",
    messages=[{"role": "user", "content": "Hello"}],
)

Non-streaming chat completions are supported. stream=true returns a clear error until Nestor has a real token-streaming transport.

Customer serving code

If a deployment includes customer PyTorch code, the supplied image exposes a readiness route and a JSON inference route:

import os
from fastapi import FastAPI

app = FastAPI()
model = load_model(os.environ["MODEL_PATH"]).eval().cuda()

@app.get("/health")
def health():
    return {"ready": True}

@app.post("/infer")
def infer(payload: dict):
    return {"prediction": predict(model, payload)}

Package that server into an image and attach it to the same Model Deployment. The model weights may be:

• baked into the image;
• downloaded during image build;
• downloaded at container startup; or
• mounted from a persistent model volume/cache.

The first three patterns work today. Customer-declared persistent volumes, tenant-isolated caches and gated-model secret resolution are not yet customer-facing primitives.

Image models

Nestor should select a tested image server for a catalog or ordinary diffusion model. A deployment may specify ComfyUI when it includes customer-authored graphs, custom nodes, or a complex image/video pipeline. Both are still Model Deployments and use the same endpoint, capacity and job machinery.

Capacity is independent

Any runtime can be fixed dedicated, warm elastic, or scale-to-zero. Runtime selection does not decide the commercial or capacity model.