Running jevimage for other people. One process holds the encoder; everyone else installs pip install jevimage (a few hundred kilobytes, no torch, no weights) and calls jevimage.connect(url). Every route under /v1 is one method on jevimage.Jev, so there is no second API to learn: /v1/ask is ask(), /v1/embed is embed(), /v1/heads is train/list_heads/delete_head. The wire format itself is in reference-http.md; this page is about operating the thing.
Every transcript below was run against a deterministic stand-in encoder (16-dim, no downloads), so the transcripts reproduce in a second. Statuses, error text, JSON shapes and commands are real. Probabilities and latencies are the toy’s, not SigLIP’s. A real encoder is slower and its probabilities are spread out rather than saturated.

​
Install

[serve] implies [local]: a server runs the encoder, so it pulls torch, transformers, fastapi and uvicorn. See install.md for what each extra costs.

​
Starting it

​
jev serve

Defaults: --host 127.0.0.1, --port 8000, the registry default encoder (siglip2-base-224), heads in $JEV_HEADS_DIR, else $JEV_HOME/heads, else ~/.jev/heads, and no key. The flags are not passed to the app object. They are written into the environment (JEV_ENCODER, JEV_HEADS_DIR, JEV_API_KEYS) and the app reads them itself, so the same knobs work when a container or a process manager starts uvicorn directly:
jev serve is a single uvicorn process with no --workers, no TLS flags and no --reload. When you want any of those, run uvicorn yourself; the two paths build the identical app. jev serve only sees the built-in encoder registry. The CLI never imports your code, so a registered custom encoder is invisible to it:

​
build_app(): your own encoder

build_app(jev=None, *, api_keys=None):
  • jev=None calls jevimage.load(batch=True), which reads $JEV_ENCODER and $JEV_HEADS_DIR. Pass a Jev to control the encoder, the heads directory, the temperature, or batching.
  • api_keys=None reads $JEV_API_KEYS. Pass a list to set them in code. Note that api_keys=[] means open, overriding the environment.
Keeping app at module level (rather than only inside __main__) is what lets you run it under uvicorn serve_toy:app --workers 2. jevimage.server.app is built lazily through a module __getattr__, so from jevimage.server import build_app does not load a model; touching jevimage.server:app does.

​
Mounting it in your own FastAPI app

build_app() returns an ordinary FastAPI instance. Add routes to it, or mount it under a prefix:
(mount_toy.py has no uvicorn.run of its own - it is a module for a server to import, which is why the ASGI path mount_toy:api is on the command line instead.) A mount has its own auth: the sub-app checks the keys build_app was given, and the outer app’s route dependencies do not reach it. FastAPI(dependencies=[...]) on the outer app leaves /jev/v1/health open. Outer middleware does wrap the mount, so if you mount an open app inside an authenticated one, put the check there, or give build_app keys anyway.

​
API keys

One header, one form:
The scheme token is case-insensitive (RFC 7235 §2.1) and any run of whitespace separates it from the key, so bearer <key> is accepted; the key itself is compared exactly. Keys come from $JEV_API_KEYS, split on commas and whitespace, so both spellings work and a unit file that copies the CLI’s comma form cannot produce a key with a space in it:
Any one of them is accepted; comparison is hmac.compare_digest against each. A junk or non-ASCII header is a 401, not a 500. Empty strings are dropped before the guard is built, so api_keys=[os.environ.get("JEV_KEY", "")] on a host where that variable is unset is an open server that says "auth": "open", not one that claims to be protected and is not. On the client side, the key goes to whatever the URL resolves to. connect() refuses redirects rather than following them, because urllib copies the Authorization header onto the redirect target (a different host, or a downgraded scheme, included), so one 302 from a hostile or compromised server would hand over the key. A 3xx surfaces as a JevError naming the Location. That still leaves the transport: over plain http://, as every example on this page does, the key is on the wire in clear. Terminate TLS in front of jev serve for anything that leaves a trusted network. $IMG in the transcripts below is one base64 image, made once:
The key alone, without the Bearer prefix, is rejected.

​
What the key does not cover

If a keyless schema page is not acceptable, build the app with FastAPI(docs_url=None, ...) (which means constructing it yourself rather than calling build_app), or block those three paths at the proxy. Auth is a route dependency, so the body is read and JSON-parsed before the key is checked. A malformed body from an anonymous caller gets a 422, not a 401:
The practical consequence: a key does not protect you from a flood of large uploads. A body-size cap at the proxy does.

​
No key means open

Not “read-only open”. Open:
Anyone who can reach the port can spend your GPU on /v1/ask, upload up to 4000 images to /v1/heads, overwrite a head that shares a name with theirs, and delete any head you have. --host 0.0.0.0 with no key is a public GPU with a public disk. 127.0.0.1 with no key, behind a proxy that does the authentication, is a reasonable setup; 0.0.0.0 with no key is not. Put the keys in a file that only the service user can read, not on the command line. jev serve --api-key hunter2 shows up in ps for every user on the box. The systemd unit below uses EnvironmentFile for exactly this.

​
The limits the server enforces

Read straight from jevimage/server.py: Everything each refusal returns, verified against a running server: The transcript behind that table:
Those bodies are /v1/ask and /v1/embed. POST /v1/heads counts and labels examples instead, with its own larger cap:
  • The pixel cap is checked from the header, before the pixels are allocated. A solid-colour 8000×8000 PNG is 263 KB of base64 and 190 MB of RAM once decoded; the 413 above cost neither.
  • 422 is the core’s own sentence, not a wrapper. ValueError and KeyError from jevimage reach the caller as 422 and 404, because “head x was trained on encoder y” is the answer the caller needs, not a symptom to hide. The one thing the server edits out is a filesystem path: a 404 body does not disclose the heads directory, however the message read at the REPL.
  • There is no cap on the total body. Only per-image size and per-request count. A training POST is 4000 images × up to 12 MB each by the server’s own arithmetic. The ceiling has to come from the proxy.

​
batch=True and the shared encode

build_app() with no jev uses jevimage.load(batch=True), because a server is what the batcher is for. Requests arriving within a few milliseconds of each other are collected into one forward pass. The mechanism, measured with a stand-in encoder that sleeps 10 ms per call to stand in for a real forward pass (the toy’s own encode is microseconds, so nothing ever queues):
32 requests became 3 passes. Only the collapse reproduces. How many passes, and how they are filled, moves from run to run and from machine to machine, and the wall times say nothing at all: on the machine above the batched side is the slower of the two. That is the honest limitation of this measurement: a sleep costs the same for 8 images as for 1, so it models the coalescing but not the payoff. The payoff is real on a GPU, where the marginal image in a batch is nearly free. jevimage/batcher.py records the project’s own measurements on an A10G. I did not reproduce them, and this machine has no encoder to reproduce them with:
Three properties worth knowing before you rely on it:
  • The wait is adaptive, and it is off when traffic is thin. max_wait is 6 ms, but the batcher only waits when its recent batch size (an EMA) is at least 1.5. On an idle service a lone request goes straight through and pays nothing. Under load the batch fills before the timer. max_batch is 16, so one forward pass never carries more than 16 images, whatever stats["max_seen"] says you were offered.
  • Only single-image requests are batched. Jev.embed routes through the batcher only when it is handed exactly one image. A /v1/ask with "image" is one image and gets batched; /v1/embed with "images": [a, b] goes straight to the encoder as its own pass:
    That is the right default (a caller who sent a list already batched), but it means a client that loops embed(one) in 8 threads behaves differently from one that sends embed([eight]).
  • One bad batch does not kill the worker. The exception is delivered to every caller in that batch and the thread carries on.
The other half of the economics is per-request: /v1/ask encodes the image once and reads every question against that one vector. The response says so:
embed_ms is the encode; total_ms - embed_ms is every question. Ten questions in one request cost one encode; ten requests cost ten. Caption embeddings are cached per string across requests (50 000 strings, then oldest-first eviction), so a stable question set warms up once.

​
Health checks

  • Never authenticated, on purpose.
  • encoder and dim are what your heads will be trained under. encoder changing is the single most disruptive thing that can happen to a deployment (see below).
  • auth is "bearer" or "open", a one-request check that you did not ship an open server by accident.
  • device is getattr(encoder, "device", "cpu"). HFCLIP and OpenCLIP set it; a custom encoder that does not will report cpu however it actually runs.
  • heads re-reads the head directory on every call (jev.heads.sync()), which is a glob plus a torch.load of anything that changed. Once a second is nothing; once a millisecond is a filesystem benchmark.
The process does not answer until the weights are loaded. The app is built (and the encoder loaded) while the app object is being imported, before uvicorn serves anything. A single process refuses the connection until then; a --workers N run accepts it (the supervisor owns the socket) and leaves it waiting for a worker. First boot also downloads weights. Give the check a startup grace period generous enough for a cold model pull, or you will restart-loop a box that is doing exactly what you asked.

​
What to monitor

There is no /metrics route. What exists:
  • timing.embed_ms and timing.total_ms on every /v1/ask response. Watch the p95 of embed_ms; that is the encoder and the queue. If total_ms - embed_ms grows instead, you are asking a lot of questions with a lot of new captions.
  • Status-code rates, split the way the table above splits them. A rising 413/415 rate is a client sending the wrong thing; a rising 422 rate is usually one caller with a malformed question or a head that no longer matches the encoder.
  • uvicorn access logs for latency and non-2xx.
  • Process RSS and (on GPU) nvidia-smi memory. One model per worker process.
If you want counters, the app is yours to extend:
text_hits/text_misses is the caption cache; items / batches is your mean batch size, which tells you whether batching is doing anything. calls counts what was submitted to the batcher and items what a forward pass has come back for, so the two agree except for whatever is in flight. Both of these are per process, so with --workers N you are looking at one worker’s view. jev._batcher is private; it is stable enough to read for metrics, and nothing else here is.

​
Reverse proxy and TLS

jevimage has no TLS and no CORS headers. A browser page on another origin cannot call it, and jev serve cannot terminate HTTPS. Both belong to a proxy in front. The proxy caps the whole body; the server does not.
Verified end to end against a local copy of that config with max_size 1MB:
Note the status. Caddy 2.6.2 logs HTTP 413: http: request body too large and returns 502 to the client, because it had already begun proxying when the limit tripped. If your clients need a clean 413, check what your proxy version actually emits rather than assuming. The nginx equivalent is client_max_body_size 100m, which does return 413. Other proxy notes:
  • If you need a browser to call this directly, add CORS at the proxy. Adding CORSMiddleware to the app is also possible, but then the policy lives in your code and the TLS lives in the proxy, which is one config too many.
  • Uvicorn trusts X-Forwarded-* from 127.0.0.1 by default; if the proxy is on another host, pass --forwarded-allow-ips or every access log line will say the proxy’s IP.
  • /v1/heads (POST) is the slow route; it uploads a dataset and fits a head. Give the proxy a read timeout well above its default for that path, or the client will see a 504 for a request the server completes anyway.
  • Client-side, jevimage.connect(url, timeout=60.0) is the default. A 4000-image training upload will blow through 60 s: connect(url, timeout=900).

​
A worked deployment: systemd

Not containerised, one box, one GPU.
Why these bits:
  • --host 127.0.0.1. The proxy is the only thing that talks to it.
  • HF_HOME inside StateDirectory; otherwise the cache lands in the service user’s home, which ProtectHome=yes has just made inaccessible, and every restart re-downloads a few gigabytes.
  • ProtectSystem=strict plus one ReadWritePaths: heads are the only thing this process writes.
  • Restart=on-failure, not always: a config error that makes the encoder unloadable should stay down and visible, not loop while pulling weights.
  • /etc/jev/keys.env should be chmod 600, owned by root, containing one line: JEV_API_KEYS=key1,key2.

​
A worked deployment: Docker

Unlike everything else on this page, this Dockerfile was not built here. The machine it was written on has 2.4 GB of free disk and torch does not fit. It is assembled from behaviour that was verified (the env-var interface, uvicorn jevimage.server:app, the health route, where heads are written), not from a build log. Treat it as a starting point and read the build output.
--host 0.0.0.0 inside the container is right; publishing it as -p 127.0.0.1:8000:8000 is what keeps it off the network. -p 8000:8000 publishes it to the world, and the container has no key unless you gave it one. For a GPU: start from nvidia/cuda:12.4.1-runtime-ubuntu22.04, drop the CPU index, and run with --gpus all. resolve_device() picks CUDA automatically when torch sees it, and $JEV_DEVICE overrides.

​
Scaling

What is GPU-bound: the encoder forward pass, and nothing else. Every question is a matmul against a 768-float vector; a trained head is one more. That is why total_ms in the transcripts above sits so close to embed_ms. What is CPU-bound: base64 decoding, PIL decoding, EXIF transpose, convert("RGB"), and the processor’s resize/normalise. For a large JPEG this is not a rounding error; it can rival a small model’s forward pass. It is also where the request threads spend their time. How the concurrency works. The route handlers are ordinary def, so uvicorn runs them in its threadpool. Decoding therefore happens in parallel across requests. With batch=True, everything after decoding (the exif transpose, the preprocess and the forward pass) runs in the single batcher thread, which is the serialisation point. Given that, the knobs in order:
  1. Ask more questions per request. Free for a question set the process has already seen: ten questions in one /v1/ask is one encode, and the second question costs a matmul. A caption this process has not encoded before costs a text-tower pass first, so a request full of fresh captions is not free; see encoders.md.
  2. Send lists. /v1/embed with 8 images is one forward pass. Client-side, Remote.embed() chunks to 8 for you.
  3. Keep batch=True (the default) if callers send single images concurrently. It turns concurrency into batch size, which is where the GPU’s throughput is.
  4. Add workers only for CPU work. uvicorn ... --workers N gives you N processes, and each one loads its own copy of the model. That is N times the GPU memory, and they contend for the same device. On one GPU, batching beats workers. Workers are the right answer when decoding is your bottleneck (large JPEGs, CPU inference) or when you have N GPUs and pin one per worker with CUDA_VISIBLE_DEVICES.
  5. Scale out. Several boxes behind the proxy, all pointed at the same heads directory on shared storage.
Heads are shared between processes through the directory, not through memory: HeadStore re-reads any file whose mtime, inode or size changed. Trained on one worker, visible from the others. Verified with two workers over the same directory:
Every worker reports the head that exactly one of them trained. Shared storage has to be real shared storage. Two nodes with separate disks are two different sets of heads, and a client that trains on one and asks on the other gets a 404 half the time.

​
Failure modes

Changing JEV_ENCODER invalidates every head. A head records the encoder that fitted it and refuses to run under another one. Silently wrong results are the alternative, and the widths often match by accident:
Roll the encoder and the heads together, or run the old encoder until every head is retrained. /v1/health tells a client which encoder it is talking to; GET /v1/heads tells it which encoder each head wants. Single-caption noul depends on the encoder. The sigmoid form needs an encoder that publishes both logit_scale and logit_bias. SigLIP does; CLIP-family models and every ensemble do not. Same request, two servers:
If your callers are not all yours, serve an encoder with a bias or tell them to use the pair form. See questions.md. Head names are global and unversioned. POST /v1/heads with an existing name overwrites it, and returns 200 either way. There is no per-caller namespace: one client’s colour is every client’s colour. If several teams share a server, prefix the names, or give each team its own process and heads directory. An unreadable head file does not disappear. A truncated or foreign .pt is recorded as broken rather than skipped, and asking for it gives a 404 whose body says the file is present and why it would not load, not “no such head”, which would be a lie about a file sitting right there. That 404 is the only place a server shows it: GET /v1/heads lists only the heads that loaded, and /v1/health’s heads count leaves it out. To see the broken ones as a set, look at the heads directory, or run jev heads --json beside it. A restart drops the caption cache, so the first request after a deploy re-encodes its captions. Milliseconds, but it shows up as a p99 spike on a busy dashboard. Timeouts on training. The default client timeout is 60 s and a real training upload exceeds it. Raise it at connect(), and at the proxy, before blaming the server for a head that appears anyway.

​
See also

← Choosing an encoder · Docs index · CLI reference →