PyTorch Serving
This page covers how PyTorch powers two-tower (and general) scoring in ecosystem.Ai. The runtime stays lightweight: PyTorch runs in a notebooks sidecar, and the runtime talks to it over HTTP when you opt in to live embeddings.
In Workbench, PyTorch is an alternative training engine for the /two-tower
module. It should produce the same run metadata and Mongo embedding export shape
as the H2O engine, so downstream deployment remains engine-agnostic.
Engine options in the runtime
| Engine | Status | Two-tower suitability |
|---|---|---|
api: HTTP serving | Production | Recommended — PyTorch in a sidecar |
| DJL + TorchScript (in-JVM) | Scaffolding only | Possible later; single model slot |
| ONNX Runtime | Not present | Would be a new integration |
| Precomputed embeddings | Pattern in use | Best latency for items |
The recommended path is api: HTTP serving: the runtime calls the sidecar over
HTTP and keeps heavy ML dependencies out of the JVM.
Two api:pytorch configuration paths
1. General PyTorch model scoring
Register an HTTP-served model directly in mojo.key:
mojo.key=api:pytorch:http://<notebooks-host>:8010:<model_id>The runtime sends feature instances to the sidecar and normalizes the response into the canonical score shape.
2. Two-tower user-tower embedding
For two-tower, the user vector can be computed live by the sidecar when no precomputed vector exists:
predictor.model.type=similarity
predictor.twotower.user.embed=pytorch:http://<notebooks-host>:8010:<model_id>PrePredictTwoTower requests a user embedding from the sidecar when needed. Item vectors stay precomputed, so there is one sidecar call per request, not one per offer. Timeouts are bounded so a hung sidecar cannot block runtime request threads indefinitely.
Training and export in Workbench
- On
/two-tower, setenginetopytorch_notebookson the saved configuration. - Run Train (same flow as
POST /public/v1/two-tower/train). - Run Concept test, then Export embeddings for runtime so MongoDB holds normalized user and item vectors for PrePredictTwoTower.
For production, prefer exported Mongo embeddings over live sidecar calls for every
customer. Use predictor.twotower.user.embed only when user vectors must be
computed from fresh request features.
The pytorch_mlrun engine value is rejected at request validation (HTTP
422). Use pytorch_notebooks for PyTorch two-tower training in Workbench.
Workbench algorithm and public API request shapes are in the API Reference. Runtime scoring properties are in Real-Time Scoring.