MolmoAct2-Think serving — vision-language-ACTION (8th modality)

Target consumer: apps/monster-studio (the Thoth body — embodied / body-agency). Pattern mirrors the existing mesh serving (openai-server.ts → fat-station), made OpenAI-compatible with extra params.

Layers

monster-studio ──OpenAI chat (image + instruction + params)──▶ openai-server (TS)
                                                                  │  decode+resize+normalize image
                                                                  │  tokenize instruction (qwen BPE)
                                                                  ▼
                                                              fat-station POST /action
                                                                  │  ViT+adapter → LM prefill
                                                                  │  → 100 AR depth tokens
                                                                  │  → extract per-layer KV
                                                                  │  → flow-matching action expert
                                                                  ▼
                                                              { action[H×D], horizon, dim }

1. fat-station `POST /action` (implemented)

Low-level primitive. Pre-processed pixels in, action chunk out.

Request:

{ "pixels": [f32 ...],   // CHW, 3×378×378 = 430074 floats, SigLIP-normalized ((px/255)-0.5)/0.5
  "tokens": [u32 ...],    // instruction token ids (qwen BPE, < 151643)
  "horizon": 30,          // action steps (<= 30)
  "seed": 42 }            // flow-noise seed (deterministic)

Response:

{ "action": [f32 ...],   // horizon*dim row-major (model space, pre-unnormalization)
  "horizon": 30, "dim": 32 }

action is the flow-integrated chunk in model space ([-1,1]-ish). To get physical robot actions, q01_q99-unnormalize per tag (norm_stats.json): a_phys = clip(a,-1,1) ; (a+1)*(q99-q01)/2 + q01 over the real action dims. For the monster-studio body-agency use, model space is fine as a normalized control signal (map to body joint deltas directly).

2. openai-server `POST /v1/chat/completions` (to wire — OpenAI-compat with params)

Detect model == "molmoact2-think". The user message carries the image (OpenAI vision format) + instruction; extra control via a top-level molmo block.

Request (OpenAI-compatible):

{ "model": "molmoact2-think",
  "messages": [
    { "role": "user", "content": [
        { "type": "text", "text": "pick up the red block" },
        { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,<...>" } }
    ]}
  ],
  "molmo": { "horizon": 30, "seed": 42 }   // optional params; sensible defaults
}

Server steps:

Extract the data-URL image → decode → resize to 378×378 → CHW f32 → SigLIP normalize.
Tokenize the text instruction with the qwen tokenizer (TOKENIZER_GGUF_PATH).
POST fat-station /action { pixels, tokens, horizon, seed }.
Format the OpenAI response, carrying the structured action chunk.

Response (OpenAI shape + _action extension):

{ "id": "chatcmpl-...", "object": "chat.completion", "model": "molmoact2-think",
  "choices": [{ "index": 0, "finish_reason": "stop",
    "message": { "role": "assistant",
      "content": "action chunk: 30 steps × 32 dims (model space)" } }],
  "_action": { "horizon": 30, "dim": 32, "space": "model",
               "action": [[f32×32], ...30 rows] } }

The _action field is the machine-readable payload monster-studio consumes; the content string keeps generic OpenAI clients happy.

3. monster-studio client (to wire)

The Thoth scene already has a frame/canvas. Per control tick (or on command):

Render the current view (or camera) to a 378×378 canvas; read RGBA → RGB.
Either: (a) normalize client-side to CHW f32 and call fat-station /action directly, or (b) base64 the frame and call openai-server /v1/chat/completions with model:"molmoact2-think" (OpenAI-compat path).
Map the returned action chunk to body-agency: the 32-dim vector per step → normalized joint/end-effector deltas driving the humanoid mesh over the horizon.

Default deployment: a dedicated Cloud Run service gnosis-openai-molmoact2-think (KNOT_URL=…/models/molmoact2-think.knot, TOKENIZER_GGUF_URL=…/qwen-tokenizer.gguf, MODEL_NAME=molmoact2-think), entrypoint openai-server, same chunked-Range knot download + RAM-resident mmap as the text models.

Status

fat-station /action: implemented (Rust).
openai-server molmo path + monster-studio client: specified here; wire after the action gate passes (knot encode → action-probe).

forkjoin-ai/gnosis