# Embedding Service Diagnostic — Central Command Rebuild

## Problem

After the Central Command rebuild, `session_embedding` is `null` on all SESSION_CLOSE envelopes and `domain_tags` show keyword-mode confidence (~0.067) instead of embedding-mode confidence (~0.3-0.5). Embeddings were working correctly before the rebuild (TRAINING_DATA_v3 had full 384-dim vectors on every session).

The endpoint responds to direct curl tests from a developer machine, but the Groove client is falling back to keyword mode during live sessions.

## What "working" looked like (v3 data)

```
domain_tags: primary=react_frontend (0.4932)
session_embedding: 384d, model=sentence-transformers/all-MiniLM-L6-v2, nonzero=377/384
```

## What's happening now (v4 data)

```
domain_tags: primary=nlp_llm (0.0769)     ← keyword mode
session_embedding: null                     ← embed() returns null in keyword mode
```

## How the client decides between keyword and embedding mode

On every agent spawn, the `DomainTagger.init()` runs this sequence:

```
1. POST to EMBEDDING_SERVICE_URL with {"input": "health check", "model": "sentence-transformers/all-MiniLM-L6-v2"}
2. If response is 200 OK:
   a. Switch to "http" mode
   b. Call _buildCentroids() — embeds all 40 domain descriptions (40 sequential POST calls)
   c. If all succeeds → ready, embedding mode active
3. If ANYTHING fails (network error, non-200, timeout, _buildCentroids() error):
   → Fall back to "keyword" mode silently
```

The client **never logs** why it fell back — it catches all errors silently. So the failure is invisible.

## Likely failure points (in order of probability)

### 1. `_buildCentroids()` is failing

Even if the health probe succeeds, `init()` then makes **40 sequential embed calls** to build domain centroids. If any single one fails, the whole init throws and falls back to keyword mode. The entire block is wrapped in a try/catch:

```javascript
try {
  const res = await fetch(this._serviceUrl, { ... });  // health probe
  if (res.ok) {
    this._mode = this._leafRegistry?.length ? 'registry' : 'http';
    await this._buildCentroids();  // ← 40 more calls, if this throws → keyword mode
    this._ready = true;
    return;
  }
} catch {
  // silently falls through to keyword mode
}
```

**To check:** Look at Central Command logs during a Groove session. You should see ~41 POST requests to `/v1/embed` in quick succession (1 health probe + 40 centroid builds). If you see fewer, one is failing.

### 2. Timeout under load

The health probe has a **5-second timeout**. The centroid embed calls have a **10-second timeout** each. If the ONNX model is cold or the server is under load, individual calls might be timing out.

**To check:** Add timing logs to the `/v1/embed` endpoint:
```javascript
const start = Date.now();
const vector = await embed(text);
console.log(`[embed] ${Date.now() - start}ms "${text.slice(0, 50)}..."`);
```

### 3. Response format mismatch

The client parses the response as:
```javascript
const data = await res.json();
const embedding = data?.data?.[0]?.embedding;
if (!Array.isArray(embedding)) {
  // returns null — "Invalid embedding response format"
}
```

The expected response is **exactly**:
```json
{
  "data": [
    {
      "embedding": [0.0123, -0.0456, ...384 floats],
      "index": 0
    }
  ],
  "model": "sentence-transformers/all-MiniLM-L6-v2"
}
```

**To check:** Verify the response structure. Common mistakes:
- Returning `{ "embedding": [...] }` instead of `{ "data": [{ "embedding": [...] }] }`
- Returning the vector as a nested array `[[0.01, ...]]` instead of flat `[0.01, ...]`
- Returning Float32Array or typed array instead of plain JSON numbers

### 4. CORS or request rejection

The Groove daemon makes requests from `localhost` (or wherever it runs). If the rebuilt server has stricter CORS, different firewall rules, or middleware that rejects requests without a specific User-Agent or Origin header, the fetch would fail.

**To check:** The original Central Command had `Access-Control-Allow-Origin: *`. Verify the rebuild has the same.

### 5. URL path mismatch

The client hits exactly: `POST https://api.groovedev.ai/v1/embed`

**To check:** Verify the route is mounted at `/v1/embed`, not `/embed` or `/v1/embeddings` or `/api/v1/embed`.

## Quick diagnostic commands

Run these on the Central Command server:

```bash
# 1. Verify endpoint exists and responds correctly
curl -s -X POST http://localhost:8443/v1/embed \
  -H "Content-Type: application/json" \
  -d '{"input": "Python programming debugging testing"}' \
  | python3 -c "
import sys, json
d = json.load(sys.stdin)
v = d.get('data', [{}])[0].get('embedding')
if v and isinstance(v, list):
    print(f'OK: {len(v)}-dim vector, type={type(v[0]).__name__}')
else:
    print(f'BROKEN: response structure = {json.dumps(d)[:200]}')"

# 2. Simulate the full init sequence (health probe + 40 centroids)
python3 -c "
import requests, time
url = 'http://localhost:8443/v1/embed'
domains = [
    'Python programming, debugging, testing with pytest',
    'TypeScript and Node.js backend, Express frameworks',
    'React frontend development, JSX components, hooks',
    # ...add all 40 domain descriptions from domain-tagger.js
]

# Health probe
start = time.time()
r = requests.post(url, json={'input': 'health check', 'model': 'sentence-transformers/all-MiniLM-L6-v2'}, timeout=5)
print(f'Health probe: {r.status_code} in {time.time()-start:.1f}s')

# Centroid batch
fails = 0
for i, desc in enumerate(domains):
    try:
        r = requests.post(url, json={'input': desc}, timeout=10)
        d = r.json()
        v = d.get('data', [{}])[0].get('embedding')
        if not v or not isinstance(v, list) or len(v) != 384:
            print(f'  Domain {i}: BAD RESPONSE')
            fails += 1
    except Exception as e:
        print(f'  Domain {i}: FAILED - {e}')
        fails += 1

print(f'Centroid build: {len(domains)-fails}/{len(domains)} succeeded, {fails} failed')
if fails > 0:
    print('THIS IS WHY EMBEDDINGS ARE NULL - any failure causes full fallback to keyword mode')
"

# 3. Check response time under sequential load
for i in $(seq 1 10); do
  time curl -s -X POST http://localhost:8443/v1/embed \
    -H "Content-Type: application/json" \
    -d "{\"input\": \"test embedding call $i\"}" > /dev/null
done

# 4. Check server logs during a live Groove session
# Start a Groove agent and watch for /v1/embed requests in the Central Command logs
```

## What success looks like

After fix, a SESSION_CLOSE envelope should show:
```json
{
  "metadata": {
    "domain_tags": {
      "primary": { "domain": "react_frontend", "confidence": 0.49 },
      "secondary": { "domain": "vue_frontend", "confidence": 0.47 },
      "tertiary": { "domain": "typescript_node", "confidence": 0.40 }
    },
    "session_embedding": {
      "model": "sentence-transformers/all-MiniLM-L6-v2",
      "vector": [0.0123, -0.0456, "...384 floats"],
      "source_text": "frontend\nI'll scaffold the project..."
    }
  }
}
```

Confidence scores in the 0.25-0.55 range (cosine similarity), NOT 0.06-0.08 (keyword hit ratio).
