{"_id":"@codecell-germany/meta-sam-3-1-image-agent-skill","name":"@codecell-germany/meta-sam-3-1-image-agent-skill","dist-tags":{"latest":"0.1.0"},"versions":{"0.1.0":{"name":"@codecell-germany/meta-sam-3-1-image-agent-skill","version":"0.1.0","private":false,"description":"Agent-first Meta SAM 3.1 skill payload and publish metadata for image reduction, OCR preparation, and downstream visual analysis workflows","keywords":["meta","sam3.1","sam3","segment-anything","image-segmentation","ocr","agent","codex-skill","computer-vision"],"license":"MIT","author":{"name":"Nikolas Gottschol"},"repository":{"type":"git","url":"git+https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill.git"},"homepage":"https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill#readme","bugs":{"url":"https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill/issues"},"bin":{"sam3-image-skill":"scripts/npm-skill-installer.mjs"},"scripts":{"pack:dry-run":"npm pack --dry-run","test:unit":"node --test tests/unit/*.test.mjs","test":"npm run test:unit && npm run test:release","test:release":"node scripts/release-smoke.mjs"},"publishConfig":{"access":"public"},"engines":{"node":">=20"},"gitHead":"81da2a8ac3eb2f27c89a0e9765719d96a30ba40c","_id":"@codecell-germany/meta-sam-3-1-image-agent-skill@0.1.0","_nodeVersion":"25.9.0","_npmVersion":"11.12.1","dist":{"integrity":"sha512-8071ixY4ZB3Czn5Sf7JJ6q0KBXFvFv9EG3mSEFMKnCGmwACF7dnUTWTqKeoO0upVJrAISTu0/E41YBMrafQBpQ==","shasum":"87560a9bd6edf41494fd0c08ceade2f5cd23305b","tarball":"https://registry.npmjs.org/@codecell-germany/meta-sam-3-1-image-agent-skill/-/meta-sam-3-1-image-agent-skill-0.1.0.tgz","fileCount":84,"unpackedSize":3078772,"signatures":[{"keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U","sig":"MEUCIG9v0OuwN/qn3ZfFb1lGO6o13PTjfYZGn2m7Y8ssFn1tAiEAwChx/MQVK+DnJAMFdB4t7vNi0/OtQomd746PICkA5Fw="}]},"_npmUser":{"name":"codecell-germany","email":"admin@codecell.de"},"directories":{},"maintainers":[{"name":"codecell-germany","email":"admin@codecell.de"}],"_npmOperationalInternal":{"host":"s3://npm-registry-packages-npm-production","tmp":"tmp/meta-sam-3-1-image-agent-skill_0.1.0_1776330232237_0.9780304769744168"},"_hasShrinkwrap":false}},"time":{"created":"2026-04-16T09:03:52.166Z","0.1.0":"2026-04-16T09:03:52.464Z","modified":"2026-04-16T09:03:52.631Z"},"maintainers":[{"name":"codecell-germany","email":"admin@codecell.de"}],"description":"Agent-first Meta SAM 3.1 skill payload and publish metadata for image reduction, OCR preparation, and downstream visual analysis workflows","homepage":"https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill#readme","keywords":["meta","sam3.1","sam3","segment-anything","image-segmentation","ocr","agent","codex-skill","computer-vision"],"repository":{"type":"git","url":"git+https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill.git"},"author":{"name":"Nikolas Gottschol"},"bugs":{"url":"https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill/issues"},"license":"MIT","readme":"# meta-sam-3-1-image-agent-skill\n\n---\n\n# English\n\n## Purpose\n\n`meta-sam-3-1-image-agent-skill` turns Meta SAM 3.1 into an agent-friendly image reduction layer.\n\nThe practical goal is not only to segment images, but to help agents work better with images afterward:\n\n- better OCR on relevant image regions instead of the whole frame\n- better object- or component-level analysis through crops\n- better downstream automation through structured JSON metadata\n- better iteration because multiple segmentation runs can coexist without overwriting each other\n\nThis repository combines:\n\n- the Meta SAM 3.1 model codebase\n- a local CLI surface built around `sam3-cli`\n- a skill payload under `skills/sam3-image-cli/`\n- setup guidance for macOS Apple Silicon and Windows with NVIDIA/CUDA\n\n## Public release surfaces\n\n- GitHub repo:\n  `codecell-germany/meta-sam-3-1-image-agent-skill`\n- npm package:\n  `@codecell-germany/meta-sam-3-1-image-agent-skill`\n- CLI binary:\n  `sam3-cli`\n- skill installer binary:\n  `sam3-image-skill`\n- skill name:\n  `sam3-image-cli`\n\n## What the repository contains\n\n- a local segmentation CLI\n- a skill that teaches agents how to install and use it\n- platform-specific setup guidance\n- references for onboarding and output handling\n- knowledge files for architecture, platform support, limitations, and release hygiene\n\n## Why this matters for agents\n\nMany image tasks get better when the original image is reduced to the most relevant regions first.\n\nInstead of sending one noisy image into OCR or visual reasoning, an agent can:\n\n1. segment the image with Meta SAM 3.1\n2. keep the overlay for quick validation\n3. use the JSON file for structured routing\n4. run OCR or visual analysis on the generated crops\n\nThat creates a new operating mode for agent workflows:\nless noise, smaller image regions, clearer semantics, and better follow-up automation.\n\n## Supported platform strategy\n\n- macOS Apple Silicon:\n  verified image segmentation path via `cpu`\n- Windows with NVIDIA/CUDA:\n  intended high-performance path via `cuda`\n\nImportant:\n\n- this repo is deliberately positioned around image segmentation\n- video and multiplex tracking are not the default product story here\n- model weights are not stored in this repository\n\n## Installation\n\n### 1. Clone the repository\n\n```bash\ngit clone https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill.git\ncd meta-sam-3-1-image-agent-skill\n```\n\n### 2. Create and activate a virtual environment\n\nmacOS or Linux:\n\n```bash\npython3.12 -m venv .venv\nsource .venv/bin/activate\n```\n\nWindows PowerShell:\n\n```powershell\npy -3.12 -m venv .venv\n.venv\\Scripts\\activate\n```\n\n### 3. Install PyTorch\n\nmacOS Apple Silicon:\n\n```bash\npip install torch==2.10.0 torchvision==0.25.0\n```\n\nWindows with CUDA:\n\n```powershell\npip install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu128\n```\n\n### 4. Install the repository\n\n```bash\npip install -e .\npip install einops\n```\n\n### 5. Verify the CLI\n\n```bash\nsam3-cli --help\nsam3-cli doctor\n```\n\n### 6. Optionally install the skill payload for Codex\n\n```bash\nnpm install -g @codecell-germany/meta-sam-3-1-image-agent-skill\nsam3-image-skill install --force\n```\n\nOr without a global npm install:\n\n```bash\nnpx -y -p @codecell-germany/meta-sam-3-1-image-agent-skill sam3-image-skill install --force\n```\n\nThis installs:\n\n- the skill payload under `~/.codex/skills/sam3-image-cli`\n- the runtime files under `~/.codex/tools/sam3-image-cli`\n- CLI shims under `~/.codex/bin/`\n  - `sam3-cli`\n  - `sam3-cli.cmd`\n  - `sam3-cli.ps1`\n\n## Canonical first-run sequence\n\n### 1. Verify the public CLI\n\n```bash\nsam3-cli --help\n```\n\n### 2. Check the environment\n\n```bash\nsam3-cli doctor\n```\n\n### 3. If setup is incomplete, print the setup guide\n\n```bash\nsam3-cli setup --language en\nsam3-cli setup --platform macos-apple-silicon --language en\nsam3-cli setup --platform windows-cuda --language en\n```\n\n### 4. Make sure Hugging Face access exists\n\nThe user must have approved access to the gated model:\n\n- `facebook/sam3.1`\n\n### 5. Download the checkpoint\n\n```bash\nsam3-cli download --version sam3.1\n```\n\n### 6. Run the first real segmentation\n\nmacOS Apple Silicon:\n\n```bash\nsam3-cli image --version sam3.1 --device cpu --image /absolute/path/image.jpg --prompt \"object of interest\"\n```\n\nWindows with CUDA:\n\n```powershell\nsam3-cli image --version sam3.1 --device cuda --image C:\\absolute\\path\\image.jpg --prompt \"object of interest\"\n```\n\n## Quick start\n\n```bash\nsam3-cli setup --language en\nsam3-cli doctor\nsam3-cli download --version sam3.1\nsam3-cli image --version sam3.1 --device cpu --image /absolute/path/image.jpg --prompt \"object of interest\"\n```\n\n## Output contract\n\nEvery successful segmentation run produces:\n\n- an overlay PNG\n- a JSON metadata file\n- a crop directory\n\nThe JSON metadata preserves bounding boxes, scores, crop paths, and other run settings.\nThe crop directory is usually the best follow-up input for OCR or fine-grained visual analysis.\n\n## Important parameters\n\n- `--device auto|cpu|cuda|mps`\n- `--threshold`\n- `--mask-threshold`\n- `--resolution`\n- `--top-k`\n- `--crop-padding`\n- `--alpha`\n- `--checkpoint`\n\nExample:\n\n```bash\nsam3-cli image \\\n  --version sam3.1 \\\n  --device cpu \\\n  --threshold 0.5 \\\n  --mask-threshold 0.5 \\\n  --resolution 1008 \\\n  --top-k 0 \\\n  --crop-padding 5 \\\n  --alpha 120 \\\n  --image /absolute/path/image.jpg \\\n  --prompt \"object of interest\"\n```\n\n## Agent guardrails\n\n- use `sam3-cli` instead of ad-hoc Python entry points\n- run `doctor` before the first real segmentation in a new shell\n- do not assume model weights are bundled with the repo\n- on Apple Silicon, prefer `cpu`\n- prefer the JSON file and crop directory over the overlay alone for downstream work\n- do not commit weights, private input images, or generated outputs\n\n## Hugging Face and model access\n\nMeta SAM 3.1 weights are hosted separately from this repository.\nThis repo does not ship them.\n\nIf `sam3-cli download --version sam3.1` fails with `401`, treat that as an access or authentication issue against the gated Hugging Face model first.\n\n## Known limitations\n\n- macOS Apple Silicon is currently a stability-first path via `cpu`\n- `mps` should not be treated as the production default yet\n- video and multiplex workflows are not the default product surface in this repository\n- the image path can emit checkpoint `missing_keys` warnings while still producing correct output\n\n## Skills ecosystem\n\nRepository listing check:\n\n```bash\nnpx -y skills add codecell-germany/meta-sam-3-1-image-agent-skill -l\n```\n\nGlobal install example:\n\n```bash\nnpx -y skills add codecell-germany/meta-sam-3-1-image-agent-skill -g --skill sam3-image-cli -a '*' -y\n```\n\nnpm skill installer example:\n\n```bash\nnpm install -g @codecell-germany/meta-sam-3-1-image-agent-skill\nsam3-image-skill install --force\n```\n\nDirect npx installer example:\n\n```bash\nnpx -y -p @codecell-germany/meta-sam-3-1-image-agent-skill sam3-image-skill install --force\n```\n\n## Release verification\n\n```bash\nsam3-cli --help\nsam3-cli setup --language en\nsam3-cli doctor\nnpm run test:unit\nnpm pack --dry-run\nnpm run test:release\n```\n\n## References\n\n- `skills/sam3-image-cli/SKILL.md`\n- `skills/sam3-image-cli/references/overview.md`\n- `skills/sam3-image-cli/references/agent-onboarding.md`\n- `skills/sam3-image-cli/references/command-cheatsheet.md`\n- `skills/sam3-image-cli/references/macos-first-run.md`\n- `skills/sam3-image-cli/references/windows-cuda-first-run.md`\n- `skills/sam3-image-cli/references/output-contract.md`\n- `knowledge/ARCHITECTURE.md`\n- `knowledge/PLATFORM_SUPPORT.md`\n- `knowledge/OUTPUT_CONTRACT.md`\n- `knowledge/KNOWN_LIMITATIONS.md`\n- `knowledge/RELEASE_CHECKLIST.md`\n\n---\n\n# Deutsch\n\n## Zweck\n\n`meta-sam-3-1-image-agent-skill` macht aus Meta SAM 3.1 eine agententaugliche Bild-Reduktionsschicht.\n\nDas praktische Ziel ist nicht nur Segmentierung, sondern bessere Folgearbeit mit Bildern:\n\n- bessere OCR auf relevanten Bildausschnitten statt auf dem ganzen Bild\n- bessere Objekt- oder Komponentenanalysen über Crops\n- bessere Automatisierung über strukturierte JSON-Metadaten\n- bessere Vergleichbarkeit, weil mehrere Segmentierungsläufe nebeneinander existieren können, ohne sich zu überschreiben\n\nDieses Repo kombiniert:\n\n- den Meta-SAM-3.1-Modellcode\n- eine lokale CLI-Oberfläche rund um `sam3-cli`\n- einen Skill unter `skills/sam3-image-cli/`\n- Setup-Anleitungen für macOS auf Apple Silicon und Windows mit NVIDIA/CUDA\n\n## Öffentliche Release-Oberflächen\n\n- GitHub-Repo:\n  `codecell-germany/meta-sam-3-1-image-agent-skill`\n- npm-Paket:\n  `@codecell-germany/meta-sam-3-1-image-agent-skill`\n- CLI-Binary:\n  `sam3-cli`\n- Skill-Installer-Binary:\n  `sam3-image-skill`\n- Skill-Name:\n  `sam3-image-cli`\n\n## Was das Repo enthält\n\n- eine lokale Segmentierungs-CLI\n- einen Skill, der Agenten die Installation und Nutzung erklärt\n- plattformspezifische Setup-Anleitungen\n- Referenzen für Onboarding und Ergebnisweiterverarbeitung\n- Knowledge-Dateien für Architektur, Plattformsupport, Grenzen und Release-Hygiene\n\n## Warum das für Agenten wichtig ist\n\nViele Bildaufgaben werden besser, wenn das Originalbild zuerst auf die relevantesten Regionen reduziert wird.\n\nStatt ein volles, verrauschtes Bild direkt an OCR oder visuelles Reasoning zu geben, kann ein Agent:\n\n1. das Bild mit Meta SAM 3.1 segmentieren\n2. das Overlay für eine schnelle Sichtprüfung behalten\n3. die JSON-Datei für strukturierte Weiterleitung nutzen\n4. OCR oder Bildanalyse auf den erzeugten Crops ausführen\n\nDadurch entsteht ein neuer Arbeitsmodus für Agenten:\nweniger Rauschen, kleinere Bildregionen, klarere Semantik und bessere Folgeautomatisierung.\n\n## Unterstützte Plattformstrategie\n\n- macOS Apple Silicon:\n  verifizierter Bildpfad über `cpu`\n- Windows mit NVIDIA/CUDA:\n  geplanter Hochleistungs-Pfad über `cuda`\n\nWichtig:\n\n- dieses Repo ist bewusst auf Bildsegmentierung ausgerichtet\n- Video und Multiplex-Tracking sind hier nicht die Standard-Produktgeschichte\n- Modellgewichte liegen nicht in diesem Repo\n\n## Installation\n\n### 1. Repo klonen\n\n```bash\ngit clone https://github.com/codecell-germany/meta-sam-3-1-image-agent-skill.git\ncd meta-sam-3-1-image-agent-skill\n```\n\n### 2. Virtuelle Umgebung anlegen und aktivieren\n\nmacOS oder Linux:\n\n```bash\npython3.12 -m venv .venv\nsource .venv/bin/activate\n```\n\nWindows PowerShell:\n\n```powershell\npy -3.12 -m venv .venv\n.venv\\Scripts\\activate\n```\n\n### 3. PyTorch installieren\n\nmacOS Apple Silicon:\n\n```bash\npip install torch==2.10.0 torchvision==0.25.0\n```\n\nWindows mit CUDA:\n\n```powershell\npip install torch==2.10.0 torchvision --index-url https://download.pytorch.org/whl/cu128\n```\n\n### 4. Repo installieren\n\n```bash\npip install -e .\npip install einops\n```\n\n### 5. CLI prüfen\n\n```bash\nsam3-cli --help\nsam3-cli doctor\n```\n\n### 6. Optional den Skill-Payload für Codex installieren\n\n```bash\nnpm install -g @codecell-germany/meta-sam-3-1-image-agent-skill\nsam3-image-skill install --force\n```\n\nOder ohne globale npm-Installation:\n\n```bash\nnpx -y -p @codecell-germany/meta-sam-3-1-image-agent-skill sam3-image-skill install --force\n```\n\nDabei werden installiert:\n\n- der Skill-Payload unter `~/.codex/skills/sam3-image-cli`\n- die Runtime-Dateien unter `~/.codex/tools/sam3-image-cli`\n- CLI-Shims unter `~/.codex/bin/`\n  - `sam3-cli`\n  - `sam3-cli.cmd`\n  - `sam3-cli.ps1`\n\n## Kanonische First-Run-Reihenfolge\n\n### 1. Öffentliche CLI prüfen\n\n```bash\nsam3-cli --help\n```\n\n### 2. Umgebung prüfen\n\n```bash\nsam3-cli doctor\n```\n\n### 3. Wenn das Setup unvollständig ist, Setup-Guide ausgeben\n\n```bash\nsam3-cli setup --language de\nsam3-cli setup --platform macos-apple-silicon --language de\nsam3-cli setup --platform windows-cuda --language de\n```\n\n### 4. Hugging-Face-Zugriff sicherstellen\n\nDer Nutzer braucht freigeschalteten Zugriff auf das gated Modell:\n\n- `facebook/sam3.1`\n\n### 5. Checkpoint herunterladen\n\n```bash\nsam3-cli download --version sam3.1\n```\n\n### 6. Erste echte Segmentierung ausführen\n\nmacOS Apple Silicon:\n\n```bash\nsam3-cli image --version sam3.1 --device cpu --image /absoluter/pfad/bild.jpg --prompt \"object of interest\"\n```\n\nWindows mit CUDA:\n\n```powershell\nsam3-cli image --version sam3.1 --device cuda --image C:\\absoluter\\pfad\\bild.jpg --prompt \"object of interest\"\n```\n\n## Schnellstart\n\n```bash\nsam3-cli setup --language de\nsam3-cli doctor\nsam3-cli download --version sam3.1\nsam3-cli image --version sam3.1 --device cpu --image /absoluter/pfad/bild.jpg --prompt \"object of interest\"\n```\n\n## Ergebnisvertrag\n\nJeder erfolgreiche Segmentierungslauf erzeugt:\n\n- ein Overlay-PNG\n- eine JSON-Metadatendatei\n- einen Crop-Ordner\n\nDie JSON-Datei enthält Bounding-Boxes, Scores, Crop-Pfade und Laufparameter.\nDer Crop-Ordner ist meist der beste Folgeinput für OCR oder feinkörnige Bildanalyse.\n\n## Wichtige Parameter\n\n- `--device auto|cpu|cuda|mps`\n- `--threshold`\n- `--mask-threshold`\n- `--resolution`\n- `--top-k`\n- `--crop-padding`\n- `--alpha`\n- `--checkpoint`\n\nBeispiel:\n\n```bash\nsam3-cli image \\\n  --version sam3.1 \\\n  --device cpu \\\n  --threshold 0.5 \\\n  --mask-threshold 0.5 \\\n  --resolution 1008 \\\n  --top-k 0 \\\n  --crop-padding 5 \\\n  --alpha 120 \\\n  --image /absoluter/pfad/bild.jpg \\\n  --prompt \"object of interest\"\n```\n\n## Agentische Guardrails\n\n- `sam3-cli` statt ad-hoc Python-Einstiegspunkte verwenden\n- vor der ersten echten Segmentierung in einer neuen Shell `doctor` ausführen\n- nicht annehmen, dass Modellgewichte im Repo enthalten sind\n- auf Apple Silicon `cpu` bevorzugen\n- für Folgearbeit JSON und Crop-Ordner dem Overlay vorziehen\n- keine Gewichte, privaten Bilder oder generierten Outputs committen\n\n## Hugging Face und Modellzugriff\n\nDie Meta-SAM-3.1-Gewichte werden separat von diesem Repo gehostet.\nDieses Repo liefert sie nicht mit.\n\nWenn `sam3-cli download --version sam3.1` mit `401` scheitert, sollte das zuerst als Zugriffs- oder Authentifizierungsproblem gegen das gated Hugging-Face-Modell behandelt werden.\n\n## Bekannte Grenzen\n\n- macOS Apple Silicon ist aktuell ein Stabilitätspfad über `cpu`\n- `mps` ist derzeit kein belastbarer Produktions-Default\n- Video- und Multiplex-Workflows sind hier nicht die Standard-Produktoberfläche\n- der Bildpfad kann `missing_keys`-Warnungen beim Checkpoint-Laden ausgeben und trotzdem korrekt funktionieren\n\n## Skills-Ökosystem\n\nRepo-Listing prüfen:\n\n```bash\nnpx -y skills add codecell-germany/meta-sam-3-1-image-agent-skill -l\n```\n\nGlobales Installationsbeispiel:\n\n```bash\nnpx -y skills add codecell-germany/meta-sam-3-1-image-agent-skill -g --skill sam3-image-cli -a '*' -y\n```\n\nnpm-Skill-Installer-Beispiel:\n\n```bash\nnpm install -g @codecell-germany/meta-sam-3-1-image-agent-skill\nsam3-image-skill install --force\n```\n\nDirektes `npx`-Installer-Beispiel:\n\n```bash\nnpx -y -p @codecell-germany/meta-sam-3-1-image-agent-skill sam3-image-skill install --force\n```\n\n## Release-Verifikation\n\n```bash\nsam3-cli --help\nsam3-cli setup --language de\nsam3-cli doctor\nnpm run test:unit\nnpm pack --dry-run\nnpm run test:release\n```\n\n## Referenzen\n\n- `skills/sam3-image-cli/SKILL.md`\n- `skills/sam3-image-cli/references/overview.md`\n- `skills/sam3-image-cli/references/agent-onboarding.md`\n- `skills/sam3-image-cli/references/command-cheatsheet.md`\n- `skills/sam3-image-cli/references/macos-first-run.md`\n- `skills/sam3-image-cli/references/windows-cuda-first-run.md`\n- `skills/sam3-image-cli/references/output-contract.md`\n- `knowledge/ARCHITECTURE.md`\n- `knowledge/PLATFORM_SUPPORT.md`\n- `knowledge/OUTPUT_CONTRACT.md`\n- `knowledge/KNOWN_LIMITATIONS.md`\n- `knowledge/RELEASE_CHECKLIST.md`\n","readmeFilename":"README.md","_rev":"1-a5ebfb7f46a083e38ab5a050aa4a71c7"}