---
name: sophnet-pdf
description: Comprehensive PDF processing for creating, editing, and analyzing PDF documents. Use when the user mentions PDF files, needs to create new PDFs (reports, invoices, certificates), edit existing PDFs (text replacement, image operations, watermarks), extract content, merge/split files, or perform OCR on scanned documents.
---

# PDF Processing Guide

## MANDATORY: Working Directory

**EVERY command in this skill MUST be executed from THIS skill's directory.** Before running ANY command — Python or bash script — you MUST `cd` into this skill's directory first. Determine the absolute path of this `SKILL.md` file and use its parent directory.

```bash
SKILL_DIR="<absolute-path-to-this-skills-sophnet-pdf-directory>"
cd "$SKILL_DIR"
```

**NEVER run commands from the repository root or any other directory.** If you do, `uv run` won't find `pyproject.toml`, and `python` won't have access to required packages.

## Overview

This guide covers essential PDF processing operations using Python libraries and command-line tools. For advanced features, JavaScript libraries, and detailed examples, see REFERENCE.md. If you need to fill out a PDF form, read FORMS.md and follow its instructions.

### ⚠️ CJK Text in PDFs — READ FIRST

**If the PDF will contain ANY Chinese/Japanese/Korean text** (including titles, body text, table cells, or watermarks), you MUST handle CJK fonts correctly:

- **reportlab**: Default fonts (`Helvetica`, `Times-Roman`) render CJK as **black boxes (█)**. You MUST register `STSong-Light` and call `addMapping()` before creating any content. See the "reportlab - Create PDFs" section for the exact setup code.
- **PyMuPDF (fitz)**: Use `fontname="china-s"` for `insert_text()` / `insert_textbox()`. This is built-in and requires no extra setup.
- **Rule of thumb**: When in doubt, always use CJK-safe fonts. `STSong-Light` (reportlab) and `china-s` (PyMuPDF) both handle mixed CJK+Latin text correctly.

## Python Runtime (uv)

**CRITICAL: All Python execution in this skill MUST use `uv run --project .` from the skill directory. NEVER use bare `python3`, `python`, or `pip install` directly — the required packages (pdfplumber, reportlab, pypdf, etc.) are ONLY available inside the uv virtual environment defined by this skill's `pyproject.toml`. Direct `python3` will fail with ModuleNotFoundError.**

**IMPORTANT: Only use libraries available in this skill's pyproject.toml (reportlab, pypdf, pdfplumber, pillow, pdf2image, pymupdf). Do NOT import matplotlib, numpy, pandas, or other packages not listed — they will cause ModuleNotFoundError. For charts/graphs, use `reportlab.graphics.charts` (VerticalBarChart, HorizontalBarChart, Pie, etc.) instead of matplotlib.**

First, ensure the environment is set up (run once per session):

```bash
cd "$SKILL_DIR"
bash scripts/ensure_uv_env.sh
```

Then ALL Python commands must use this prefix:

```bash
cd "$SKILL_DIR" && uv run --project . python <script-or-module>
```

This applies to both the provided scripts AND any inline Python code you write. For inline code, use:

```bash
cd "$SKILL_DIR" && uv run --project . python -c "import pdfplumber; ..."
```

## Delivery

Local PDF creation/editing/analysis does not require any Sophnet API key.

**IMPORTANT: After creating or modifying a PDF, ALWAYS upload it and return the download URL to the user.** This is the default behavior — do not skip the upload step.

```bash
bash scripts/upload_file.sh --file <absolute-path-to-pdf>
```

Upload command output contract:

- `FILE_PATH=<absolute-path>`
- `UPLOAD_STATUS=uploaded|skipped`
- `DOWNLOAD_URL=<https://...>` (present only when uploaded)

Delivery rules:

- **ALWAYS call `scripts/upload_file.sh` after producing a PDF file**, then return the `DOWNLOAD_URL` to the user.
- If `UPLOAD_STATUS=uploaded`, return the exact `DOWNLOAD_URL` value to the user as a clickable link.
- If `UPLOAD_STATUS=skipped` (missing API key), return `FILE_PATH` instead of failing the whole task.
- Keep URL output logic independent inside `sophnet-pdf/scripts`. Do not call other skills' upload scripts.

## Quick Start

```python
from pypdf import PdfReader, PdfWriter

# Read a PDF
reader = PdfReader("document.pdf")
print(f"Pages: {len(reader.pages)}")

# Extract text
text = ""
for page in reader.pages:
    text += page.extract_text()
```

## Python Libraries

### PyMuPDF (fitz) - Text Search & Replace

**This is the ONLY method for replacing text in existing PDFs. Do NOT fall back to nano-pdf, pdftotext, tesseract, or any external CLI tool — use PyMuPDF exclusively.**

**CRITICAL FONT RULES:**
- NEVER use the original PDF's embedded font names (e.g. `Unnamed-T3`, `NotoSansCJKsc-Regular`, `LiberationSans`) with `insert_text()` — they are embedded fonts and will cause `Exception: need font file or buffer`.
- Always use `page.search_for()` to locate text (it works across spans, unlike iterating `get_text("dict")` spans which may split multi-word terms).
- Use the **fallback font picker** below to choose a suitable built-in font.

**Built-in font aliases (Latin):** `helv` (Helvetica), `heit` (Helvetica-Bold), `tiro` (Times-Roman), `tibo` (Times-Bold), `cour` (Courier), `cobo` (Courier-Bold).

**Built-in CJK font aliases:** `china-s` (Simplified Chinese), `china-t` (Traditional Chinese), `japan` (Japanese), `korea` (Korean). These also render Latin characters correctly.

#### Find & Replace Text in PDF (Complete Example)

```python
import fitz  # PyMuPDF

# --- Font fallback logic ---
BUILTIN_FONTS = {"helv","heit","tiro","tibo","tibi","toit","cour","cobo","cobi","cout","symb","zadb"}
CJK_FONTS = {"china-s","china-ss","china-t","china-ts","japan","japan-s","korea","korea-s"}

def has_cjk(text):
    """Check if text contains CJK characters."""
    return any("\u4e00" <= c <= "\u9fff" or "\u3000" <= c <= "\u303f" for c in text)

def pick_font(original_font, text):
    """Pick a built-in font. Falls back from the original embedded font name."""
    if original_font in BUILTIN_FONTS or original_font in CJK_FONTS:
        return original_font
    if has_cjk(text):
        return "china-s"  # handles both CJK and Latin
    lower = original_font.lower()
    if "bold" in lower:
        return "heit"
    if "mono" in lower or "courier" in lower or "code" in lower:
        return "cour"
    return "helv"

def get_span_info(page, rect):
    """Get font size, font name, and color from the span overlapping rect."""
    for block in page.get_text("dict")["blocks"]:
        if "lines" not in block:
            continue
        for line in block["lines"]:
            for span in line["spans"]:
                if fitz.Rect(span["bbox"]).intersects(rect):
                    return span["size"], span["font"], span.get("color", 0)
    return 12, "helv", 0

# --- Main replacement logic ---
search_terms = ["OldText", "Old Text"]  # all variants to search for
replacement  = "NewText"

doc = fitz.open("input.pdf")

for page in doc:
    # Step 1: Find all instances using search_for (works across spans)
    hits = []
    for term in search_terms:
        for rect in page.search_for(term):
            size, font, color = get_span_info(page, rect)
            hits.append({"rect": rect, "size": size, "font": font, "color": color})

    # Step 2: Redact (erase) old text
    for h in hits:
        page.add_redact_annot(h["rect"], fill=(1, 1, 1))
    page.apply_redactions()

    # Step 3: Insert replacement text with fallback font
    for h in hits:
        font = pick_font(h["font"], replacement)
        c = h["color"]
        rgb = (((c>>16)&0xFF)/255, ((c>>8)&0xFF)/255, (c&0xFF)/255) if isinstance(c, int) else (0,0,0)
        page.insert_text(
            (h["rect"].x0, h["rect"].y0 + h["size"] * 0.85),
            replacement,
            fontname=font,
            fontsize=h["size"],
            color=rgb,
        )

doc.save("output.pdf", garbage=4, deflate=True)
```

**After replacement, always verify** by extracting text from the output PDF:

```python
doc2 = fitz.open("output.pdf")
for i, p in enumerate(doc2):
    text = p.get_text()
    old_count = text.lower().count("oldtext")
    new_count = text.lower().count("newtext")
    print(f"Page {i+1}: OldText={old_count}, NewText={new_count}")
```

**Notes:**
- PDF text replacement is inherently imperfect — PDFs are presentation-focused, not text-edit-focused.
- The replacement font may look slightly different from the original embedded font. This is expected.
- `china-s` is the best fallback for mixed CJK+Latin text; `helv` for Latin-only text.

### PyMuPDF (fitz) - Image Operations

**Use PyMuPDF for ALL image operations in existing PDFs.** Key APIs:
- `page.insert_image(rect, filename=...)` — insert a new image
- `page.replace_image(xref, filename=...)` — replace an existing image in-place (keeps original rect)
- `page.delete_image(xref)` — remove an image (replaces with 1×1 transparent pixel)
- `page.get_images(full=True)` — list images on a page (returns xref, etc.)
- `page.get_image_info(xrefs=True)` — list images with bounding box info

**CRITICAL: Always use `page.get_image_info(xrefs=True)` (not `get_images`) when you need the on-page bounding box of each image.**

#### Insert Image into Existing PDF

```python
import fitz

doc = fitz.open("input.pdf")
page = doc[0]

# Define target rectangle (x0, y0, x1, y1) — image will be scaled to fit
# To center horizontally with 50% page width:
pw = page.rect.width
img_w = pw * 0.5
# Load image to get aspect ratio
img = fitz.Pixmap("image.png")
aspect = img.width / img.height
img_h = img_w / aspect
x0 = (pw - img_w) / 2
y0 = 700  # vertical position

rect = fitz.Rect(x0, y0, x0 + img_w, y0 + img_h)
page.insert_image(rect, filename="image.png")
doc.save("output.pdf")
```

#### Replace an Existing Image

```python
import fitz

doc = fitz.open("input.pdf")
page = doc[0]

# Find the image to replace
images = page.get_images(full=True)
target_xref = images[0][0]  # xref of the first image

# Replace it — the new image fills the SAME rect as the original
page.replace_image(target_xref, filename="new_image.png")
doc.save("output.pdf")
```

**Note:** `replace_image` keeps the original bounding box. The new image is scaled to fit.

#### Modify Image Layout (Reposition / Resize)

```python
import fitz

doc = fitz.open("input.pdf")
page = doc[0]

# Get image placements with bounding boxes
img_info = page.get_image_info(xrefs=True)
target = img_info[0]  # first image
xref = target["xref"]
orig_rect = fitz.Rect(target["bbox"])

# Extract original image bytes
pix = fitz.Pixmap(doc, xref)
img_bytes = pix.tobytes("png")

# Delete the original (replaces with 1x1 transparent pixel)
page.delete_image(xref)

# Re-insert at new position/size (e.g. 70% size, centered)
pw = page.rect.width
new_w = orig_rect.width * 0.7
new_h = orig_rect.height * 0.7
new_x0 = (pw - new_w) / 2
new_rect = fitz.Rect(new_x0, orig_rect.y0, new_x0 + new_w, orig_rect.y0 + new_h)
page.insert_image(new_rect, stream=img_bytes)

doc.save("output.pdf", garbage=4, deflate=True)
```

**Notes on image operations:**
- `delete_image` replaces the image data with a 1×1 transparent pixel (the page reference remains but is invisible). This is normal PyMuPDF behavior.
- For layout changes: extract → delete → re-insert at new rect. Do NOT try `doc.update_image()` (does not exist).
- `insert_image` accepts `filename=` (file path) or `stream=` (bytes). Use `stream=` when re-inserting extracted images.
- Use `garbage=4, deflate=True` in `doc.save()` to clean up unused objects and compress.

### pypdf - Basic Operations

#### Merge PDFs

```python
from pypdf import PdfWriter, PdfReader

writer = PdfWriter()
for pdf_file in ["doc1.pdf", "doc2.pdf", "doc3.pdf"]:
    reader = PdfReader(pdf_file)
    for page in reader.pages:
        writer.add_page(page)

with open("merged.pdf", "wb") as output:
    writer.write(output)
```

#### Split PDF

```python
reader = PdfReader("input.pdf")
for i, page in enumerate(reader.pages):
    writer = PdfWriter()
    writer.add_page(page)
    with open(f"page_{i+1}.pdf", "wb") as output:
        writer.write(output)
```

#### Extract Metadata

```python
reader = PdfReader("document.pdf")
meta = reader.metadata
print(f"Title: {meta.title}")
print(f"Author: {meta.author}")
print(f"Subject: {meta.subject}")
print(f"Creator: {meta.creator}")
```

#### Rotate Pages

```python
reader = PdfReader("input.pdf")
writer = PdfWriter()

page = reader.pages[0]
page.rotate(90)  # Rotate 90 degrees clockwise
writer.add_page(page)

with open("rotated.pdf", "wb") as output:
    writer.write(output)
```

### pdfplumber - Text and Table Extraction

#### Extract Text with Layout

```python
import pdfplumber

with pdfplumber.open("document.pdf") as pdf:
    for page in pdf.pages:
        text = page.extract_text()
        print(text)
```

#### Extract Tables

```python
with pdfplumber.open("document.pdf") as pdf:
    for i, page in enumerate(pdf.pages):
        tables = page.extract_tables()
        for j, table in enumerate(tables):
            print(f"Table {j+1} on page {i+1}:")
            for row in table:
                print(row)
```

#### Advanced Table Extraction

```python
import pandas as pd

with pdfplumber.open("document.pdf") as pdf:
    all_tables = []
    for page in pdf.pages:
        tables = page.extract_tables()
        for table in tables:
            if table:  # Check if table is not empty
                df = pd.DataFrame(table[1:], columns=table[0])
                all_tables.append(df)

# Combine all tables
if all_tables:
    combined_df = pd.concat(all_tables, ignore_index=True)
    combined_df.to_excel("extracted_tables.xlsx", index=False)
```

### reportlab - Create PDFs

> ⚠️ **If the PDF contains ANY Chinese/Japanese/Korean text, you MUST follow the CJK rules below. Skipping them causes black boxes (█) or crash errors.**

**FORBIDDEN patterns (these ALL break CJK rendering):**
- ❌ Using `Helvetica`, `Times-Roman`, or `Courier` fonts for CJK text → renders as **black boxes (█)**
- ❌ Registering `STSong-Light` without calling `addMapping()` → `Paragraph()` crashes with `ValueError: Can't map determine family/bold/italic for stsong-light`
- ❌ Using `<font face="STSong-Light">` in Paragraph XML without `addMapping()` → same crash
- ❌ Setting `style.fontName = "STSong-Light"` without `addMapping()` → same crash

**CJK font setup (MUST be at the top of EVERY script with CJK text):**

```python
from reportlab.pdfbase import pdfmetrics
from reportlab.pdfbase.cidfonts import UnicodeCIDFont
from reportlab.lib.fonts import addMapping

# Step 1: Register the CJK font
pdfmetrics.registerFont(UnicodeCIDFont("STSong-Light"))
# Step 2: MANDATORY — register font family mapping (without this Paragraph() crashes)
addMapping("STSong-Light", 0, 0, "STSong-Light")  # normal
addMapping("STSong-Light", 1, 0, "STSong-Light")  # bold
addMapping("STSong-Light", 0, 1, "STSong-Light")  # italic
addMapping("STSong-Light", 1, 1, "STSong-Light")  # bold+italic
```

**After setup, use `"STSong-L
