# Translation Alignment System

A sophisticated block-aware alignment system for maintaining translations of markdown documents. This system intelligently detects changes, handles reordering, and minimizes unnecessary AI translation calls.

## Overview

When you have an English document and its French translation, and you make changes to the English version, this system:

1. **Segments** both documents into semantic blocks (headings, paragraphs, lists, code blocks, etc.)
2. **Fingerprints** each block using structural anchors (special characters, numbers, punctuation)
3. **Aligns** blocks between English and French using the Needleman-Wunsch algorithm
4. **Detects** which blocks changed, were added, deleted, or reordered
5. **Optimizes** by only sending changed/new blocks to AI for translation
6. **Reconstructs** the French document with proper ordering and spacing

## Key Features

### Language-Agnostic Alignment

Uses **anchor text** (special characters like `[]`124567890-=!@#$%^&*()><`) rather than language-specific content. This makes alignment robust across different languages.

```typescript
// Example anchor text extraction:
"Hello [World](https://example.com) - 2024"
// Becomes: "[ ](://.)-2024"
```

### Automatic Reordering Detection

If you swap two paragraphs in the English document and both remain unchanged in content, the system automatically reorders them in the French translation without re-translating.

### Smart Change Detection

Uses Git changed lines (if available) to precisely identify which blocks need review. Falls back to content comparison if Git info is unavailable.

### Block Types Supported

- `heading` - Markdown headings (# ## ###)
- `paragraph` - Regular text paragraphs
- `list_item` - Bullet and numbered lists
- `code_block` - Fenced code blocks
- `blockquote` - Quote blocks
- `table` - Markdown tables
- `horizontal_rule` - Horizontal rules (---)
- `html` - Embedded HTML
- `unknown` - Fallback for blank lines

## Architecture

### Core Components

```
translation-alignment/
├── types.ts                    # Type definitions
├── segmentDocument.ts          # Document → Blocks
├── normalizeBlock.ts           # Block → Normalized (semantic + anchor)
├── fingerprintBlock.ts         # Normalized → Fingerprinted (with digests)
├── computeSimilarity.ts        # Jaccard similarity calculation
├── alignBlocks.ts              # Needleman-Wunsch alignment
├── mapChangedLinesToBlocks.ts  # Git lines → Block indexes
├── planActions.ts              # Alignment → Action plan
├── rebuildDocument.ts          # Action plan → Segments + Merge
└── pipeline.ts                 # High-level orchestration
```

### Data Flow

```
English Text ─────────┐
                      │
                      ├─→ Segment → Normalize → Fingerprint
                      │
French Text ──────────┘
                      │
                      ↓
              Align (Needleman-Wunsch)
                      │
                      ↓
              Map Changed Lines
                      │
                      ↓
              Plan Actions
                      │
                      ↓
    ┌─────────────────┴─────────────────┐
    │                                   │
    ↓                                   ↓
Segments to Review              Reusable Segments
    │                                   │
    ↓                                   │
AI Translation                          │
    │                                   │
    └─────────────────┬─────────────────┘
                      │
                      ↓
              Merge & Output
```

## How It Works

### 1. Segmentation

```typescript
const blocks = segmentDocument(text);
// Input: "# Title\n\nParagraph 1\n\nParagraph 2"
// Output: [
//   { type: "heading", content: "# Title\n", lineStart: 1, lineEnd: 1 },
//   { type: "paragraph", content: "Paragraph 1\n", lineStart: 3, lineEnd: 3 },
//   { type: "paragraph", content: "Paragraph 2\n", lineStart: 5, lineEnd: 5 }
// ]
```

### 2. Normalization

```typescript
const normalized = normalizeBlock(block);
// Creates two representations:
// - semanticText: lowercase, no markdown formatting
// - anchorText: only special chars, numbers, symbols
```

### 3. Fingerprinting

```typescript
const fingerprinted = fingerprintBlock(normalized, prev, next);
// Adds:
// - semanticDigest: hash of semantic content
// - anchorDigest: hash of structural anchors
// - compositeKey: combined unique identifier
// - contextKey: hash of surrounding blocks
```

### 4. Alignment

Uses Needleman-Wunsch algorithm with custom scoring:

- **Match score**: Based on type similarity, anchor similarity, and length ratio
- **Gap penalty**: Penalizes insertions/deletions
- **Type bonus**: +2 if block types match
- **Length bonus**: +1 if lengths within 75%
- **Anchor similarity**: 0-1 Jaccard score × 8

### 5. Action Planning

Produces one of four actions for each block:

```typescript
type PlannedAction =
  | { kind: "reuse" }        // Unchanged, use existing translation
  | { kind: "review" }       // Changed, needs AI review
  | { kind: "insert_new" }   // New block, needs translation
  | { kind: "delete" };      // Removed from source, skip
```

### 6. Reconstruction

```typescript
// Only segments needing review are sent to AI
const reviewedMap = new Map<actionIndex, translatedText>();

// Final document merges reused + reviewed segments in correct order
const output = mergeReviewedSegments(plan, frenchBlocks, reviewedMap);
```

## Integration

### Basic Usage

```typescript
import { reviewFileBlockAware } from "./reviewDocBlockAware";

await reviewFileBlockAware(
  "docs/en/guide.md",      // English source
  "docs/fr/guide.md",      // French translation
  Locales.FRENCH,          // Target locale
  Locales.ENGLISH,         // Base locale
  aiOptions,               // AI configuration
  configOptions,           // App configuration
  customInstructions,      // Optional AI instructions
  [10, 15, 20]            // Git changed lines (optional)
);
```

### Replace Existing Implementation

In your main review script, replace:

```typescript
// Old:
await reviewFile(
  baseFilePath,
  outputFilePath,
  locale,
  baseLocale,
  aiOptions,
  configOptions,
  customInstructions,
  changedLines
);

// New:
await reviewFileBlockAware(
  baseFilePath,
  outputFilePath,
  locale,
  baseLocale,
  aiOptions,
  configOptions,
  customInstructions,
  changedLines
);
```

### Configuration Options

```typescript
const { plan, segmentsToReview } = buildAlignmentPlan({
  englishText,
  frenchText,
  changedLines: [10, 15, 20],  // Optional Git changed lines
  similarityOptions: {
    minimumMatchForReuse: 0.9,        // Default: 0.9 (90% similar)
    minimumMatchForNearDuplicate: 0.8  // Default: 0.8 (80% similar)
  }
});
```

## Examples

### Example 1: Simple Change

**English v1:**
```markdown
# Introduction
Hello world
```

**French v1:**
```markdown
# Introduction
Bonjour le monde
```

**English v2 (changed second line):**
```markdown
# Introduction
Hello universe
```

**Result:**
- Block 1 (heading): **reuse** existing translation
- Block 2 (paragraph): **review** with AI
- Final output preserves structure, only paragraph updated

### Example 2: Reordering

**English v1:**
```markdown
## Section A
Content A

## Section B
Content B
```

**English v2 (swapped order):**
```markdown
## Section B
Content B

## Section A
Content A
```

**Result:**
- All blocks identified as unchanged
- Automatically reordered in French
- No AI calls needed

### Example 3: Insertion

**English v1:**
```markdown
Para 1

Para 3
```

**English v2:**
```markdown
Para 1

Para 2

Para 3
```

**Result:**
- Para 1: **reuse**
- Para 2: **insert_new** (translate with AI)
- Para 3: **reuse**

### Example 4: Deletion

**English v1:**
```markdown
Para 1

Para 2

Para 3
```

**English v2:**
```markdown
Para 1

Para 3
```

**Result:**
- Para 1: **reuse**
- Para 2: **delete** (skip in output)
- Para 3: **reuse**

## Performance Optimization

### With Git Integration

When Git changed lines are available, the system:
1. Maps changed lines to block indexes
2. Only marks those blocks as needing review
3. Dramatically reduces AI calls for large documents

```typescript
// With git info: only reviews blocks 5, 6, 7 (3 AI calls)
changedLines: [45, 46, 47, 48, 49, 50]

// Without git info: reviews all changed blocks by content comparison
changedLines: undefined
```

### Reuse Rate

Typical reuse rates:
- **Minor edits**: 90-95% blocks reused
- **Moderate changes**: 70-80% blocks reused
- **Major refactor**: 40-60% blocks reused
- **New document**: 0% blocks reused (but alignment helps with structure)

## Best Practices

### 1. Preserve Structure

Keep consistent markdown structure between versions. The alignment algorithm works best when block types remain stable.

✅ Good:
```markdown
# Heading
Paragraph

# Heading
Paragraph (updated content)
```

❌ Avoid:
```markdown
# Heading
Paragraph

## Different heading level
List instead of paragraph
- Item 1
```

### 2. Use Git Integration

Always pass `changedLines` when available for maximum efficiency:

```typescript
const changedLines = await listGitLines(filePath, gitOptions);
await reviewFileBlockAware(..., changedLines);
```

### 3. Adjust Similarity Thresholds

For different document types:

- **Technical docs** (lots of code/numbers): Lower threshold (0.85)
- **Marketing content** (more text): Higher threshold (0.95)
- **Mixed content**: Default (0.90)

### 4. Monitor Action Distribution

Log action distribution to understand system behavior:

```typescript
const stats = {
  reuse: plan.actions.filter(a => a.kind === "reuse").length,
  review: plan.actions.filter(a => a.kind === "review").length,
  insertNew: plan.actions.filter(a => a.kind === "insert_new").length,
  delete: plan.actions.filter(a => a.kind === "delete").length,
};
console.log(`Efficiency: ${(stats.reuse / plan.actions.length * 100).toFixed(1)}% reused`);
```

## Troubleshooting

### Issue: Too Many Blocks Marked for Review

**Cause**: Similarity threshold too high
**Solution**: Lower `minimumMatchForReuse` to 0.85 or 0.80

### Issue: Blocks Not Aligning Correctly

**Cause**: Document structure changed significantly
**Solution**: 
- Ensure block types are consistent
- Check that special characters/numbers are preserved
- Verify the anchor text is being extracted correctly

### Issue: Translations Not Merging

**Cause**: Action indexes not matching
**Solution**: Ensure `reviewedSegmentsMap` uses `segment.actionIndex` as key

### Issue: Extra Blank Lines in Output

**Cause**: Block content doesn't preserve trailing newlines consistently
**Solution**: Review `trimTrailingNewlines` in segmentation logic

## Testing

### Unit Tests

```typescript
import { segmentDocument } from "./segmentDocument";
import { alignEnglishAndFrenchBlocks } from "./alignBlocks";

describe("Block Alignment", () => {
  it("should detect reordered paragraphs", () => {
    const english = "A\n\nB\n\nC";
    const french = "B\n\nA\n\nC";
    // ... test alignment
  });
});
```

### Integration Tests

```typescript
it("should handle complete document translation workflow", async () => {
  const result = await reviewFileBlockAware(
    testEnglishPath,
    testFrenchPath,
    Locales.FRENCH,
    Locales.ENGLISH
  );
  // Verify output structure, reuse rate, etc.
});
```

## Future Enhancements

- [ ] Support for more block types (footnotes, definition lists)
- [ ] Fuzzy matching for minor structural variations
- [ ] Parallel AI translation for multiple segments
- [ ] Caching of fingerprints for repeated processing
- [ ] Visual diff tool for reviewing alignment decisions
- [ ] Machine learning for optimal threshold selection

## Contributing

When contributing to this system:

1. **Follow naming conventions**: No abbreviations, descriptive names
2. **Use arrow functions**: Consistent function syntax
3. **Add JSDoc comments**: Document complex logic
4. **Write tests**: Cover edge cases
5. **Maintain backwards compatibility**: Ensure existing integrations work

## License

See main project LICENSE file.

