---
name: lakehouse-llms-index
description: |
  Generate or update llms.txt and llms-full.txt for the Lakehouse documentation repository (based on SUMMARY.md),
  or verify that URLs in these two files are consistent with SUMMARY.md and up to date.
  Applies to both the Singdata English documentation (singdata.com) and the ClickZetta Chinese documentation (www.yunqi.tech).
  Use when the user says "generate llms index", "update llms.txt/llms-full.txt", "verify doc URLs", or "sync SUMMARY to llms".
---

# Lakehouse LLMs Index Generator

## Overview

`llms.txt` and `llms-full.txt` are LLM navigation index files for the Lakehouse documentation, generated from `SUMMARY.md`. The files are hosted at `git@k.topthink.com:4v2dmg3x2e/k7pl9zonpy.git`, local path: `/Users/guanyangw/工作/AI_CODE_COLLECTIONS/lakehouse/clickzetta-documents/`.

## Applicable Documentation Sites (Dual Brand / Dual Repository)

The documentation has two independent repositories and sites:

| Brand | Language | Local Repository | BASE_URL |
|------|------|----------|----------|
| ClickZetta | Chinese | `clickzetta-documents/` | `https://www.yunqi.tech/documents/` |
| Singdata | English | `singdata-documents-en/` | `https://singdata.com/documents/` |

When generating or validating, first confirm the `BASE_URL` for the target repository. **Note: the Singdata domain is `singdata.com` (no leading www, and not clickzetta.com).** If the user does not specify, default to the ClickZetta Chinese site `https://www.yunqi.tech/documents/`.

**Both sites use the same URL slug rules**: slug = filename from SUMMARY.md with `.md` removed, **preserving original case and separators** (verified via singdata.com's sitemap.xml — e.g., `overview.md`→`overview`, `Ingestion.md`→`Ingestion`, `RemoteFunction-as-udf.md`→`RemoteFunction-as-udf`). The "URL Generation Rules" below apply to both sites; the only difference is the domain.

> ⚠️ Validation note: When determining "actual live URLs", **do not treat the existing `llms-full.txt` in the repository as the live truth** — it may itself be stale or incorrect. To verify real slugs, fetch the site's `sitemap.xml` (e.g., `https://singdata.com/sitemap.xml`), which is the authoritative source.

## Difference Between the Two Files

| File | Content Scope | Structure |
|------|---------|------|
| `llms.txt` | Only top-level sections (`##`) and first-level sub-pages (`*` direct children, depth=0) | Flat, no indentation |
| `llms-full.txt` | All pages at all levels in SUMMARY.md | 2-space indentation to represent hierarchy |

Both files use the same format:
```
- [Title](URL): one-sentence summary (30-50 words)
```

## URL Generation Rules

**Strictly concatenate based on filename (including directory path), preserving original case:**

```
Remove .md suffix from filename → append to BASE_URL
```

Examples (using ClickZetta as example; Singdata only replaces domain with singdata.com):
- `overview.md` → `https://www.yunqi.tech/documents/overview`
- `LakehouseStudio-tour.md` → `https://www.yunqi.tech/documents/LakehouseStudio-tour`
- `eco_integration/sqlworkbench-j-lakehouse.md` → `https://www.yunqi.tech/documents/eco_integration/sqlworkbench-j-lakehouse`
- `sql_functions/aggregate_functions/avg.md` → `https://www.yunqi.tech/documents/sql_functions/aggregate_functions/avg`

**Note:** Do not perform any case conversion, camelCasing, or path simplification — preserve the original file path exactly.

## Summary Source

Summaries are generated by reading the content of the corresponding `.md` file, 30-50 words, describing the core content of the document. Existing summaries are reused directly; new summaries are only generated for newly added entries.

## Generation Steps

### Step 1: Parse SUMMARY.md

```python
import re

# Select domain based on target repository
BASE_URL = "https://www.yunqi.tech/documents/"   # ClickZetta Chinese site
# BASE_URL = "https://singdata.com/documents/"   # Singdata English site (note: no www)

with open('SUMMARY.md', 'r') as f:
    lines = f.readlines()

entries = []  # (type, title, url, filepath, depth)
for line in lines:
    # Section headings
    section = re.match(r'^## (.+)', line)
    if section:
        entries.append(('section', section.group(1).strip(), None, None, -1))
        continue
    # List items: number of indent spaces / 4 = depth
    item = re.match(r'^(\s*)\* \[(.+?)\]\((.+?)\)', line)
    if item:
        depth = len(item.group(1)) // 4
        title = item.group(2).strip()
        filepath = item.group(3).strip()
        url = BASE_URL + filepath[:-3]  # remove .md, preserve original path and case
        entries.append(('item', title, url, filepath, depth))
```

### Step 2: Reuse Existing Summaries

```python
with open('llms-full.txt', 'r') as f:
    existing_txt = f.read()

existing = {}
for line in existing_txt.split('\n'):
    # Compatible with both domains
    m = re.match(r'\s*- \[.+?\]\((https://[^)]+)\): (.+)', line)
    if m:
        # Use .md path as key to reuse summaries, avoiding key mismatch when switching domains
        path = re.sub(r'https://[^/]+/documents/', '', m.group(1))
        existing[path] = m.group(2)
```

### Step 3: Generate Summaries for New Entries

For entries where `path not in existing`, read the first 100 lines of the corresponding `.md` file and generate a 30-50 word summary.

### Step 4: Generate llms-full.txt

All entries, following SUMMARY.md hierarchy, with sub-items indented by 2 spaces × depth:

```python
indent = "  " * depth
line = f"{indent}- [{title}]({url}): {summary}"
```

### Step 5: Generate llms.txt

Keep only:
- `##` section headings (formatted as `## Section Name`)
- depth=0 top-level pages (`*` direct children, no indentation)

## URL Validation

Check whether URLs in `llms-full.txt` are consistent with the current SUMMARY.md:

```python
# Extract all valid URLs from SUMMARY.md
valid_urls = {BASE_URL + f[:-3] for _, _, _, f, _ in entries if f}

# Extract all URLs from llms-full.txt (by current brand domain)
domain = BASE_URL.split('/documents/')[0]  # https://www.yunqi.tech or https://singdata.com
used_urls = set(re.findall(re.escape(domain) + r'/documents/[^\)]+', full_txt))

# Find issues
stale = used_urls - valid_urls      # in llms-full.txt but not in SUMMARY.md (old paths/deleted)
missing = valid_urls - used_urls    # in SUMMARY.md but not in llms-full.txt (new pages)
```

Common issues:
- Old format for `eco_integration/` subdirectory entries (missing path prefix) → update to include full path
- New pages added to SUMMARY.md not yet synced to llms-full.txt
- Domain inconsistency: ClickZetta files mixing in singdata.com links or vice versa → unify to the target site domain (Chinese: www.yunqi.tech, English: singdata.com)

> ⚠️ **NEVER treat the existing `llms-full.txt` (or `llms.txt`) as the source of truth for "actual live URLs".**
> These files may themselves be stale or incorrect (old slugs, wrong domain) — they are the thing being validated, not the basis for validation.
> The consistency check above only guarantees "llms files agree with SUMMARY.md" — it does NOT guarantee the URLs actually resolve online.
> If SUMMARY.md filenames differ from the live published slugs, the consistency check cannot catch it. To confirm URLs are genuinely valid, use the sitemap cross-check below.

### Sitemap Cross-Check (live truth, authoritative source)

The site's `sitemap.xml` is the **only authoritative source** for live URLs. Fetch it and three-way cross-check the URLs generated from SUMMARY.md:

```python
import re, urllib.request

# Chinese site: https://www.yunqi.tech/sitemap.xml ; English site: https://singdata.com/sitemap.xml
SITEMAP_URL = "https://singdata.com/sitemap.xml"
sitemap_xml = urllib.request.urlopen(SITEMAP_URL).read().decode("utf-8")

# Extract all /documents/ page URLs from the sitemap, normalizing domain (strip www. difference)
def norm(u):
    return re.sub(r'https://(www\.)?', 'https://', u).rstrip('/')

sitemap_urls = {norm(u) for u in re.findall(r'<loc>(https://[^<]+/documents/[^<]+)</loc>', sitemap_xml)}
valid_norm   = {norm(u) for u in valid_urls}   # valid_urls from SUMMARY.md (see above)

# Cross-compare
in_sitemap     = valid_norm & sitemap_urls   # published live → can verify against the real URL
not_in_sitemap = valid_norm - sitemap_urls   # not live → treat as a NEW file (not yet published)
orphan         = sitemap_urls - valid_norm   # live page not covered by SUMMARY.md → possibly missing entry

print("Published, verifiable via sitemap:", len(in_sitemap))
print("New files (not in sitemap, can only be judged by rule):", len(not_in_sitemap))
for u in sorted(not_in_sitemap)[:30]:
    print("   NEW_FILE:", u)
```

Interpretation (important):
- **Hit in sitemap** → the URL is published live, so its slug is confirmed correct against the real URL. The higher the hit ratio, the more trustworthy the slug rule.
- **Miss in sitemap** → **this is a new file** (already in SUMMARY.md / git, but not yet published to the live docs site).
  - A miss does NOT mean the URL is wrong: the page simply does not exist online yet, so there is no real URL to compare against and an HTTP liveness check would also necessarily fail (the page is unpublished). **Live-verification methods are therefore useless for new files.**
  - For new files you can **only fall back to the slug rule** (filename minus `.md`, preserving case and separators exactly); you cannot verify against a real live URL.
- Therefore: **never treat "missing from sitemap" as a signal that the slug rule is wrong.** To judge whether the slug rule itself is correct, look only at the **hits**. If many pages that should have long been published also fail to match the sitemap, only then is the rule wrong.
- The sitemap may be paginated (sitemap_index.xml) or trimmed on JS-heavy sites — make sure you fetch it all; normalize the `www.` prefix difference when comparing.

## File Header Format

**llms.txt:**
```markdown
# ClickZetta Lakehouse Documentation (LLM Navigation)

> ClickZetta Lakehouse is a fully managed lakehouse architecture platform... (product description)
```

**llms-full.txt:**
```markdown
# Lakehouse Documentation (LLM Full Index)

> ClickZetta Lakehouse is a fully managed lakehouse architecture platform... (product description)
```

> If the target is the Singdata English site, replace the brand name in the title and description with Singdata, use `singdata.com` (no www) as the domain in the body, and apply the same slug rule (filename minus .md, preserving original case).
