# BigQuery Schema & Views

SQL definitions for the BigQuery analytics layer. These create the flattened
tables and views that power SQL analytics, BI dashboards (Looker, Sheets), and
ad-hoc data exploration.

## Architecture

```
Sanity Content Lake
       │
       ▼
Airbyte (scheduled sync)
  ├─ Stream: reports    → ailf_raw.reports     (flat, GROQ-projected)
  └─ Stream: area_scores → ailf_raw.area_scores (nested model→area arrays)
       │
       ▼
BigQuery views (this directory)
  ├─ ailf.reports       → direct passthrough (already flat from GROQ projection)
  └─ ailf.area_scores   → UNNEST flattening (one row per area per model per report)
```

## Files

| File                             | Purpose                                                                                                                                                      |
| -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `views/area_scores.sql`          | Flattens nested `model_scores` array into one row per area per model per report                                                                              |
| `views/reports.sql`              | Clean passthrough view with correct types and column ordering; also materializes `validity_*` columns and the `include_in_default_trends` trust gate (D0059) |
| `views/official_runs.sql`        | Canonical trend series (D0037): `classification='official' AND trigger_type='scheduled' AND owner_team='core-docs'`                                          |
| `views/official_area_scores.sql` | `area_scores` joined to `official_runs` — inherits the official-run predicate for area-level dashboards                                                      |
| `views/team_runs_template.sql`   | Recipe/template for instantiating per-team filtered views                                                                                                    |

## Setup

The Airbyte connection loads data into a raw dataset (e.g., `ailf_raw`). The
views defined here reference that raw dataset and present the designed schema
from `docs/design-docs/report-store/bigquery.md`.

### 1. Create the raw dataset (Airbyte writes here)

```bash
bq --project_id=data-platform-302218 --location=EU mk --dataset ailf_raw
```

### 2. Create the analytics dataset (views live here)

```bash
bq --project_id=data-platform-302218 --location=EU mk --dataset ailf
```

### 3. Create the views

**Important ordering (learned 2026-04-23):** Airbyte must be redeployed with the
current manifest **before** you run these view SQLs. Each view binds to specific
columns on `ailf_raw.reports`; if the raw table is missing columns the Airbyte
projection expects, the `CREATE VIEW` statement fails with
`Unrecognized name: <column>`.

If your Airbyte destination has **schema evolution enabled** ("Propagate column
changes" in the UI), new columns appear automatically on the next incremental
sync. If not, flip it on, trigger a resync, and confirm the expected columns
exist before creating views:

```bash
bq --project_id=data-platform-302218 --location=EU query --use_legacy_sql=false \
  "SELECT column_name FROM ailf_raw.INFORMATION_SCHEMA.COLUMNS WHERE table_name = 'reports' ORDER BY column_name"
```

If propagation is disabled and you can't flip it quickly, manually
`ALTER TABLE ailf_raw.reports ADD COLUMN IF NOT EXISTS …` for each missing
column as a stop-gap. Values will be `NULL` until Airbyte writes to them on the
next sync.

Once the raw table has the expected columns:

```bash
cd packages/eval/config/bigquery
bq --project_id=data-platform-302218 --location=EU query --use_legacy_sql=false < views/reports.sql
bq --project_id=data-platform-302218 --location=EU query --use_legacy_sql=false < views/area_scores.sql
bq --project_id=data-platform-302218 --location=EU query --use_legacy_sql=false < views/official_runs.sql
bq --project_id=data-platform-302218 --location=EU query --use_legacy_sql=false < views/official_area_scores.sql
# per-team views are optional — copy views/team_runs_template.sql,
# fill in the slug, and run.
```

> `--project_id` / `--location=EU` are required because `bq` needs an explicit
> billing project and the `ailf*` datasets live in the EU multi-region. If you
> run `bq query` from this repo regularly, consider setting the default with
> `gcloud config set project data-platform-302218`.

## Querying default (trustworthy) reports

`ailf.reports` exposes every synced report, including development runs and
structurally incomplete ones. BI dashboards and ad-hoc analysis should gate on
the **`include_in_default_trends`** boolean, which materializes the single
trustworthiness predicate (D0059) shared with the dashboard and Studio — so all
three surfaces show the same default population:

```sql
SELECT *
FROM   `data-platform-302218.ailf.reports`
WHERE  include_in_default_trends = TRUE;
```

`include_in_default_trends` is the SQL form of the `includeInDefaultTrends`
predicate (`INCLUDE_IN_DEFAULT_TRENDS_SQL` in `@sanity/ailf-shared`):

```text
(validity_status IS NULL OR validity_status = 'ok')
  AND (classification IS NULL OR classification NOT IN ('test', 'experimental'))
```

To audit the _excluded_ (untrusted) reports — the inverse population, useful for
the Tier-2 review pass — flip the predicate to `= FALSE`. The `validity_status`
/ `validity_reasons` columns explain why each was excluded.

> The `official_runs` view (above) is a **stricter** canonical-series gate
> (`classification='official' AND trigger_type='scheduled' AND owner_team='core-docs'`);
> `include_in_default_trends` is the **broader** "trustworthy enough to show by
> default" gate. Use `official_runs` for the tracked trend line,
> `include_in_default_trends` for general default views.

## Naming conventions

- **`ailf_raw.*`** — raw Airbyte-loaded tables (nested JSON, Airbyte metadata
  columns)
- **`ailf.*`** — analytics views (flat, typed, designed schema from bigquery.md)

Airbyte adds metadata columns (`_airbyte_raw_id`, `_airbyte_extracted_at`,
`_airbyte_meta`) to its output tables. The views strip these so downstream
consumers see only the designed schema.

## Schema evolution

Views are the transformation layer. When the report format evolves:

1. Update the GROQ projections in the Airbyte connector YAML
2. Update the view SQL to map new fields
3. Backfill is automatic — views always reflect current data

@see docs/design-docs/report-store/bigquery.md — canonical schema definition
@see packages/eval/config/airbyte/ — Airbyte connector configuration
