# TP Automation API — 3-Step Workflow for Dev/QA

Minimum HTTP contract for driving a `build → upgrade → run tests →
collect results` cycle against an automation testbed on TP. All
endpoints are stateless except for the durable `PytestExecution` /
`SingleDevice` rows the celery workers update — callers just POST then
poll, no websockets/sse/callbacks.

```
<TP-base-url>/<endpoint>
```

`<TP-base-url>`:
- Production: `https://nac-tp.fortinet-us.com`
- Test: `http://10.15.33.25:8000`
- Dev (.11): `http://10.15.33.11:8000`

## Authentication

Every endpoint requires a JWT in the `Authorization` header:

```
Authorization: JWT <token>
```

Mint a token:

```bash
T=$(curl -s -X POST <TP-base-url>/token-auth/ \
       -H 'Content-Type: application/json' \
       --data-binary @- <<JSON | jq -r .token
{"username":"<user>","password":"<pw>"}
JSON
)
```

In examples below `$T` stands for the JWT.

> **Note on prod:** `/token-auth/` is **not** exposed at the public
> reverse proxy on prod (SSO-only). Automation scripts on prod-facing
> endpoints need to be run from a host that can reach the internal
> port, or they need an SSO-issued token. On the dev `.11` server,
> `/token-auth/` works directly with username + password.

---

## Step 1 — Upgrade the testbed

```
POST /adc/automation/upgrade/                — kick off (HTTP 202)
GET  /adc/automation/upgrade/<testbed>/      — poll status
```

The dev team has three modes available; pick whichever matches your
build pipeline:

| `mode`     | When to use it | Required extra fields |
|---|---|---|
| `command`  | You have a Jenkins-built image and Jenkins already produces the full `execute restore image scp ...` command string. | `command` |
| `build`    | You have a build number from the official build server. | `build_number` |
| `ga`       | You want the latest GA build for a specific FortiNAC version. The system resolves the build number via `BuildHistory.GA=True`. | `version` |

### Request

Content-Type: `application/json`.

```json
{
  "testbed": "AT16_Combined_FSW",
  "mode":    "command",
  "command": "execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet"
}
```

```json
{
  "testbed": "AT16_Combined_FSW",
  "mode":    "build",
  "build_number": "0815"
}
```

```json
{
  "testbed": "AT16_Combined_FSW",
  "mode":    "ga",
  "version": "7.6.5"
}
```

Internal behavior:

1. Looks up the testbed's `deployinfo`, extracts every device whose
   name contains `nac` or `ncm`. A multi-NAC testbed gets one celery
   task per device, running in parallel.
2. `mode=ga` resolves `version` → `build_number` via `BuildHistory`,
   then dispatches the same path as `mode=build`.
3. Concurrency guard: HTTP **409** if any target IP is already
   `PROGRESS`.
4. `mode=command` validates the command starts with
   `execute restore image` (security guard — keeps arbitrary CLI from
   riding through this endpoint).

### Response (HTTP 202)

```json
{
  "testbed": "AT16_Combined_FSW",
  "mode": "command",
  "build_number": "",
  "target_ips": ["10.15.52.152"],
  "tasks": [
    {"ip": "10.15.52.152", "celery_id": "f62a4cb7-d435-41de-adb6-12cca405d507"}
  ]
}
```

Returns in <500 ms. The actual upgrade work runs asynchronously in
celery workers — the HTTP call returns the moment the task is queued.

### Poll for status

```
GET /adc/automation/upgrade/<testbed>/
```

```json
{
  "testbed": "AT16_Combined_FSW",
  "status":  "PROGRESS",
  "target_ips": ["10.15.52.152"],
  "per_device": [
    {
      "ip": "10.15.52.152",
      "status": "PROGRESS",
      "last_task_type": "command",
      "last_build_number": null,
      "updated_at": "2026-05-28T18:14:19+00:00",
      "log_tail": "...last 2 KB of the NAC's SSH output..."
    }
  ]
}
```

Aggregate `status` values:

| Value | Meaning |
|---|---|
| `UNKNOWN`  | No upgrade has been dispatched against this testbed (yet). |
| `PROGRESS` | At least one device is still mid-upgrade. |
| `SUCCESS`  | Every device finished with `SUCCESS`. |
| `FAILURE`  | At least one device finished with `FAILURE` / `TIMEOUT` (and no device is still in flight). |

`per_device[].log_tail` is the last ~2 KB of the device's SSH session
output, updated every ~3 s by the worker. Useful for live debugging
without flooding the response.

### Concrete polling loop

```bash
while :; do
  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/AT16_Combined_FSW/)
  STATUS=$(echo "$R" | jq -r .status)
  echo "$(date +%H:%M:%S) $STATUS"
  case "$STATUS" in
    PROGRESS) sleep 15 ;;
    SUCCESS)  echo "upgrade complete"; break ;;
    FAILURE)  echo "upgrade FAILED"; echo "$R" | jq .per_device; exit 1 ;;
    *)        echo "$R"; exit 1 ;;
  esac
done
```

A real upgrade takes 2-15 minutes depending on the image and how many
devices the testbed has.

---

## Step 2 — Run pytest cases on that testbed

```
POST /adc/automation/pytest/             — kick off (HTTP 202)
GET  /adc/automation/pytest/<exec_id>/   — poll status + results
```

### Request

Content-Type: `application/json`.

```json
{
  "user":     "<TP username>",
  "lab":      "AT16_Combined_FSW",
  "testcase": [
    "/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"
  ],
  "argument": "-vv --tb=short",
  "extra_pytest_options": ""
}
```

| Field | Type | Notes |
|---|---|---|
| `user`     | string | TP username of the caller. Must own the lab (be in `users` or `usedby` on the `AutomationTBUser` row). |
| `lab`      | string | AT lab name. Same `<testbed>` value as Step 1. |
| `testcase` | **list of strings** | Each entry is a pytest test-id path **relative to the tests repo root** (`/root/fnac_auto/tests` on the controller). Leading slash is OK; both `/Tests_CLI/...` and `Tests_CLI/...` work. The handler iterates the list, joins with spaces. |
| `argument` | string | Raw pytest CLI args injected between the testcase paths and the framework's `--html`/`--rack-file` flags. `-k`, `-m`, `-vv`, `--tb=short`, `--maxfail=N`, etc. |
| `extra_pytest_options` | string | (optional) Extra options appended *after* `--html`/`--rack-file` on the real run. Use for env-specific flags that shouldn't apply to the `--collect-only` dry run. |

### Response (HTTP 202)

```json
{
  "exec_id": 446,
  "status":  "Initiating",
  "lab":     "AT16_Combined_FSW",
  "testcase_count": 1
}
```

Returns in ~300 ms. Save `exec_id` — it's the handle for Step 3.

### What runs on the controller

The celery worker SFTP-uploads this script to the controller VM and
launches it under `nohup setsid`:

```bash
#!/bin/bash
set +e
cd /root/fnac_auto/tests          && git pull origin main || true
cd /root/fnac_auto/test-framework && git pull origin main || true
cd /root/fnac_auto
source venv/bin/activate
export PYTHONPATH=/root/fnac_auto/test-framework:/root/fnac_auto/tests
export DISPLAY=:99
pytest <expanded testcase paths> <argument> --collect-only
pytest <expanded testcase paths> <argument> --html=<report> --rack-file <rack> <extra_pytest_options>
echo $? > /tmp/pytest_<exec_id>.exit
```

Key properties:

- **Pytest is fully detached** from the launching ssh session via
  `nohup setsid`. The ssh session can drop, the network can blip, the
  TP backend can restart — pytest keeps running on the controller.
- **TP polls via short, separate ssh round-trips** to read the log
  file and check for the terminal `/tmp/pytest_<exec_id>.exit` marker.
- **Surviving a TP backend restart** is automatic — the celery worker
  is a separate process and just keeps polling. Verified end-to-end on
  `.11` (exec_id 447: killed runserver mid-run, exec completed
  normally and report was fetched back).

### Concurrency guard

If another `PytestExecution` on the same lab has `status` =
`Running` or `Initiating`, the POST returns HTTP **409**:

```json
{"error": "another execution is already in flight on 'AT16_Combined_FSW'"}
```

---

## Step 3 — Read per-test results

```
GET /adc/automation/pytest/<exec_id>/
```

Same endpoint as the Step-2 poll — there's nothing distinct to call.

### Response

```json
{
  "exec_id":          446,
  "status":           "Done",
  "lab":              "AT16_Combined_FSW",
  "controller":       "10.15.52.159",
  "remote_pid":       "177854",
  "pass_count":       1,
  "fail_count":       0,
  "skip_count":       0,
  "error_count":      0,
  "total_count":      1,
  "start_timestamp":  "1780090011",
  "end_timestamp":    "1780090041",
  "report_file_path": "data/AutomationTest/test_446_report_2026-05-29_21:26:50.html",
  "log_file_path":    "/tmp/pytest_446.log",
  "log_size":         25338,
  "log_tail":         "...last 4 KB of pytest stdout..."
}
```

| Field | Meaning |
|---|---|
| `status` | One of `Initiating`, `Running`, `Done`, `Failed`, `Cancelled`. **Treat `Done` as terminal — success/failure is inferred from counts.** |
| `pass_count`, `fail_count`, `skip_count`, `error_count`, `total_count` | Parsed from pytest's own summary line. Populated **mid-run** (updated every 15 s), so you can see partial progress without waiting for the run to finish. |
| `start_timestamp` / `end_timestamp` | Unix epoch seconds, written by the worker. |
| `report_file_path` | Server-relative path to the pytest-html report. Fetch via `<TP>/<report_file_path>` with the JWT for per-test rows + tracebacks. |
| `log_file_path` | On-controller path to the live log. Mostly informational — use `log_tail` to read the last 4 KB without an extra round-trip. |
| `log_tail` | Last 4 KB of pytest stdout. |
| `remote_pid` | The detached pytest tree's pid on the controller. Useful if you ever need to ssh to the controller and kill it manually. |

### Concrete polling loop

```bash
EXEC=$(curl -sX POST -H "Authorization: JWT $T" \
     -H 'Content-Type: application/json' \
     -d '{"user":"alice","lab":"AT16_Combined_FSW","testcase":["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],"argument":"-vv"}' \
     $TP/adc/automation/pytest/ | jq -r .exec_id)
echo "started exec $EXEC"

while :; do
  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
  STATUS=$(echo "$R" | jq -r .status)
  COUNTS=$(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count)/S\(.skip_count)/E\(.error_count) of T\(.total_count)"')
  echo "$(date +%H:%M:%S) $STATUS $COUNTS"
  case "$STATUS" in
    Initiating|Running) sleep 15 ;;
    Done)
      FAIL=$(echo "$R" | jq -r .fail_count)
      ERR=$(echo "$R"  | jq -r .error_count)
      if [ "$FAIL" -gt 0 ] || [ "$ERR" -gt 0 ]; then
        echo "tests had failures — fetch report at $TP/$(echo "$R" | jq -r .report_file_path)"
        exit 1
      fi
      echo "all pass"; break ;;
    Failed|Cancelled)
      echo "execution terminated: $STATUS"
      echo "$R" | jq -r .log_tail; exit 1 ;;
  esac
done
```

---

## End-to-end script

```bash
#!/usr/bin/env bash
set -euo pipefail
TP=http://10.15.33.11:8000
USER=admin
TESTBED=AT16_Combined_FSW

T=$(curl -s -X POST $TP/token-auth/ -H 'Content-Type: application/json' \
       --data-binary @- <<JSON | jq -r .token
{"username":"$USER","password":"<your-pw>"}
JSON
)

# ---- Step 1: upgrade testbed ----
COMMAND='execute restore image scp /var/lib/jenkins/jobs/fortinac-build-7.6-1/builds/3983/archive/nacos/FNAC_ESX-v7-build6956-FORTINET_job1_3983.out 10.15.33.5 jenkins fortinet'

curl -s -X POST -H "Authorization: JWT $T" \
     -H 'Content-Type: application/json' \
     -d "$(jq -n --arg cmd "$COMMAND" --arg tb "$TESTBED" \
            '{testbed:$tb, mode:"command", command:$cmd}')" \
     $TP/adc/automation/upgrade/ > /dev/null

while :; do
  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/upgrade/$TESTBED/)
  S=$(echo "$R" | jq -r .status)
  echo "upgrade: $S"
  [ "$S" = "SUCCESS" ] && break
  [ "$S" = "FAILURE" ] && { echo "$R" | jq .per_device; exit 1; }
  sleep 30
done

# ---- Step 2 + 3: run tests, collect results ----
EXEC=$(curl -s -X POST -H "Authorization: JWT $T" \
     -H 'Content-Type: application/json' \
     -d "$(jq -n --arg tb "$TESTBED" --arg user "$USER" \
            '{user:$user, lab:$tb,
              testcase:["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"],
              argument:"-vv --tb=short"}')" \
     $TP/adc/automation/pytest/ | jq -r .exec_id)
echo "started exec $EXEC"

while :; do
  R=$(curl -sH "Authorization: JWT $T" $TP/adc/automation/pytest/$EXEC/)
  S=$(echo "$R" | jq -r .status)
  echo "pytest: $S $(echo "$R" | jq -r '"P\(.pass_count)/F\(.fail_count) of T\(.total_count)"')"
  case "$S" in
    Done)
      F=$(echo "$R" | jq -r .fail_count)
      E=$(echo "$R" | jq -r .error_count)
      [ "$F" -gt 0 ] || [ "$E" -gt 0 ] && {
        echo "report: $TP/$(echo "$R" | jq -r .report_file_path)"
        exit 1
      }
      echo "PASS"; break ;;
    Failed|Cancelled)
      echo "exec terminated: $S"; echo "$R" | jq -r .log_tail; exit 1 ;;
    *) sleep 15 ;;
  esac
done
```

---

## Behaviors worth knowing

### Concurrency

For each `<testbed>`:
- **Upgrade**: HTTP 409 if any target IP currently has
  `SingleDevice.last_task_status='PROGRESS'`.
- **Pytest**: HTTP 409 if any `PytestExecution` with this `lab` has
  `status` in `{Running, Initiating}`.

You can have an upgrade AND a pytest running simultaneously on
*different* testbeds. The guards are per-testbed.

### Survival semantics

| Failure | Upgrade | Pytest v2 |
|---|---|---|
| ssh session drops mid-run | N/A — TP only ssh's briefly | Pytest keeps running (`nohup setsid` on controller) |
| TP backend restart mid-run | Celery worker keeps going, completes the run, writes terminal state | Same — celery `pytest_poll` keeps short-polling the controller |
| Controller VM reboot mid-run | Upgrade reboots the NAC; this is expected/desired. | Pytest is lost (controller died); polling will eventually fail with the max-runtime safety net (6 h cap). |
| Celery worker restart | In-flight task may be lost (celery uses ack_late=false by default). Worth a follow-up. | Same. |

### Per-test pass/fail visibility

The `passed_nodes` / `failed_nodes` fields on the legacy `pytest_run`
response shape don't apply to v2. v2 surfaces:

- **aggregate counts** (`pass_count`, `fail_count`, etc.) in the GET
  response — populated mid-run as pytest emits them
- **the HTML report** at `report_file_path` — the standard
  pytest-html document with per-test rows, durations, tracebacks, and
  captured logs. Fetch via `<TP>/<report_file_path>` with the JWT.

If you need per-test JSON instead of the HTML report, ask — it's a
trivial extra endpoint that parses the HTML and returns
`{passed: [...], failed: [...]}`.

### Test-case path format

Whatever you'd type after `pytest` on the command line, prefixed (or
not) with `/`. Relative to `/root/fnac_auto/tests/` on the controller.

| Want to | Use |
|---|---|
| Run one test | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status"]` |
| Run several | `["/Tests_CLI/test_cli_sanity.py::TestGetBasic::test_get_system_status", "/Tests_CLI/test_cli_sanity.py::TestShowFullConfig::test_nacos_show_config"]` |
| Run a whole file | `["/Tests_CLI/test_cli_sanity.py"]` |
| Run by marker | `["/Tests_CLI"]` + `argument: "-m smoke"` |
| Dry-run / list | `["/Tests_CLI"]` + `argument: "--collect-only"` |

Tip: discover paths via `GET /adc/get_testcases` (legacy endpoint;
returns the full test tree). Note: this endpoint does a `git pull` on
TP's local mirror of the tests repo as a side effect — fine in normal
use, just be aware.

---

## Source files

| Concern | File |
|---|---|
| URL routes | `backend/adc/urls.py` |
| Upgrade API | `backend/adc/views/automation/upgrade_api.py` |
| Upgrade tasks | `backend/adc/tasks.py` (`upgrade_nac_command`, `upgrade_nac_build`) |
| Pytest v2 API | `backend/adc/views/automation/pytest_api.py` |
| Pytest v2 tasks | `backend/adc/tasks.py` (`pytest_launch`, `pytest_poll`) |
| Per-device state | `backend/adc/models.py` (`SingleDevice`, `PytestExecution`) |
| GA-build resolution | `backend/adc/models.py` (`BuildHistory.GA=True`) |

Legacy endpoints (still live, used by the `/automation` UI page, kept
for backward compatibility):

- `POST /adc/nac-upgrade-testbed/`
- `POST /adc/pytest_run`
- `POST /adc/get_test_execution_by_id`

Avoid these for new automation — they're synchronous-in-the-request,
don't survive TP restarts, and the upgrade variant blocks the HTTP
request for the full restore duration.
