{"_id":"@builder.io/kube-cron-jobs","_rev":"13-c46af53c295da1da2123419b9afb2eb2","name":"@builder.io/kube-cron-jobs","dist-tags":{"latest":"0.1.1"},"versions":{"0.1.0":{"name":"@builder.io/kube-cron-jobs","version":"0.1.0","_id":"@builder.io/kube-cron-jobs@0.1.0","maintainers":[{"name":"steve8708","email":"sewell.steve@gmail.com"},{"name":"samijaber","email":"jabersami@gmail.com"},{"name":"teleaziz123","email":"aziz@builder.io"},{"name":"mrkoreye","email":"korey@builder.io"},{"name":"strd6","email":"daniel@danielx.net"},{"name":"mhevery","email":"misko@hevery.com"},{"name":"gustavohgs","email":"gustavo@builder.io"},{"name":"manucorporat","email":"manu.mtza@gmail.com"},{"name":"sanyamkamat","email":"sanyamkamat@gmail.com"},{"name":"midhunadarvin","email":"midhunadarvin@gmail.com"},{"name":"kylefowler","email":"kyle.e.fowler@gmail.com"},{"name":"builderio-bot","email":"builder-bot@builder.io"},{"name":"armela","email":"armela@builder.io"},{"name":"anaghavarhade","email":"anagha@builder.io"},{"name":"sidmohanty11","email":"sidmohanty11@gmail.com"},{"name":"yash-wadhia-builder","email":"yash@builder.io"},{"name":"paprikaf","email":"felfelahmeed@gmail.com"},{"name":"pabloelisseo","email":"pabloelisseo@gmail.com"},{"name":"liamdebeasi","email":"ldebeasi@gmail.com"}],"dist":{"shasum":"e3a1f2dba4acd4f873d5a8ed751b2b3e1e55917c","tarball":"https://registry.npmjs.org/@builder.io/kube-cron-jobs/-/kube-cron-jobs-0.1.0.tgz","fileCount":4,"integrity":"sha512-KPwn8a4eU8UsGhw429Rxf/yY6vHC/Fpr94XbKSpHwk5LkoifaargOqU/Co+oFw2QnG3rBLyMiw/sZ3QBKR3bNA==","signatures":[{"sig":"MEQCIBb+lvW/jNv6Q6mNY8SZBWVeZAoS9xGYQh+epzNa5ps6AiB35SI8Odwh5qtKcDFpJUD0LWr8WhPXPz9OovtO4v+lbw==","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":21864},"type":"module","gitHead":"e7b67603b44f032bcfe3400b333204cf9f4b045f","_npmUser":{"name":"kylefowler","email":"kyle.e.fowler@gmail.com"},"_npmVersion":"10.9.2","description":"Kubernetes CronJobs for resources management and cleanup","directories":{},"_nodeVersion":"22.15.1","dependencies":{"tsx":"^4.7.0","@kubernetes/client-node":"^0.20.0"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/kube-cron-jobs_0.1.0_1762263745963_0.721032496614284","host":"s3://npm-registry-packages-npm-production"}},"0.1.1":{"name":"@builder.io/kube-cron-jobs","version":"0.1.1","_id":"@builder.io/kube-cron-jobs@0.1.1","maintainers":[{"name":"steve8708","email":"sewell.steve@gmail.com"},{"name":"samijaber","email":"jabersami@gmail.com"},{"name":"teleaziz123","email":"aziz@builder.io"},{"name":"mrkoreye","email":"korey@builder.io"},{"name":"mhevery","email":"misko@hevery.com"},{"name":"gustavohgs","email":"gustavo@builder.io"},{"name":"manucorporat","email":"manu.mtza@gmail.com"},{"name":"sanyamkamat","email":"sanyamkamat@gmail.com"},{"name":"bengirone","email":"benjamingirone@gmail.com"},{"name":"midhunadarvin","email":"midhunadarvin@gmail.com"},{"name":"kylefowler","email":"kyle.e.fowler@gmail.com"},{"name":"builderio-bot","email":"builder-bot@builder.io"},{"name":"armela","email":"armela@builder.io"},{"name":"aishwarya_parab","email":"aishwaryaparab1@gmail.com"},{"name":"emmaac","email":"emma@builder.io"},{"name":"anaghavarhade","email":"anagha@builder.io"},{"name":"sidmohanty11","email":"sidmohanty11@gmail.com"},{"name":"jcortesebuilder","email":"jcortese@builder.io"},{"name":"vishwasgopinath","email":"vishwas@builder.io"},{"name":"nicholaskoech","email":"nicholas@builder.io"},{"name":"floating_dynamo","email":"shridharmaskeri@gmail.com"},{"name":"paprikaf","email":"felfelahmeed@gmail.com"},{"name":"pabloelisseo","email":"pabloelisseo@gmail.com"},{"name":"liamdebeasi","email":"ldebeasi@gmail.com"}],"dist":{"shasum":"b858791d455749677752ba36bd29441d3943e392","tarball":"https://registry.npmjs.org/@builder.io/kube-cron-jobs/-/kube-cron-jobs-0.1.1.tgz","fileCount":8,"integrity":"sha512-ZkFr23xqnGOWM3Bf8u0Q7lm4d/BjtPNkt8Z1NGDmSTMLUAYaMBEc9SSw8pOQ2XnERcigSfm7SR0Goe2sUmmzrw==","signatures":[{"sig":"MEYCIQDj6E9bKakv6+5G75CFw4acVxMdUuUFwFI/4IQT9SYyNwIhAKwD2EXFjlxVt763S3AdFWse2CiAaLFvx7IIttmEHQrg","keyid":"SHA256:DhQ8wR5APBvFHLF/+Tc+AYvPOdTpcIDqOhxsBHRwC7U"}],"unpackedSize":110721},"type":"module","gitHead":"9422c719c4c2ccc063e416564f8a03e23f39e091","imports":{"#ai-utils":"@builder.io/ai-utils"},"_npmUser":{"name":"manucorporat","email":"manu.mtza@gmail.com"},"_npmVersion":"10.9.2","description":"Kubernetes CronJobs for resources management and cleanup","directories":{},"_nodeVersion":"22.15.1","dependencies":{"tsx":"^4.7.0","@builder.io/ai-utils":"*","@kubernetes/client-node":"^1.4.0"},"_hasShrinkwrap":false,"_npmOperationalInternal":{"tmp":"tmp/kube-cron-jobs_0.1.1_1780079818258_0.3363643011246795","host":"s3://npm-registry-packages-npm-production"}}},"time":{"created":"2025-11-04T13:42:25.868Z","modified":"2026-09-09T09:16:32.666Z","0.1.0":"2025-11-04T13:42:26.153Z","0.1.1":"2026-05-29T18:36:58.456Z"},"description":"Kubernetes CronJobs for resources management and cleanup","maintainers":[{"email":"sewell.steve@gmail.com","name":"steve8708"},{"email":"jabersami@gmail.com","name":"samijaber"},{"email":"aziz@builder.io","name":"teleaziz123"},{"email":"korey@builder.io","name":"mrkoreye"},{"email":"misko@hevery.com","name":"mhevery"},{"email":"gustavo@builder.io","name":"gustavohgs"},{"email":"manu.mtza@gmail.com","name":"manucorporat"},{"email":"sanyamkamat@gmail.com","name":"sanyamkamat"},{"email":"benjamingirone@gmail.com","name":"bengirone"},{"email":"kyle.e.fowler@gmail.com","name":"kylefowler"},{"email":"builder-bot@builder.io","name":"builderio-bot"},{"email":"armela@builder.io","name":"armela"},{"email":"aishwaryaparab1@gmail.com","name":"aishwarya_parab"},{"email":"emma@builder.io","name":"emmaac"},{"email":"sidmohanty11@gmail.com","name":"sidmohanty11"},{"email":"jcortese@builder.io","name":"jcortesebuilder"},{"email":"vishwas@builder.io","name":"vishwasgopinath"},{"email":"nicholas@builder.io","name":"nicholaskipchumba"},{"email":"shridharmaskeri@gmail.com","name":"floating_dynamo"},{"email":"lautarogalarza@gmail.com","name":"lihuelg"},{"email":"felfelahmeed@gmail.com","name":"paprikaf"},{"email":"pabloelisseo@gmail.com","name":"pabloelisseo"},{"email":"ldebeasi@gmail.com","name":"liamdebeasi"}],"readme":"# Kubernetes CronJobs for Resources Management\n\nThis package contains Kubernetes CronJobs for managing different resources and cleaning up project resources (PVCs, pods, ingresses...) automatically.\n\n## Pod Cleanup CronJob\n\nThe `pod-cleanup` CronJob prunes inactive fusion-managed pods from the cluster. It targets pods that are older than a configurable threshold and have not received recent traffic, keeping namespaces tidy and costs low.\n\n### How It Works\n\n1. Initializes the Kubernetes client using in-cluster config or external credentials.\n2. Lists `fusion-managed=true` pods across all namespaces in paginated batches (`POD_LIST_LIMIT`, default `100`).\n3. Filters candidates that:\n   - Are in the `Running` phase.\n   - Are older than `POD_AGE_THRESHOLD_HOURS` (default `5` hours).\n   - Have the `fusion-project` and `fusion-branch` labels so a hostname can be constructed.\n4. Groups hostnames into batches (`BATCH_SIZE`, default `50`) and calls the Cloud Run webhook `/projects/kube/webhook/check-traffic` to see if they received traffic within the `TRAFFIC_CHECK_WINDOW_MINUTES` window (default `10` minutes).\n5. Deletes pods that report no recent traffic via `deleteNamespacedPod`, logging successes and failures.\n\nThe job continues processing subsequent pages until no `continue` token remains, logging aggregate metrics for observability.\n\n### Environment Variables\n\n- `POD_AGE_THRESHOLD_HOURS` – Minimum age before a pod becomes a deletion candidate (default `5`).\n- `TRAFFIC_CHECK_WINDOW_MINUTES` – Look-back window for ingress traffic (default `10`).\n- `BATCH_SIZE` – Number of hostnames per traffic check request (default `50`).\n- `POD_LIST_LIMIT` – Kubernetes API page size when listing pods (default `100`).\n- `CLOUD_RUN_SERVICE_URL` – Base URL for the Cloud Run service handling traffic checks (required).\n- `KUBERNETES_WEBHOOK_TOKEN` – Token appended as `token` query parameter when calling the webhook (required).\n- `KUBE_DOMAIN` – Domain suffix used to construct pod hostnames (default `builderio.xyz`).\n\n> **Important:** If `CLOUD_RUN_SERVICE_URL` or `KUBERNETES_WEBHOOK_TOKEN` are missing, or if the webhook call fails, the job currently treats pods as having no traffic and deletes them. Ensure these variables are set and the webhook is healthy before running in production.\n\n### Performance Configuration\n\n- Pagination keeps memory usage low when large numbers of pods are present.\n- Batch traffic checks minimize Cloud Logging queries and reduce latency.\n- A 60-second timeout guards the webhook request to avoid hanging the CronJob.\n\n### Kubernetes Configuration\n\nDeployment is defined in Terraform at `terraform/kubernetes/main.tf`. GitHub Actions workflows automate image builds (`.github/workflows/build-pod-cleanup.yml`) and deployments (`.github/workflows/deploy-pod-cleanup.yml`).\n\n### Local Development\n\n```bash\n# Set required environment variables\nexport CLOUD_RUN_SERVICE_URL=\"https://your-service.run.app\"\nexport KUBERNETES_WEBHOOK_TOKEN=\"your-token\"\n\n# Optional tuning overrides\nexport POD_AGE_THRESHOLD_HOURS=\"5\"\nexport TRAFFIC_CHECK_WINDOW_MINUTES=\"10\"\n\n# Run the job with tsx\nnpx tsx packages/kube-cron-jobs/src/pod-cleanup.ts\n```\n\n## PVC Lifecycle Manager (Unified)\n\nThe `pvc-lifecycle-manager` CronJob is a unified solution that handles the complete PVC lifecycle in a single job. It soft-deletes old PVCs and unblocks PVCs stuck in `Terminating` by removing the `snapshot.fusion.io/protect` finalizer.\n\n### How It Works\n\nThe lifecycle manager runs in two phases with **parallel processing** (up to 5 PVCs concurrently):\n\n**Phase 1: Policy check and Soft-Delete Old PVCs**\n1. Lists all PVCs with label `fusion-managed: true`\n2. Identifies PVCs older than threshold (default: 1 day)\n3. Calls Cloud Run `/projects/kube/webhook/can-delete-volume` to decide whether deletion is allowed (git backup validity + LaunchDarkly gating)\n4. If allowed, soft-deletes by calling `deleteNamespacedPersistentVolumeClaim` (sets `deletionTimestamp`)\n5. Immediately attempts to remove the finalizer via `/projects/kube/webhook/remove-finalizer` (falls back to Phase 2 if it fails)\n\n**Phase 2: Process Terminating PVCs (Fallback/Recovery)**\n1. Discovers PVCs with:\n   - Label: `fusion-managed: true`\n   - Finalizer: `snapshot.fusion.io/protect`\n   - `deletionTimestamp` set (PVC is being deleted)\n2. Removes the finalizer via `/projects/kube/webhook/remove-finalizer` to unblock deletion (this is intentionally “cluster-unblock” oriented)\n3. Kubernetes hard-deletes the PVC after finalizer removal\n\n### Building the Docker Image\n\nImages are automatically built and pushed by GitHub Actions workflows. For manual builds:\n\n```bash\n# From the ai-services root directory\ndocker build -f Dockerfile.pvc-lifecycle-manager -t us-central1-docker.pkg.dev/PROJECT_ID/kube/pvc-lifecycle-manager:latest .\n\n# Push to Artifact Registry\ndocker push us-central1-docker.pkg.dev/PROJECT_ID/kube/pvc-lifecycle-manager:latest\n```\n\n### Environment Variables\n\n- `PVC_AGE_THRESHOLD_DAYS` - Age threshold for PVC deletion (default: 1 day)\n- `CLOUD_RUN_SERVICE_URL` - URL of the Cloud Run service (e.g., `https://your-service.run.app`)\n- `KUBERNETES_WEBHOOK_TOKEN` - Token for authenticating with Cloud Run endpoints\n\n### Performance Configuration\n\n- **Parallel Processing**: Up to 5 PVCs processed concurrently (`MAX_CONCURRENT_OPERATIONS`)\n- **Decision Timeout**: 2 minutes per `/can-delete-volume` call\n- **Job Deadline**: 30 minutes total for the entire CronJob execution\n\n### Kubernetes Configuration\n\nThe CronJob is configured via Terraform in `terraform/kubernetes/main.tf`. Key settings:\n\n- **Schedule**: Daily at 3:00 AM (`0 3 * * *`)\n- **Concurrency Policy**: Allow parallel executions (multiple PVCs processed simultaneously)\n- **Job Deadline**: 30 minutes per job (`active_deadline_seconds: 1800`)\n- **Service Account**: `pvc-lifecycle-manager-service-account` (with required RBAC permissions)\n- **Parallel Operations**: 5 PVCs processed concurrently within each job\n\n### Local Development\n\n```bash\n# Set environment variables\nexport PVC_AGE_THRESHOLD_DAYS=\"1\"\nexport CLOUD_RUN_SERVICE_URL=\"https://your-service.run.app\"\nexport KUBERNETES_WEBHOOK_TOKEN=\"your-token\"\n\n# Run directly with tsx\nnpx tsx packages/kube-cron-jobs/src/pvc-lifecycle-manager.ts\n```\n\n### Safety Features\n\n1. **Policy-gated deletion**: Old PVCs are only deleted when `/can-delete-volume` allows it (backup validity + LaunchDarkly gating)\n2. **In-use protection**: PVCs with `kubernetes.io/pvc-protection` finalizer are skipped (pod is using them)\n3. **Parallel processing**: Errors in one PVC don't stop processing of others\n4. **Cluster-unblock**: Terminating PVCs always have finalizer removal attempted to avoid scale/deadlock scenarios\n\n### Cloud Logging Queries\n\nThe CronJob outputs structured JSON logs to GCP Cloud Logging. You can query these logs in **GCP Console → Logging → Logs Explorer**.\n\n#### Queryable Fields\n\nAll logs include the following structured fields:\n- `severity` - Log level: `DEBUG`, `INFO`, `WARNING`, `ERROR`, `CRITICAL`\n- `timestamp` - ISO 8601 timestamp\n- `message` - Human-readable message\n- `operation` - Operation identifier (see list below)\n- `namespace` - Kubernetes namespace\n- `pvcName` - PVC name\n- `pvcKey` - Namespace/PVC identifier (`namespace/pvcName`)\n- `ageDays` - Age of PVC in days\n- `error` - Error message/details (when applicable)\n\n**Operation Values:**\n- `cronjob_start` - Job initialization\n- `cronjob_complete` - Successful completion\n- `cronjob_execution` - Job failures\n- `categorize_pvcs` - PVC categorization phase\n- `phase_1_start` / `phase_1_complete` - Old PVC cleanup phase\n- `phase_2_start` / `phase_2_complete` - Terminating PVC processing phase\n- `pvc_soft_deletion` - PVC soft-deletion attempts\n- `backup_and_delete` - Policy check + deletion workflow\n- `process_terminating_pvc` - Processing terminating PVCs\n- `finalizer_removal` - Finalizer removal operations\n\n#### Example Queries\n\n**1. View all logs from the CronJob:**\n```\nresource.type=\"k8s_container\"\nresource.labels.pod_name=~\"pvc-lifecycle-manager-.*\"\n```\n\n**2. Monitor critical issues requiring manual intervention:**\n```\nresource.type=\"k8s_container\"\nresource.labels.pod_name=~\"pvc-lifecycle-manager-.*\"\n(severity=\"CRITICAL\" OR jsonPayload.requiresManualIntervention=true)\n```\n\n**3. Track a specific PVC through its lifecycle:**\n```\nresource.type=\"k8s_container\"\nresource.labels.pod_name=~\"pvc-lifecycle-manager-.*\"\njsonPayload.pvcName=\"your-pvc-name\"\n```\n\n**4. View only errors and warnings:**\n```\nresource.type=\"k8s_container\"\nresource.labels.pod_name=~\"pvc-lifecycle-manager-.*\"\nseverity >= WARNING\n```\n\n**5. Monitor policy check + deletion operations:**\n```\nresource.type=\"k8s_container\"\nresource.labels.pod_name=~\"pvc-lifecycle-manager-.*\"\njsonPayload.operation=\"backup_and_delete\"\n```\n\n## Namespace Cleanup CronJob\n\nThe `namespace-cleanup` CronJob deletes empty Kubernetes namespaces for projects that are no longer active.\n\n### How It Works\n\n1. Lists all namespaces with label `fusion-managed=true` in paginated batches (`NAMESPACE_LIST_LIMIT`, default `100`).\n2. For each namespace, skips if:\n   - **Too young** — created less than `MIN_NAMESPACE_AGE_HOURS` ago (default `12`). Guards against a race condition where `ensure-container` has created the namespace but not yet provisioned the PVC.\n   - **Has PVCs** — at least one `fusion-managed=true` PVC exists. Means the pod is still running or `pvc-lifecycle-manager` hasn't cleaned up yet.\n   - **Has VolumeSnapshots** — a `base-snapshot` VolumeSnapshot exists. This is a pre-built disk image used to fast-clone PVCs for new branches. Deleting the namespace would destroy it.\n3. If none of the above apply the namespace is empty and safe to delete.\n\n### Environment Variables\n\n- `MIN_NAMESPACE_AGE_HOURS` – Minimum namespace age before it becomes a deletion candidate (default `12`).\n- `NAMESPACE_LIST_LIMIT` – Kubernetes API page size when listing namespaces (default `100`).\n- `CONCURRENCY_LIMIT` – Number of namespaces processed in parallel per batch (default `10`).\n- `DRY_RUN` – When `true`, all checks run normally but no namespaces are deleted. Useful for testing (default `false`).\n\n### Kubernetes Configuration\n\nThe CronJob is configured via Terraform in `terraform/kubernetes/main.tf`. Key settings:\n\n- **Schedule**: Daily at 4:00 AM (`0 4 * * *`) — one hour after `pvc-lifecycle-manager` runs, so PVCs are already cleaned up before this job checks.\n- **Concurrency Policy**: Replace (only one instance runs at a time).\n- **Job Deadline**: 30 minutes (`active_deadline_seconds: 1800`).\n- **Service Account**: `namespace-cleanup-service-account` with `list/delete` on namespaces, `list` on PVCs and VolumeSnapshots.\n\n### Local Development\n\n```bash\n# Run in dry-run mode against your local kubeconfig\nDRY_RUN=true npx tsx packages/kube-cron-jobs/src/namespace-cleanup.ts\n\n```\n","readmeFilename":"README.md"}