# PXL Backend Import

The import module provides a set of helpful utilities to implement data imports. Out of the box, it comes with a complete import from Excel to the database - including the ability to detect data issues and derive a delta to the current data.

## Module components

### Load & decode data

This module does not provide specific functionality to load and decode data. Instead, it relies on the [decoders](../decoders) package to decode data (e.g. from flattenend Excel data to nested objects). To load data from Excel files, we use the [xlport](../xlport) package.

### Detect delta

-> see [./detect-delta](./detect-delta) folder

Set of functions and types to detect the delta between the current data in the repositories and newly provided data

The main function is `detectDelta`, which takes the data service (to access existing data in the repositories) and the decoded new data - and returns a delta for each model.

### Convert functions

-> see [./convert-import.functions](./convert-import.functions.ts)

Provides two functions:

- `deltaToBulkMutations` converts the delta to bulk mutations
- `importedDataToBulkMutations` converts the decoded data to bulk mutations

The latter function does not require the delta, ie it should only be used for seeding!

### Import service

-> see [./import.service.ts](./import.service.ts)

This service provides the dispatcher for the different import actions. By default, it provides the following actions:

- `excel`: Imports data from an Excel file
- `excelSeed`: Imports data from an Excel file and seeds the data
- `data`: Imports data from a data file
- `dataSeed`: Imports data from a data file and seeds the data

These actions can be used in seeding or in other custom import scenarios.

## Side note: Transformations

During seeding, Excel imports and potentially other larger migrations/data operations, we typically create an Action - which then naturally results in multiple Mutations (create, update, edit, createMany, etc.) to alter the data.
Before triggering the mutations, we typically need to preprocess the data, e.g.:

- Extract the data from the Excel file, then decode it with Zod
- Read the CSV/JSON file(s), then decode it with Zod
- After decoding, we need to detect the delta and convert it to BulkUpdates

Each of these steps can fail and lead to distinct errors. To debug these, we create a "Transformation" operation for each each step - and store it (together with the step's payload and result) in the operations table, assigned to the action. This uses the same mechanism as the "Mutation" operation.
