# Migration Guide: 11.23.x → 11.24.0

## Overview

| Category | Details |
|----------|---------|
| **Breaking Changes** | None (one edge-case: `checkRestricted` 4th parameter type changed — see below) |
| **New Features** | `CrudService.pushToArray()`, `CrudService.pullFromArray()` |
| **Bugfixes** | `checkRestricted` group member checks broken (concat bug), `prepareInput`/`prepareOutput` array element skipping (splice bug), RoleGuard missing `$pull.roles` guard |
| **Performance** | `checkRestricted` sample-based array validation, `WeakSet` for circular reference detection, single-pass DFS in `prepareOutput` (secret removal + ObjectId conversion merged) |
| **Security** | Recursive secret removal in `prepareOutput`, RoleGuard now guards `$pull.roles`, field-name validation on `pushToArray`/`pullFromArray` |
| **Migration Effort** | 0 minutes (no code changes required) |

---

## Quick Migration

No code changes required. All improvements activate automatically.

```bash
# Update package
pnpm add @lenne.tech/nest-server@11.24.0

# Verify build
pnpm run build

# Run tests
pnpm test
```

---

## What's New in 11.24.0

### 1. `CrudService.pushToArray()` — Atomic Array Append

New first-class method for appending items to array fields without loading the full array into memory. Bypasses the `process()` pipeline entirely, preventing OOM on large subdocument arrays. Mongoose plugins (Tenant, Audit, RoleGuard) remain active via `pre('findOneAndUpdate')`.

```typescript
// Append a single item
await this.entityService.pushToArray(id, 'logs', newLog);

// Append with $slice to cap array length (keep last 500)
await this.entityService.pushToArray(id, 'logs', newLog, { $slice: -500 });

// Append multiple items at once
await this.entityService.pushToArray(id, 'tags', ['urgent', 'reviewed']);

// Append with sort and position control
await this.entityService.pushToArray(id, 'scores', newScore, {
  $sort: { value: -1 },
  $slice: 10,  // keep top 10
});
```

**When to use:**
- Appending logs, history, metrics, comments, tags
- Any growing array regardless of size
- Whenever you would have done `entity.field.push(item); await service.update(id, { field: entity.field })`

**When to still use `CrudService.update()`:**
- Replacing the entire array content
- Small arrays (< 10 items) where full validation is desired
- User-controlled field selection requiring `@Restricted` input checks

### 2. `CrudService.pullFromArray()` — Atomic Array Removal

New method for removing items from array fields by condition.

```typescript
// Remove by exact match
await this.entityService.pullFromArray(id, 'tags', 'obsolete');

// Remove by condition (all DEBUG logs)
await this.entityService.pullFromArray(id, 'logs', { level: 'DEBUG' });
```

### 3. `checkRestricted()` Array Performance Optimization

For typed arrays (all items share the same class), class-level `@Restricted` metadata is now validated once on the first item instead of on every item. Per-item checks are only performed when `S_CREATOR` or `S_SELF` restrictions exist (because `createdBy`/`id` differ per item).

**Impact:** Reduces redundant class-level validation from O(n) to O(1) for homogeneous arrays. Property-level checks still run per item with cached metadata lookups.

### 4. `checkRestricted()` Circular Reference Protection — WeakSet

The `processedObjects` tracking changed from `Array` with O(n) `.includes()` to `WeakSet` with O(1) `.has()`. This improves performance for deeply nested object graphs and enables garbage collection of processed objects.

### 5. Recursive Secret Removal in `prepareOutput()`

Secret fields (`password`, `verificationToken`, `passwordResetToken`) are now removed **recursively** from nested objects and arrays, not just at the top level. Previously, `output.author.password` would survive — now it is correctly removed.

**Note:** `refreshTokens` and `tempTokens` are intentionally NOT removed by `prepareOutput` — they are needed internally for token validation and process flows (password reset, email verification). The `CheckSecurityInterceptor` (Safety Net) removes these from HTTP responses as a separate layer.

### 6. Recursive ObjectId Conversion in `prepareOutput()`

ObjectId→string conversion now uses `processDeep()` (recursive) instead of a flat `Object.entries()` loop, consistent with `prepareInput()`. Previously, nested ObjectIds (e.g. `output.nested.refId`) were not converted.

### 7. Single-Pass DFS in `prepareOutput()`

Secret removal and ObjectId conversion are now merged into a single `processDeep()` traversal instead of two separate recursive walks. This halves the object graph traversal cost for API responses with nested objects.

### 8. Field-Name Validation on `pushToArray()` / `pullFromArray()`

Both methods now validate the `field` parameter at runtime. Field names starting with `$` or containing null bytes are rejected with an error. This prevents MongoDB operator injection when field names are accidentally derived from unvalidated sources.

```typescript
// Throws: "pushToArray: invalid field name "$where""
await this.entityService.pushToArray(id, '$where', 'value');
```

### 9. `pushToArray()` Empty Array Guard

Passing an empty array to `pushToArray()` now returns immediately without issuing a database round-trip. Previously, `$push: { field: { $each: [] } }` was sent to MongoDB as a valid no-op.

---

## Bugfixes

### Fix: `checkRestricted` Group Member Checks Were Silently Broken

**File:** `src/core/common/decorators/restricted.decorator.ts`

`members.concat(items)` returned a new array but the result was discarded. This meant `@Restricted({ memberOf: 'members' })` group-based access checks **never worked** for array-valued membership properties.

**Before:**
```typescript
members.concat(items);  // BUG: result discarded, members stays empty
```

**After:**
```typescript
members.push(...items);  // FIXED: items added to members array
```

**Impact:** If your project uses `@Restricted({ memberOf: 'propertyName' })` where the property is an array of user IDs, access was incorrectly denied for all users. After this fix, group member checks work correctly.

### Fix: `prepareInput()` / `prepareOutput()` Array Element Skipping

**File:** `src/core/common/helpers/service.helper.ts`

Both functions used `splice(i, 1)` inside a forward `for` loop when `removeUndefined` was active. This caused array elements to be skipped after each removal because indices shifted.

**Before:**
```typescript
for (let i = 0; i <= input.length - 1; i++) {
  processedArray[i] = await prepareInput(input[i], ...);
  if (processedArray[i] === undefined && config.removeUndefined) {
    processedArray.splice(i, 1);  // BUG: shifts indices, next element skipped
  }
}
```

**After:**
```typescript
const result = [];
for (let i = 0; i < input.length; i++) {
  const processed = await prepareInput(input[i], ...);
  if (processed !== undefined || !config.removeUndefined) {
    result.push(processed);
  }
}
return result;
```

**Impact:** Arrays with `undefined` elements (rare in practice) could have had elements silently dropped. The fix also ensures arrays are never mutated in place — a new array is always returned.

### Fix: RoleGuard Plugin Did Not Guard `$pull.roles`

**File:** `src/core/common/plugins/mongoose-role-guard.plugin.ts`

The `mongooseRoleGuardPlugin` checked `$push.roles`, `$set.roles`, and `$addToSet.roles` for unauthorized role changes, but not `$pull.roles`. This meant `Model.findByIdAndUpdate(id, { $pull: { roles: 'admin' } })` could strip roles without authorization.

**Impact:** The new `pullFromArray()` method exposed this gap. After this fix, `$pull.roles` is also guarded — unauthorized callers cannot remove roles via any MongoDB operator.

---

## Edge-Case Breaking Change

### `checkRestricted()` 4th Parameter Type: `any[]` → `WeakSet<object>`

The `processedObjects` parameter of the exported `checkRestricted()` function changed from `any[]` to `WeakSet<object>`. This parameter has a default value (`new WeakSet()`) and is only used internally for recursive calls.

**Who is affected:** Only projects that call `checkRestricted()` directly AND pass a custom 4th argument. This is extremely unlikely — the function is typically called by the framework interceptors, not by consuming projects.

**If you are affected:**
```typescript
// Before (11.23.x):
checkRestricted(data, user, options, []);

// After (11.24.0):
checkRestricted(data, user, options, new WeakSet());
// Or simply omit the 4th parameter (recommended):
checkRestricted(data, user, options);
```

---

## Deprecations

### `getNewArray` Option in `PrepareInputOptions` / `PrepareOutputOptions`

The `getNewArray` option is now deprecated. Array processing always returns a new array to prevent mutation bugs. The option is still accepted for backward compatibility but has no effect.

---

## Compatibility Notes

### Existing Code

All existing code continues to work without changes:

- `CrudService.update()` works identically — the `process()` pipeline is unchanged
- `@Restricted()` decorators behave the same (with the group member bug now fixed)
- `prepareInput()` / `prepareOutput()` produce the same results (with bugs fixed)
- Direct Mongoose operations (`Model.findByIdAndUpdate()`, `$push`, etc.) work as before

### Adopting `pushToArray()` / `pullFromArray()` (Recommended)

If your project has patterns like:

```typescript
entity.logs.push(newLog);
await this.entityService.update(id, { logs: entity.logs });
```

Consider migrating to:

```typescript
await this.entityService.pushToArray(id, 'logs', newLog);
```

This is especially important for:
- Arrays that grow over time (logs, history, comments)
- Documents with hundreds of subdocument entries
- High-frequency operations (monitoring, metrics)

### Security Architecture

The multi-layered security architecture ensures safety regardless of which method is used:

| Layer | What it does | Active for pushToArray? |
|-------|-------------|------------------------|
| Mongoose Tenant Plugin | Scopes queries to tenant | Yes |
| Mongoose Audit Plugin | Sets `updatedBy` | Yes |
| Mongoose RoleGuard Plugin | Prevents unauthorized role changes | Yes |
| `process()` pipeline | Input validation, authorization, output filtering | No (bypassed) |
| CheckResponseInterceptor | Filters `@Restricted` fields from HTTP response | Yes (Safety Net) |
| CheckSecurityInterceptor | Runs `securityCheck()`, removes secrets from HTTP response | Yes (Safety Net) |

---

## Troubleshooting

### Group member checks now grant access where they were denied before

This is expected — the `concat()` bug meant group member checks were silently broken. If your project relied on the (incorrect) denial behavior, review your `@Restricted({ memberOf: '...' })` usage.

### Nested ObjectIds now appear as strings in API responses

Previously, ObjectIds nested inside objects (e.g. `response.nested.refId`) were returned as ObjectId objects. They are now consistently converted to strings, matching the behavior of top-level ObjectIds. If your frontend parsed ObjectId objects, this is now a plain string.

### Nested secret fields now removed from service-level output

If your code accessed `result.author.password` after a CrudService call (within the service layer, not in API responses), this value is now `undefined`. This is the correct behavior — passwords should never be accessible in output. If you need the raw data, use `updateRaw()` / `getRaw()` or direct Mongoose queries.

---

## References

- [SubDocument Array Optimization](../docs/subdocument-array-optimization-plan.md) — Detailed OOM analysis and implementation notes
- [process() Performance Optimization](../docs/process-performance-optimization.md) — Pipeline optimization details
- [nest-server-starter](https://github.com/lenneTech/nest-server-starter) — Reference implementation
