snapshot of https://github.com/RocketChat/Rocket.Chat.git (develop, 2026-09-16; blobs > 131072B omitted)
Some checks failed
Create issues from TODOs / todos (push) Failing after 3s
Close inactive issues / close-issues (push) Has been cancelled
Deploy GitHub Pages / deploy-preview (push) Has been cancelled
CI / ⚙️ Variables Setup (push) Has been cancelled
CI / ⚙️ Test Guard (push) Has been cancelled
CI / 🚀 Notify external services - draft (push) Has been cancelled
CI / 📦 Build Packages (push) Has been cancelled
CI / 📦 Meteor Build (coverage) (push) Has been cancelled
CI / 📦 Meteor Build (production) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [account-service presence-service omnichannel-transcript-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [account-service presence-service omnichannel-transcript-service], fips) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [authorization-service queue-worker-service ddp-streamer-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [authorization-service queue-worker-service ddp-streamer-service], fips) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], coverage) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], fips) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [account-service presence-service omnichannel-transcript-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [authorization-service queue-worker-service ddp-streamer-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [rocketchat], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [rocketchat], coverage) (push) Has been cancelled
CI / 🚢 Publish Docker Images (ghcr.io) (push) Has been cancelled
CI / 📦 Track Image Sizes (push) Has been cancelled
CI / 🔎 Code Check - Action Lint (push) Has been cancelled
CI / 🔎 Code Check (push) Has been cancelled
CI / 🔨 Test Storybook (push) Has been cancelled
CI / 🔨 Test Unit (push) Has been cancelled
CI / 🔨 Test API (CE) (push) Has been cancelled
CI / 🔨 Test API Livechat (CE) (push) Has been cancelled
CI / 🔨 Test UI (CE) (push) Has been cancelled
CI / 🔨 Test API (EE) (push) Has been cancelled
CI / 🔨 Test API Livechat (EE) (push) Has been cancelled
CI / 🔨 Test API Apps (deno-runtime - EE) (push) Has been cancelled
CI / 🔨 Test UI (EE) (push) Has been cancelled
CI / 🔨 Test API (FIPS) (push) Has been cancelled
CI / 🔨 Test API Livechat (FIPS) (push) Has been cancelled
CI / 🔨 Test UI (FIPS) (push) Has been cancelled
CI / 🔨 Test Federation Matrix (push) Has been cancelled
CI / 📊 Report Coverage (push) Has been cancelled
CI / ✅ Tests Done (push) Has been cancelled
CI / 🚀 Publish build assets (push) Has been cancelled
CI / 🚀 Publish Docker Images (DockerHub) (push) Has been cancelled
CI / 🚀 Notify external services (push) Has been cancelled
CI / Update Version Durability (push) Has been cancelled
Release / Release (push) Has been cancelled
Code scanning - action / CodeQL-Build (push) Failing after 2s
Auto-close duplicate issues / auto-close-duplicates (push) Failing after 1s
Some checks failed
Create issues from TODOs / todos (push) Failing after 3s
Close inactive issues / close-issues (push) Has been cancelled
Deploy GitHub Pages / deploy-preview (push) Has been cancelled
CI / ⚙️ Variables Setup (push) Has been cancelled
CI / ⚙️ Test Guard (push) Has been cancelled
CI / 🚀 Notify external services - draft (push) Has been cancelled
CI / 📦 Build Packages (push) Has been cancelled
CI / 📦 Meteor Build (coverage) (push) Has been cancelled
CI / 📦 Meteor Build (production) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [account-service presence-service omnichannel-transcript-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [account-service presence-service omnichannel-transcript-service], fips) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [authorization-service queue-worker-service ddp-streamer-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [authorization-service queue-worker-service ddp-streamer-service], fips) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], coverage) (push) Has been cancelled
CI / 🚢 Build Docker (amd64, [rocketchat], fips) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [account-service presence-service omnichannel-transcript-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [authorization-service queue-worker-service ddp-streamer-service], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [rocketchat], ${{ (github.event_name != 'release' && github.ref != 'refs/heads/develop') && 'coverage' || 'production' }}) (push) Has been cancelled
CI / 🚢 Build Docker (arm64, [rocketchat], coverage) (push) Has been cancelled
CI / 🚢 Publish Docker Images (ghcr.io) (push) Has been cancelled
CI / 📦 Track Image Sizes (push) Has been cancelled
CI / 🔎 Code Check - Action Lint (push) Has been cancelled
CI / 🔎 Code Check (push) Has been cancelled
CI / 🔨 Test Storybook (push) Has been cancelled
CI / 🔨 Test Unit (push) Has been cancelled
CI / 🔨 Test API (CE) (push) Has been cancelled
CI / 🔨 Test API Livechat (CE) (push) Has been cancelled
CI / 🔨 Test UI (CE) (push) Has been cancelled
CI / 🔨 Test API (EE) (push) Has been cancelled
CI / 🔨 Test API Livechat (EE) (push) Has been cancelled
CI / 🔨 Test API Apps (deno-runtime - EE) (push) Has been cancelled
CI / 🔨 Test UI (EE) (push) Has been cancelled
CI / 🔨 Test API (FIPS) (push) Has been cancelled
CI / 🔨 Test API Livechat (FIPS) (push) Has been cancelled
CI / 🔨 Test UI (FIPS) (push) Has been cancelled
CI / 🔨 Test Federation Matrix (push) Has been cancelled
CI / 📊 Report Coverage (push) Has been cancelled
CI / ✅ Tests Done (push) Has been cancelled
CI / 🚀 Publish build assets (push) Has been cancelled
CI / 🚀 Publish Docker Images (DockerHub) (push) Has been cancelled
CI / 🚀 Notify external services (push) Has been cancelled
CI / Update Version Durability (push) Has been cancelled
Release / Release (push) Has been cancelled
Code scanning - action / CodeQL-Build (push) Failing after 2s
Auto-close duplicate issues / auto-close-duplicates (push) Failing after 1s
This commit is contained in:
285
docs/proposals/apps-converters-zod/README.md
Normal file
285
docs/proposals/apps-converters-zod/README.md
Normal file
@@ -0,0 +1,285 @@
|
||||
# Proposal: Migrate the Apps-Engine converters to Zod codecs
|
||||
|
||||
## Status
|
||||
|
||||
In progress (PR #41205). Phases 0–5 landed; `transformMappedData` removed. See the per-phase
|
||||
notes below, including the messages/threads deviation in Phase 4.
|
||||
|
||||
## Problem
|
||||
|
||||
`apps/meteor/app/apps/server/converters/` translates data between two models in both
|
||||
directions:
|
||||
|
||||
- **Rocket.Chat → Apps-Engine** ("to app"): Mongo documents (`IUser`, `IRoom`, `IMessage`, …)
|
||||
become Apps-Engine objects (`IAppsUser`, `IAppsRoom`, `IAppsMessage`, …). Methods such as
|
||||
`convertToApp` / `convertRoom` / `convertMessage`.
|
||||
- **Apps-Engine → Rocket.Chat** ("from app"): the reverse. Methods such as `convertToRocketChat` /
|
||||
`convertApp*`.
|
||||
|
||||
Today the two directions are implemented by two unrelated mechanisms:
|
||||
|
||||
- The **to-app** direction is driven by `transformMappedData`, a declarative field remapper
|
||||
(`{ to: 'from' | fn | { from, map, list } }`) that also collects everything it did not map into an
|
||||
`_unmappedProperties_` bucket.
|
||||
- The **from-app** direction is hand-written object construction with long chains of
|
||||
`...(cond && { … })` spreads, re-merging `_unmappedProperties_` at the end.
|
||||
|
||||
This split has real costs:
|
||||
|
||||
1. **Two sources of truth per entity.** The field mapping is expressed once in a `map` and again,
|
||||
inverted, in a hand-written builder. They drift.
|
||||
2. **No runtime validation at the boundary.** The from-app path trusts whatever an app sends and
|
||||
spreads it onto a Mongo document.
|
||||
3. **Weak typing.** Several converters were untyped `.js`; the mapping is stringly-typed and the
|
||||
relationship between the two directions is invisible to the compiler.
|
||||
|
||||
### Inventory
|
||||
|
||||
| Converter | Was | To-app | From-app | Async / DB | Notable hazards |
|
||||
|--------------------|-----------|:------:|:--------:|:----------:|-----------------|
|
||||
| `settings` | js → ts | ✅ | — | sync | `SettingType` enum |
|
||||
| `roles` | ts | ✅ | — | sync | trivial |
|
||||
| `videoConferences` | ts | ✅ | ✅ | sync | pass-through clone |
|
||||
| `visitors` | js → ts | ✅ | ✅ | sync | `_unmappedProperties_` bucket |
|
||||
| `departments` | js → ts | ✅ | ✅ | sync | `_unmappedProperties_` bucket |
|
||||
| `users` | js → ts | ✅ | ✅ | sync | `UserType` / `UserStatusConnection` enums, contextual `console.warn` |
|
||||
| `uploads` | js → ts | ✅ | ✅ | **async** | cross-converter (rooms/users/visitors) |
|
||||
| `rooms` | js → ts | ✅ | ✅ | **async** | cross-converter fan-out, `isPartial`, `secureFieldsMapper`, `RoomType` enum |
|
||||
| `messages` | js → ts | ✅ | ✅ | **async** | `WeakMap` + `cachedFunction` memoization, `isPartial`, attachment sub-map, visitor-sender fallback |
|
||||
| `threads` | ts | ✅ | — | **async** | duplicates the messages attachment map |
|
||||
| `contacts` | ts | ✅ | ✅ | sync | deep nested `list` reverse map |
|
||||
|
||||
Infrastructure: `transformMappedData.ts`, `cachedFunction.ts`, `convertMessageFiles.ts`.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Model each entity as a **Zod codec** — a single bidirectional artifact that replaces the split
|
||||
"map + hand-written inverse" pair and adds runtime validation.
|
||||
|
||||
Zod 4.3 (`~4.3.6`) is already a dependency, and the repo already ships a codec —
|
||||
`TimestampSchema` in `packages/core-typings/src/utils.ts`:
|
||||
|
||||
```ts
|
||||
export const TimestampSchema = z.codec(z.iso.datetime(), z.date(), {
|
||||
encode: (date) => date.toISOString(),
|
||||
decode: (str) => new Date(str),
|
||||
});
|
||||
```
|
||||
|
||||
`z.codec(RcSchema, AppSchema, { decode, encode })` maps directly onto our two directions:
|
||||
|
||||
- `decode` : Rocket.Chat → Apps-Engine (today's `convertToApp` / `convertRoom` / …)
|
||||
- `encode` : Apps-Engine → Rocket.Chat (today's `convertToRocketChat` / `convertApp*`)
|
||||
|
||||
`z.decode(codec, x)` / `z.encode(codec, x)` run them synchronously; `z.decodeAsync` / `z.encodeAsync`
|
||||
run them when the transforms are async.
|
||||
|
||||
The converter **classes stay** as the public façade (they implement the `IAppXConverter` interfaces
|
||||
and are reached via `orch.getConverters().get('x')`, used in ~25 files). Codecs are an internal
|
||||
implementation detail those methods delegate to.
|
||||
|
||||
## Key design decisions
|
||||
|
||||
1. **Async transforms → `z.decodeAsync` / `z.encodeAsync`.** rooms/messages/uploads/threads perform
|
||||
DB lookups inside the mapping. Their codec transforms are async and must be driven with the async
|
||||
entry points; the sync `z.decode` throws `$ZodAsyncError` on an async codec. Sync converters
|
||||
(settings, roles, users, visitors, departments, contacts, videoConferences) use plain
|
||||
`z.decode` / `z.encode`. *Validated against the installed Zod build during Phase 0.*
|
||||
|
||||
2. **Orchestrator dependency → codec factories.** Cross-converter converters cannot be static
|
||||
singletons — they need `orch`. Expose `createRoomCodec(orch)`, `createUploadsCodec(orch)`, etc.,
|
||||
returning a closure-bound codec. DB-free converters export a static codec constant.
|
||||
|
||||
3. **Preserve the `_unmappedProperties_` contract — do not drop it.** It is load-bearing: the reverse
|
||||
converters merge it back, the EE redactor (`ee/server/apps/lib/redactor.ts`) references the path,
|
||||
and `RoomBridge` reads it. A plain `z.object` strips unknown keys; `z.looseObject` keeps them
|
||||
inline but *without* the bucket. We will build a small reusable helper (working name
|
||||
`mappedCodec`) that reproduces `transformMappedData`'s bucket semantics exactly, so output is
|
||||
byte-identical. This keeps the migration behaviour-preserving rather than a behaviour change.
|
||||
The helper is intentionally **not** built up-front — it will be co-designed with the first
|
||||
bidirectional converter (Phase 2) to avoid guessing the abstraction.
|
||||
|
||||
4. **`isPartial` (rooms/messages from-app) stays in the class method, not the codec.** Partial mode
|
||||
skips required-field generation, skips the unmapped merge, and strips `undefined`. Model it as the
|
||||
class calling either `z.encode` (full) or a partial path; do not encode the flag into the schema.
|
||||
|
||||
5. **Enum conversions → shared codec constants.** `UserType`, `UserStatusConnection`, `RoomType`,
|
||||
`SettingType` become small codecs (like `TimestampSchema`) in `converters/codecs/enums.ts`,
|
||||
reproducing the current `switch` logic including the pass-through/upper-case fallbacks.
|
||||
Contextual `console.warn` calls that need data unavailable to a pure enum mapping (e.g. the
|
||||
affected user's id/username in the status-connection warning) stay in the converter layer.
|
||||
|
||||
6. **Memoization (messages/threads) stays.** The `WeakMap` + `cachedFunction` dedup of user/room
|
||||
lookups within a single conversion is preserved by constructing the message codec per-conversion
|
||||
through the factory, passing in the memoized lookups.
|
||||
|
||||
7. **Loose validation on the from-app path.** Apps send arbitrary data today; strict schemas would
|
||||
reject payloads that currently pass. Use `.loose()` / `.optional()` generously on the app-side
|
||||
schemas so the migration introduces no rejections. Tightening is a deliberate, separate follow-up.
|
||||
|
||||
8. **Preserve the public surface.** The `IAppXConverter` interfaces and the
|
||||
`orch.getConverters().get('x')` usage must not change.
|
||||
|
||||
### Direction convention (illustrative)
|
||||
|
||||
```ts
|
||||
const UserCodec = z.codec(UserRocketChatSchema, AppsUserSchema, {
|
||||
decode: (user) => ({ id: user._id, /* … */ }), // convertToApp
|
||||
encode: (appUser) => removeEmpty({ _id: appUser.id, /* … */ }), // convertToRocketChat
|
||||
});
|
||||
|
||||
// RC -> App: z.decode(UserCodec, user)
|
||||
// App -> RC: z.encode(UserCodec, appUser)
|
||||
```
|
||||
|
||||
## Migration phases
|
||||
|
||||
Each phase is independently shippable, keeps the class façade, and is gated by the golden tests.
|
||||
|
||||
### Phase 0 — Scaffolding & de-risk (done, PR #41205)
|
||||
|
||||
- Converted the 7 remaining `.js` converters to `.ts`, behaviour-preserving. `rooms`/`messages` keep
|
||||
loose typing on their transform maps for now; that tightens when each is codec-ified.
|
||||
- Fixed three pre-existing latent bugs that surfaced once the converters were typed (see
|
||||
[Incidental fixes](#incidental-fixes)). These are the only intentional behaviour changes in the
|
||||
phase and are each pinned by a golden/focused test.
|
||||
- Added `converters/codecs/` with the first shared primitives: bidirectional enum codecs
|
||||
(`UserType`, `UserStatusConnection`, `RoomType`, `SettingType`).
|
||||
- Added the **behavioural safety net**: enum-codec unit tests (asserting parity with the legacy
|
||||
helpers) and **golden-snapshot tests** locking the current RC ↔ Apps-Engine field mapping,
|
||||
including `_unmappedProperties_` bucketing, for settings, users, visitors, departments, roles,
|
||||
videoConferences, contacts and uploads.
|
||||
- Validated the async-codec approach (`z.decodeAsync` / `z.encodeAsync`, `$ZodAsyncError`) that
|
||||
Phases 3–4 depend on.
|
||||
|
||||
#### Incidental fixes
|
||||
|
||||
Typing the converters exposed three latent bugs in the legacy code. They are small, self-contained,
|
||||
and fixed in Phase 0 rather than carried forward as "behaviour to preserve":
|
||||
|
||||
1. **`users` — `utfOffset` typo (from-app).** `convertToRocketChat` read the misspelled
|
||||
`appUser.utfOffset`, so a round-tripped user always lost its `utcOffset`. It now reads
|
||||
`utcOffset` first and keeps `utfOffset` only as a compatibility fallback for any app still
|
||||
emitting the old key. The golden round-trip fixture uses `utcOffset`; a focused test pins the
|
||||
legacy fallback.
|
||||
2. **`messages` — dead visitor-sender fallback (to-app).** The `sender` resolver deleted
|
||||
`message.u` *before* the "old system message without token" fallback re-read it, so the fallback
|
||||
always received `undefined`. It now captures the sender before deletion. A focused test drives a
|
||||
missing primary lookup and asserts the fallback resolves the original sender.
|
||||
3. **`messages` — null editor deref (from-app).** `convertAppMessage` dereferenced
|
||||
`Users.findOneById(editor.id)` through a non-null assertion, throwing if the editor no longer
|
||||
exists. It now falls back to the editor data carried on the app payload, mirroring the adjacent
|
||||
sender handling.
|
||||
|
||||
Alongside these, a few defensive guards were added where a destructuring or lookup could hit an
|
||||
optional value that the types claimed was always present (`uploads` `rid`/`room`, `users`
|
||||
`convertByUsername`, `rooms` `visitorChannelInfo`). These only turn a latent `TypeError` into the
|
||||
already-intended empty result and match the guards their sibling fields already had, so they change
|
||||
no covered output.
|
||||
|
||||
### Phase 1 — Trivial, one-way, sync
|
||||
|
||||
`settings`, `roles`, `videoConferences`. Proves the codec + enum-codec pattern end-to-end at the
|
||||
lowest risk.
|
||||
|
||||
### Phase 2 — Self-contained, bidirectional, sync
|
||||
|
||||
`visitors`, `departments`, `users`. Introduces and hardens the `mappedCodec` unmapped-bucket helper
|
||||
and exercises the enum codecs with the `console.warn` side effects preserved in both directions.
|
||||
|
||||
### Phase 3 — Async, cross-converter
|
||||
|
||||
`uploads`, then `rooms`. Introduces the codec-factory pattern, `z.decodeAsync` / `z.encodeAsync`,
|
||||
`isPartial` handled in the class layer, and `secureFieldsMapper` integration.
|
||||
|
||||
### Phase 4 — Async + memoized + nested (done)
|
||||
|
||||
`messages`, `threads` and `contacts`. `contacts` became a clean bidirectional `ContactCodec`
|
||||
(`decode` clones; `encode` maps every attribute via the nested field map). `mappedDecodeAsync` was
|
||||
extended with the nested `{ from, map, list }` branch so it fully reproduces `transformMappedData`.
|
||||
|
||||
**Deviation — messages and threads are not standalone codecs.** Unlike every other converter, these
|
||||
two were migrated by swapping their `transformMappedData` calls for the shared `mappedDecodeAsync`
|
||||
helper *in place*, rather than being wrapped in `create*Codec` factories. Reasons:
|
||||
|
||||
1. **Test harness.** `messages.tests.js` uses `proxyquire` to stub `@rocket.chat/models`,
|
||||
`@rocket.chat/random` and `@rocket.chat/core-typings` (`isMessageFromVisitor`) on the `messages.ts`
|
||||
module. `proxyquire` only replaces a module's *own* `require`s, so moving `convertAppMessage`'s
|
||||
model/`Random` usage into a separate `codecs/messages.ts` module would bypass the stubs and break
|
||||
the test (real `Rooms.findOneById` → no `GENERAL` room → the "proper schema"/"invalid room" cases
|
||||
fail). Keeping that usage in the class preserves the harness without editing it.
|
||||
2. **Poor fit for the codec shape.** `convertMessage` builds its field map per call, closing over a
|
||||
per-message memoization cache (`WeakMap` + `cachedFunction`), the source `msgObj`
|
||||
(`isMessageFromVisitor`) and `mainFile`; `convertAppMessage` adds an `isPartial` flag and
|
||||
`Random.id`/`new Date()` defaults. None of this expresses cleanly as a static bidirectional
|
||||
`z.codec`, and forcing it would add casts and risk for no real gain.
|
||||
|
||||
The migration goal for these two is therefore narrower but still met: the `transformMappedData`
|
||||
dependency is removed and the mapping logic is the shared, tested `mappedDecodeAsync`. The
|
||||
attachment sub-maps stayed per-converter because messages' and threads' variants differ (threads has
|
||||
extra `author`/`timestamp`/`fileId` guards) — sharing them would change behaviour. Wrapping messages
|
||||
in a codec later remains possible (e.g. via `proxyquire`'s `@global` stubs) if desired.
|
||||
|
||||
### Phase 5 — Cleanup (done)
|
||||
|
||||
Every converter is off `transformMappedData`, so `transformMappedData.ts` was deleted and its
|
||||
(previously importer-located) spec was retired — its coverage now lives in the `mappedDecodeAsync`
|
||||
tests under `tests/unit/app/apps/server/codecs/`. `cachedFunction` stays: it is still used by the
|
||||
messages and threads memoization, which remained in the class layer (see the Phase 4 deviation).
|
||||
|
||||
## Testing strategy
|
||||
|
||||
- **Golden snapshots** (added in Phase 0) are the equivalence oracle: each phase must keep them green
|
||||
while swapping internals. Output is compared after `JSON.parse(JSON.stringify(...))` so Dates
|
||||
normalise to ISO strings and `undefined` fields drop — matching how payloads cross the app bridge.
|
||||
- **Enum-codec unit tests** assert `decode`/`encode` match the legacy `_convert*` helpers for every
|
||||
input, including fallbacks.
|
||||
- The existing `rooms.tests.ts` and `messages.tests.js` remain the oracle for the two hardest
|
||||
converters.
|
||||
- New codecs get their own focused tests as they are introduced.
|
||||
|
||||
## Risks
|
||||
|
||||
- **`_unmappedProperties_` fidelity** — the single biggest behavioural trap; the `mappedCodec` helper
|
||||
plus golden tests are the mitigation.
|
||||
- **From-app validation rejecting live app payloads** — mitigated by loose schemas; do not tighten
|
||||
during the migration.
|
||||
- **Async codec ergonomics** — de-risked in Phase 0, but the factory wiring for rooms/messages should
|
||||
still be spiked before committing to Phase 3.
|
||||
- **Test rewrite** — the existing tests use `proxyquire.noCallThru()` against module paths; codec-ing
|
||||
each converter means updating those loaders.
|
||||
|
||||
## Type strategy for the mapping helpers
|
||||
|
||||
The mapping helpers in `converters/codecs/mappedData.ts` are split by how well the source type is
|
||||
known, which is what makes generic typing worthwhile in some cases and not others:
|
||||
|
||||
- **Sync string maps → fully generic and inferred.** `mappedDecode<Source>(fieldMap)` and
|
||||
`createMappedCodec<Source>(fieldMap)` are generic over the source document type. The field map is
|
||||
typed `FieldMap<Source> = Record<string, Extract<keyof Source, string>>`, so a renamed or
|
||||
misspelled source field is a **compile error** (a `@ts-expect-error` test locks this). The decode
|
||||
result is the inferred `Decoded<Source, Map>` — each renamed target as an optional property plus the
|
||||
`_unmappedProperties_` bucket typed as `Omit<Source, mappedKeys>` — rather than `Record<string, any>`.
|
||||
`mappedEncode` is the inverse and returns `Partial<Source>`. `Source` defaults to a loose record, so
|
||||
untyped call sites (e.g. tests) still work. Consumers with clean document types (`visitors`,
|
||||
`departments`, `roles`) pass their `Source` and get the typo-safety for free.
|
||||
|
||||
- **Async/function/nested maps → loose map, generic result.** `mappedDecodeAsync` keeps its
|
||||
`AsyncFieldMap` loosely typed on purpose: its consumers (`rooms`, `messages`, `uploads`, `contacts`)
|
||||
map many optional/livechat-only fields that are *not* on the base document types (which is why the
|
||||
original converters used dynamic access), and the return shape depends on arbitrary function/nested
|
||||
entries. Fully inferring that would resurrect the large conditional type the old helper avoided. It
|
||||
does take a `Result` type parameter — `mappedDecodeAsync<IAppsUpload>(data, map)` — so call sites
|
||||
assert the produced shape once instead of trailing every call with `as unknown as Promise<…>`.
|
||||
|
||||
- **Each apps-type boundary keeps a single, documented cast.** The Apps-Engine interfaces
|
||||
(`IAppsVisitor`, …) are hand-authored and don't structurally equal the produced `Decoded`/mapped
|
||||
shape, so each affected codec boundary — supplied role, department, user and visitor — retains one
|
||||
localized `as unknown as <AppsType>` where the codec meets that interface. Those casts are the
|
||||
deliberate loose-validation seams (see design decision 7), not incidental looseness.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- Changing the `IAppXConverter` interfaces or any app-facing behaviour.
|
||||
- Tightening the from-app validation surface (tracked as a separate follow-up).
|
||||
- Migrating converters that are not in the directory above.
|
||||
104
docs/proposals/compact-ast-format.md
Normal file
104
docs/proposals/compact-ast-format.md
Normal file
@@ -0,0 +1,104 @@
|
||||
# Proposal: Compact AST Format for message-parser
|
||||
|
||||
## Status
|
||||
|
||||
Draft
|
||||
|
||||
## Problem
|
||||
|
||||
The current message-parser AST is verbose — every node carries its full text content as self-contained strings. For a message like `**hello** world`, the AST stores `"hello"` inside the Bold node and `" world"` in a separate Plain node, even though both substrings already exist in the original message. This redundancy inflates payload size, especially for messages with deeply nested formatting (bold inside italic inside strikethrough, etc.).
|
||||
|
||||
In high-traffic environments, parsed messages are stored and transmitted frequently. Reducing the AST footprint has a direct impact on storage costs, cache efficiency, and wire transfer size.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Introduce a **compact AST format** that replaces self-contained string values with **span references** (`[start, end]`) into the original message text.
|
||||
|
||||
### Core Idea
|
||||
|
||||
Instead of:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "PARAGRAPH",
|
||||
"value": [
|
||||
{ "type": "BOLD", "value": [{ "type": "PLAIN_TEXT", "value": "hello" }] },
|
||||
{ "type": "PLAIN_TEXT", "value": " world" }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
The compact format stores:
|
||||
|
||||
```json
|
||||
{ "t": "p", "c": [{ "t": "b", "c": [[2, 7]] }, [8, 14]] }
|
||||
```
|
||||
|
||||
Plain text nodes become simple `[start, end]` tuples. Structural nodes use short type keys (`b`, `i`, `s`, `p`, `h`, etc.) and reference children via the same span mechanism.
|
||||
|
||||
### Key Operations
|
||||
|
||||
| Function | Description |
|
||||
|---|---|
|
||||
| `compactify(ast, msg)` | Converts a verbose AST + original message into a compact AST |
|
||||
| `expand(compactAst, msg)` | Reconstructs the full verbose AST from a compact AST + original message |
|
||||
| `validateRoundtrip(ast, msg)` | Verifies `expand(compactify(ast, msg), msg)` equals the original AST |
|
||||
|
||||
### Compact Type Mapping
|
||||
|
||||
| Verbose Type | Compact Key | Notes |
|
||||
|---|---|---|
|
||||
| `PLAIN_TEXT` | `[start, end]` | Span tuple, no wrapper object |
|
||||
| `BOLD` | `b` | |
|
||||
| `ITALIC` | `i` | |
|
||||
| `STRIKE` | `s` | |
|
||||
| `SPOILER` | `\|\|` | |
|
||||
| `INLINE_CODE` | `` ` `` | |
|
||||
| `MENTION_USER` | `@` | |
|
||||
| `MENTION_CHANNEL` | `#` | |
|
||||
| `INLINE_KATEX` | `$` | |
|
||||
| `LINK` | `a` | |
|
||||
| `IMAGE` | `img` | |
|
||||
| `EMOJI` | `:` | |
|
||||
| `TIMESTAMP` | `ts` | |
|
||||
| `COLOR` | `c` | Stores RGBA as `[r, g, b, a]` |
|
||||
| `PARAGRAPH` | `p` | |
|
||||
| `HEADING` | `h` | Includes level `l: 1..4` |
|
||||
| `CODE` | ```` ``` ```` | |
|
||||
| `BLOCKQUOTE` | `>` | |
|
||||
| `QUOTE` | `q` | |
|
||||
| `SPOILER_BLOCK` | `\|\|\|` | |
|
||||
| `ORDERED_LIST` | `ol` | |
|
||||
| `UNORDERED_LIST` | `ul` | |
|
||||
| `TASKS` | `tl` | |
|
||||
| `KATEX` | `$$` | |
|
||||
| `LINE_BREAK` | `br` | |
|
||||
| `BIG_EMOJI` | `E` | |
|
||||
|
||||
## Trade-offs
|
||||
|
||||
### Advantages
|
||||
|
||||
- Significant size reduction on typical messages (observed 30-60% in initial tests)
|
||||
- Lossless — roundtrip conversion preserves the full AST
|
||||
- The original message is already stored alongside the AST, so no additional data is needed
|
||||
- Short keys further reduce payload size
|
||||
|
||||
### Concerns
|
||||
|
||||
- Requires the original message text to be available at expansion time
|
||||
- Adds a conversion layer — any bug in `compactify`/`expand` could corrupt message rendering
|
||||
- Span-based references are fragile if the message text is modified after compaction
|
||||
- Increases complexity in the message-parser package
|
||||
|
||||
## Open Questions
|
||||
|
||||
1. **Storage strategy** — Should compact ASTs replace verbose ones in the database, or coexist (e.g., compact for wire transfer, verbose for rendering)?
|
||||
2. **Migration path** — How do we handle existing messages already stored with verbose ASTs?
|
||||
3. **Rendering integration** — Should gazzodown learn to render compact ASTs directly, or always expand first?
|
||||
4. **Message edits** — When a message is edited, do we reparse and recompact, or invalidate the compact form?
|
||||
5. **Performance budget** — Is the compactify/expand overhead acceptable on the hot path, or should it be deferred to a background job?
|
||||
|
||||
## Reference
|
||||
|
||||
A working proof-of-concept implementation exists with full bidirectional conversion and roundtrip validation tests. It covers all current AST node types including BigEmoji, lists, tasks, code blocks, KaTeX, and color nodes.
|
||||
114
docs/proposals/offline-license-capability-enforcement.md
Normal file
114
docs/proposals/offline-license-capability-enforcement.md
Normal file
@@ -0,0 +1,114 @@
|
||||
# Proposal: License Capability Enforcement via Branded Proofs
|
||||
|
||||
## Status
|
||||
|
||||
Draft — v2. Supersedes the earlier `CloudConnection`-only draft: the single-purpose connection capability is generalized into one proof mechanism that can guard any license-controlled capability, with the specific entitlement captured in the type.
|
||||
|
||||
## Problem
|
||||
|
||||
License enforcement today is a runtime side-condition the compiler knows nothing about, in two distinct places:
|
||||
|
||||
1. **Module entitlements.** `License.hasModule('auditing')` returns a `boolean`. Nothing ties that boolean to the code it guards: a feature entry point can be called without any check, a check for the *wrong* module still typechecks, and a refactor that moves code out from under its `if` block breaks enforcement silently.
|
||||
2. **Offline (air-gapped) licenses.** When `license.information.offline` is true, the workspace must never initiate outbound connections to Rocket.Chat-owned endpoints. This is enforced by ~15 scattered `hasOfflineLicense()` checks (sync, marketplace, telemetry, push gateway, Gravatar). Any new feature can `import { serverFetch }` and ship a compliance violation no type error catches — and two real bugs of exactly this class were found during QA (a startup race in the usage report, and a stale verdict held by the push retry chain).
|
||||
|
||||
Both are the same underlying flaw: **the license check produces no evidence**. Nothing forces the check to happen, to happen for the right entitlement, or to happen at the right time.
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Make every license check return an unforgeable **proof value**, and make guarded code demand that proof in its signature. The pattern has three parts.
|
||||
|
||||
### 1. The lock — branded proof types
|
||||
|
||||
One brand mechanism, parameterized by what it proves. The generic parameter captures the exact literal passed to the checker, so proofs for different modules are not interchangeable:
|
||||
|
||||
```ts
|
||||
// packages/core-typings/src/license/LicenseModule.ts (already exists)
|
||||
export type InternalModuleName = (typeof CoreModules)[number];
|
||||
export type ExternalModuleName = `${string}.${string}`;
|
||||
export type LicenseModule = InternalModuleName | ExternalModuleName;
|
||||
|
||||
// ee/packages/license/src/proofs.ts (new)
|
||||
declare const LicenseAuthorized: unique symbol; // NOT exported — unforgeable outside the package
|
||||
|
||||
/** Proof that the current license grants module M. */
|
||||
export type ModuleProof<M extends LicenseModule> = {
|
||||
readonly [LicenseAuthorized]: M;
|
||||
};
|
||||
|
||||
/** Proof that the current license permits outbound calls to Rocket.Chat-owned
|
||||
* endpoints (absent when the license carries the offline flag). */
|
||||
export type CloudEgressProof = {
|
||||
readonly [LicenseAuthorized]: 'cloud:egress';
|
||||
};
|
||||
```
|
||||
|
||||
The brand property is phantom — no such field exists at runtime. Proofs are frozen empty objects: zero allocation cost (a shared singleton per kind), zero serialization surface.
|
||||
|
||||
### 2. The keymaker — the license package owns construction
|
||||
|
||||
The only casts live inside `ee/packages/license`, next to the state they attest to:
|
||||
|
||||
```ts
|
||||
// on LicenseManager — additive API; the existing boolean hasModule() stays
|
||||
public proveModule<M extends LicenseModule>(module: M): ModuleProof<M> | undefined {
|
||||
return this.hasModule(module) ? (PROOF as ModuleProof<M>) : undefined;
|
||||
}
|
||||
|
||||
public proveCloudEgress(): CloudEgressProof | undefined {
|
||||
return this.hasOfflineLicense() ? undefined : (PROOF as CloudEgressProof);
|
||||
}
|
||||
```
|
||||
|
||||
Design decisions:
|
||||
|
||||
- **Additive, not a signature change.** `hasModule(): boolean` has hundreds of call sites, including display logic and the REST surface the client consumes. Those keep the boolean. `proveModule()` is what *server-side entry points* migrate to.
|
||||
- **Synchronous.** License state is in memory; proofs are free to acquire, which matters for the per-attempt rule below.
|
||||
- **The offline gate is a proof family, not a module.** The offline flag has inverted semantics — it *revokes* a permission rather than granting a feature — but the same brand expresses it: `proveCloudEgress()` returns `undefined` exactly when the offline license forbids egress. A `proveCloudEgressOrThrow(message?)` variant throws the existing `CloudOfflineLicenseError` for interactive flows (registration, cloud login, billing), preserving today's error contract.
|
||||
|
||||
### 3. The guard — signatures demand proofs
|
||||
|
||||
```ts
|
||||
// Module-gated feature entry point:
|
||||
function initAuditing(proof: ModuleProof<'auditing'>) { ... }
|
||||
|
||||
initAuditing(); // ❌ compile error: expected 1 argument
|
||||
const token = License.proveModule('auditing');
|
||||
initAuditing(token); // ❌ compile error: possibly undefined
|
||||
if (token) {
|
||||
initAuditing(token); // ✅ narrowed to ModuleProof<'auditing'>
|
||||
}
|
||||
|
||||
// Cloud I/O — the single fetch wrapper in apps/meteor/server/lib/cloud/cloudClient.ts:
|
||||
export function cloudFetch(proof: CloudEgressProof, input: string, options?: ExtendedFetchOptions): Promise<Response>;
|
||||
```
|
||||
|
||||
The developer experience does the enforcement: to call the function you need the token; the only way to get the token is the checker whose name and JSDoc explain the rule; the `| undefined` return forces an explicit decision about the denied case (skip silently vs. throw); and truthiness narrowing makes the happy path read naturally.
|
||||
|
||||
The proof is compile-time evidence, not the runtime gate itself: `cloudFetch` **re-validates `hasOfflineLicense()` immediately before dispatching** and drops the request if the license changed while the operation was in flight. The proof parameter guarantees the check can't be forgotten; the dispatch-time re-check guarantees it can't go stale.
|
||||
|
||||
Inner helpers that are only reachable from a guarded entry point take the proof as a parameter rather than re-checking — the signature documents "this function performs cloud I/O / module-X work", and the compiler walks the requirement up the call graph to wherever the proof is legitimately acquired.
|
||||
|
||||
## The staleness rule: acquire per attempt, never cache
|
||||
|
||||
Licenses change at runtime — swapped, upgraded, invalidated, or an offline license applied mid-flight. A proof is a **point-in-time verdict**, and both QA-discovered bugs in the offline work were staleness bugs:
|
||||
|
||||
- the usage report evaluated its gate at function entry, then spent seconds generating statistics before fetching (the verdict predated license application);
|
||||
- the push gateway retry chain captured its decision in `setTimeout` closures that outlived a license change.
|
||||
|
||||
Rules, to be stated in the proofs' JSDoc and enforced in review:
|
||||
|
||||
1. Acquire the proof at the top of each operation *attempt*; retry closures re-acquire on every attempt (the factories are sync — this costs nothing).
|
||||
2. Never store a proof on a class field, module scope, or queue payload.
|
||||
3. A proof is not a subscription. Long-lived module features (services started when a module is granted) must still use the existing lifecycle events — `License.onValidFeature` / `onInvalidFeature` / `onToggledFeature` — for startup and teardown. Proofs guard entry points; events manage lifetimes.
|
||||
4. Proofs are server-only and must never cross the API boundary; the client keeps boolean checks driven by `licenses.info`.
|
||||
|
||||
## Migration
|
||||
|
||||
1. **Cloud egress first** (the offline license feature, already enforced at runtime in ~27 files): introduce `proveCloudEgress()` and `cloudFetch(proof, ...)`, then convert the four established patterns — interactive flows (`OrThrow`), background jobs (`undefined` → keep side effects, skip), the marketplace client (proof acquired per `fetch()` call, never stored on the instance), and retry closures (per-attempt). The existing `hasOfflineLicense()` helpers remain for behavior gates that don't perform I/O themselves (`shouldUseGateway()`, cron-body skips, Gravatar suggestion filtering).
|
||||
2. **Module proofs opportunistically**: as EE features are touched, their server entry points gain `ModuleProof<'...'>` parameters, starting with features whose checks have historically drifted from their code. No big-bang rewrite; the boolean API keeps working throughout.
|
||||
|
||||
## Limitations
|
||||
|
||||
TypeScript cannot forbid an import: a new file can still `import { serverFetch }` directly and hardcode an endpoint, and `{} as ModuleProof<'auditing'>` defeats the brand (both are greppable and glaring in review — the cast requires importing a type whose only documented constructor is the license manager). What the type system guarantees is narrower but real: **code routed through proof-demanding signatures cannot skip the check, cannot check the wrong module, and must handle the denied case explicitly** — and violations shrink to two obvious review signals: a raw `serverFetch` near a Rocket.Chat domain, or a forged cast.
|
||||
|
||||
A directory-scoped lint rule (`no-restricted-imports` on `@rocket.chat/server-fetch` with `cloudClient.ts` exempt) or a CI grep could close the import hole; both were considered and deliberately left out in favor of a types-only approach.
|
||||
Reference in New Issue
Block a user