# Notification, Reminder, and Broadcast Operations

## Runtime model

Phase 16 separates database state changes from Telegram network calls. A web transaction writes a uniquely keyed `outbox_messages` row. The worker claims eligible rows in small leased batches with `FOR UPDATE SKIP LOCKED`, commits the lease, and only then calls Telegram. The scheduler materializes reminder and broadcast recipients into the same outbox contract; it never sends a message itself.

Run the three processes independently after migration and environment verification:

```bash
npm run start:web
npm run start:worker
npm run start:scheduler
```

Stopping a worker or scheduler begins drain, interrupts its polling wait, finishes the active iteration, and closes MySQL. Multiple schedulers are safe because a database lease guards the notification scheduling cycle. Multiple workers claim disjoint outbox rows.

## Safety and delivery semantics

- Every message has a unique business-event deduplication key.
- Claim leases recover work abandoned by a crashed worker.
- Retry delay is exponential, capped at one hour, and respects Telegram `retry_after`.
- HTTP 429 and provider 5xx responses retry; permanent provider rejection fails without a retry loop.
- Delivery attempts store only outcome, attempt number, provider status, retry delay, error code, and message ID. Raw provider responses and message bodies are not copied into logs or attempt evidence.
- Reminder and broadcast deliveries store the rendered-payload hash and exact template version.
- Telegram does not provide an idempotency key for `sendMessage`. Therefore the system is **at-least-once** around the narrow case where Telegram accepts a message but the database acknowledgement fails. Deduplication eliminates application replays but cannot promise provider-level exactly-once delivery.

## Template contract

Admin templates support only these balanced Telegram tags: `b`, `strong`, `i`, `em`, `u`, `ins`, `s`, `strike`, `del`, and `code`. Attributes, arbitrary links, scripts, and unmatched angle brackets are rejected. Every substituted value is HTML-escaped.

Allowed placeholders are closed to:

```text
{{first_name}} {{username}} {{balance}} {{currency}} {{task_count}}
{{active_referrals}} {{withdraw_amount}} {{withdraw_status}}
{{app_url}} {{support_url}}
```

Buttons are either `mini_app` or `url`, may use only `app_url` or `support_url`, and must resolve to HTTPS. Updating or publishing a template appends an immutable version row. Preview uses non-sensitive sample data and sends nothing.

The default templates remain `draft`. Publish each reviewed template before enabling the related event. An inactive or missing template causes transactional notification production to skip safely rather than roll back a financial operation.

## Reminder scheduling

Reminder input uses an IANA timezone, local `HH:mm`, and ISO weekdays (`1` Monday through `7` Sunday). The scheduler calculates the next real UTC minute by timezone, including daylight-saving transitions. It uses `next_run_at` as the deterministic occurrence boundary and creates at most one delivery per schedule/user/occurrence.

Audience rules are closed JSON fields:

- `user_statuses`: `active` and/or `limited`;
- optional language and country allowlists;
- optional `seen_within_days` from 1 to 365.

Deleted users and Telegram profiles that disallow private writes are excluded. The `reminders` feature flag is disabled by default.

## Broadcast approval and cancellation

A broadcast begins as a draft. An Admin with `broadcasts.create` creates it; a **different Admin** with `broadcasts.approve` must approve it. Separation is enforced in service logic and by the database check constraint. Approval requires an active, valid template.

When due, the scheduler freezes the audience timestamp and materializes recipients in bounded batches. Users created after that timestamp are excluded from that broadcast. Counts are rebuilt from delivery rows. Cancellation marks unsent delivery rows skipped and cancels only pending outbox rows; an in-flight Telegram request may already complete.

Feature flag `broadcasts` is disabled by default. Keep it disabled until templates, permissions, worker health, audience preview, and operational alerting are verified.

## API summary

| API                                   | Purpose                                                |
| ------------------------------------- | ------------------------------------------------------ |
| `GET /api/v1/notifications`           | Authenticated user inbox; optional unread filter       |
| `POST /api/v1/notifications/:id/read` | Ownership-scoped immutable read state                  |
| `POST /api/v1/notifications/read-all` | Marks current non-expired inbox items read             |
| `/api/v1/admin/bot-templates/*`       | View, preview, update, publish, and archive templates  |
| `/api/v1/admin/reminders/*`           | Manage reminder schedules and lifecycle                |
| `/api/v1/admin/broadcasts/*`          | Draft, inspect, approve, cancel, and report broadcasts |

Unsafe methods retain Admin/User authentication, CSRF, strict schemas, rate limits, Admin IP policy, exact permissions, optimistic versions, and audit context.

## Deployment checklist

1. Back up MySQL and apply migration `021_harden_notification_delivery.sql`.
2. Configure bounded queue/scheduler values from `.env.example`.
3. Publish reviewed templates using two-person operational review where appropriate.
4. Start one scheduler and one worker; confirm clean startup logs without payloads or tokens.
5. Test one internal recipient before enabling `reminders` or `broadcasts`.
6. Enable the desired feature flag only after delivery and failure reporting are monitored.
7. Scale workers gradually; keep the Telegram rate limit and database pool budget below provider/infrastructure capacity.

No live database, Telegram bot, or deployment environment was changed during Phase 16 implementation.