sdp-request-watcher/docs/NOTES.md

# Notes

## SDP v3 API gotchas

Docs: <https://www.manageengine.com/products/service-desk/sdpod-v3-api/getting-started/search-criteria.html>

- **Condition names have spaces:** `greater than`, `is not`, `lesser than`. `greater_than` isn't valid, and SDP silently ignored the **entire** search criteria because of it, returning every ticket (124,050). If a query's `total_count` is huge, suspect the criteria first.
- **Paging:** `row_count` is at most 100. Use `start_index`, and stop when `list_info.has_more_rows` is false. Ask for `get_total_count: true` on the first page. `getAllRequests` dedupes by ID and stops at 20 pages or when the total is unreachable.
- **Multiple values:** use a `values: [...]` array, and join criteria with `logical_operator: 'AND'` (see `allOf()` in `tickets.js`).
- **"Technician is empty":** written as `{ field: 'technician', condition: 'is' }` with no value. This isn't documented, since the docs only show the "is not" form, but it's verified to work.
- **Field names that work in filters:** `site.name`, `group.name`, `technician.name`, `status.name`, `last_updated_time`.
- **Group names:** the ManageEngine UI shows groups as "Customer Support « KVCC", but the API name is plain "Customer Support".
- **Sign-in scope:** `SDPOnDemand.requests.READ` is enough for every SDP query so far. `AaaServer.profile.READ` is added only for the signed-in user's name (`GET accounts.zoho.com/oauth/user/info`). If that call fails, sign-in still works. Changing the scopes means everyone signs in again.

## Zoho OAuth gotchas

- **Client type:** web sign-in needs a **Server-based Application** client. The old Non-Browser (device-code) client can't do redirects. The redirect URI must match `PUBLIC_URL/auth/callback` exactly.
- **Refresh tokens:** you only get one with `access_type=offline`, and `prompt=consent` makes Zoho issue one on every sign-in. Access tokens last about an hour; renew with `POST accounts.zoho.com/oauth/v2/token`, `grant_type=refresh_token`.
- **20 refresh tokens per user per client:** Zoho silently deletes the oldest when a 21st is issued. Each browser sign-in creates one, which is why Sign out revokes (`POST /oauth/v2/token/revoke?token=...`). A user signed in on more than 20 browsers would find the oldest one signed out.
- **Errors come back as HTTP 200:** a bad code or refresh token returns `{ "error": "invalid_code" }` with status 200. `auth.js` checks for `access_token` instead of relying on axios throwing.
- **Data centre:** everything points at `accounts.zoho.com` (US). The callback also gets `location` and `accounts-server` parameters, which would matter only for users in another Zoho DC.

## Deployment gotchas

- **NPM runs in Docker**, so `localhost` in a proxy host is NPM's own container. For a bare-metal test run, forward to the gateway of NPM's Docker network and allow that subnet through ufw. The production container is reached by name (`sdp-dashboard:3000`) because it joins NPM's network.
- **Port 3000 is Gitea's on the VPS.** Bare-metal test runs use `PORT=3005`. Never put `PORT` in the production `.env`; the container listens on 3000, and its health check and the NPM host expect that.
- **Cloudflare proxy and the NPM access list:** a proxied record can make NPM see Cloudflare's IPs, so the allow list blocks everyone. The fix is DNS only, or `real_ip_header CF-Connecting-IP;` in the proxy host's Advanced config. The live test on 2026-10-15 worked with the access list (an MCCS-listed IP got in, mobile data got a 403).

## Decisions

- **2026-10-15:** **MIT license.** Copyright Skylar Grant.
- **2026-10-15:** **OpenWeatherMap** for weather. The free tier covers current weather and the 5-day forecast, and results are cached for 10 minutes. The National Weather Service API was the alternative (free, no key), but it needs three calls per lookup.
- **2026-10-15:** **Unassigned counter** uses its own status exclusions (`UNASSIGNED_EXCLUDED_STATUSES`) to match the ManageEngine view. The debug list is kept but hidden (`SHOW_UNASSIGNED_LIST` in `index.html`).
- **2026-10-15:** **Per-user Zoho sign-in replaces the shared service token.** This reverses the earlier "one service token plus region config" decision, and drops the `ADMIN_KEY` and `data/token.json` from phase 1. Leadership's concern was that anyone who reached the page could see data while a shared token was live. Now every request needs the viewer's own sign-in, and NPM limits the site to MCCS addresses. The cost is that each open dashboard polls SDP itself; acceptable for a small team.
- **2026-10-15:** **Sessions in an encrypted cookie, not a server store.** AES-256-GCM with a key from `SESSION_SECRET`, `HttpOnly`, `SameSite=Lax` (Lax so the cookie survives the redirect back from Zoho), and `Secure` when `PUBLIC_URL` is https. The container needs no volume or database, and restarts don't sign anyone out. Renewing the access token rewrites the cookie but keeps `startedAt`, so `SESSION_DAYS` is a hard limit. Concurrent calls with an expired token share one renewal (`renewals` map in `session.js`).
- **2026-10-15:** **The browser runs the checks** and keeps the closed feed (first load asks for 12 hours, then 10-minute windows every minute). The server has no per-user timers or state. This also suits per-browser settings later: the browser will send its settings with each call.
- **2026-10-15:** **Per-user settings are stored on the server, keyed by Zoho email** (lowercased), in `DATA_DIR/settings.json`. Chosen over the planned `localStorage` so settings follow people between browsers and no export/import is needed. This gives the container state, so compose mounts the `sdp-data` volume at `/app/data`. Only values that differ from `.env` are stored, so `.env` changes still reach people who never changed that field. A plain JSON file, rewritten via temp file and rename, is enough for a small team with one process; move to SQLite if that stops being true.
- **2026-10-15:** **The server reads settings itself** rather than the browser sending them with each call, as first planned. The browser can only change them through `PUT /api/settings`, which validates everything (`LIMITS` in `settings.js`). That's for protecting SDP and the server, not access control: queries use the user's own token, and everyone at MCCS can see every site in SDP anyway.
- **2026-10-15:** **Name suggestions without a new scope.** `/api/names` runs one unscoped query (last 30 days, up to 500 tickets, names only) with the caller's token, at most hourly for everyone, and merges into `DATA_DIR/names.json` so the list grows over time. Typed names are still accepted; a name matching a suggestion except for case is saved with SDP's spelling, because the technician filters compare exactly. Proper pick lists from SDP's setup/users APIs would need another scope and a re-sign-in for everyone.
- **2026-10-15:** **Weather locations are a fixed list** (`src/places.js`), not free zip codes, so a browser can't make the server look up arbitrary places and the per-zip cache stays small. The `.env` zip is added to the list if it's missing.
- **2026-10-15:** **Status rules stay in `.env`** (`STATUS_NAMES`, `UNASSIGNED_EXCLUDED_STATUSES`). They're the help desk's definition of open and closed, not a personal preference. Hiding a status column is per user, but hidden columns still count toward Total open.
- **2026-10-19:** **The closed feed's statuses are per user (`feedStatuses`, v2.1.1).** This narrows the decision above: `STATUS_NAMES` still defines open and closed for the status table, Closed today and the unassigned count, and is the default for `feedStatuses`. Our SDP closes a resolved ticket on its own after 2 or 3 days if it isn't reopened, so watching only Closed gave the ding days late. When Resolved is in a user's list, the feed times a ticket by `resolved_time` before `completed_time`, so the automatic close (new `completed_time`, old `resolved_time`) falls outside the check window and doesn't ding again. A reopen and re-resolve gives a new `resolved_time`, which counts as a fresh close. An empty list is refused, because the feed would then take any status. Before this, an empty `STATUS_NAMES` made the feed take any status; now it falls back to `Closed,Resolved,Canceled` like the rest.
- **2026-10-19:** **The technician table sorts by clicking its headers, remembered per browser** (`localStorage` key `sdp-table-sort`, like the sound). This replaced a short-lived `tableSort` setting in the Settings dialog (never released): headers are easier to find and switch, and a wallboard and a desk on the same account can sort differently. The default is name A to Z; a new column starts A to Z for names and highest first for counts, and a second click reverses it. Ties always go A to Z. The arrow points toward the larger end, so ↓ means A to Z or lowest first and ↑ means highest first. Skylar found the usual ↑-for-ascending convention backwards, so don't "fix" it. A saved status sort whose column has no tickets right now just sorts everyone as 0 (A to Z).
- **2026-10-19:** **Contrast pass before any theme picker (v2.2.0).** Feedback asked for higher contrast or theme options. We fixed the one dark theme first and will add a small per-browser picker (Dark, Light, High contrast) only if people still ask; a gallery of themes was ruled out because each one has to be rechecked against SDP's status colors, the heat scale and every new panel. Every color is now a token in `:root` in `index.html` (no inline or one-off colors), with text checked against `--card` and `--card-raised` for WCAG AA: `--faint` went from 2.9:1 to 6:1, `--muted` from 7:1 to 9:1. The table heat scale uses its own `--heat-max` (2.9:1 against the card, ink on it 5.2:1) with a floor of 25%, so a cell with one ticket is visible.
- **2026-10-19:** **MCCS colors (v2.2.0).** There's no MCCS brand guide that we know of, so navy `#1a3d6b` and green `#4f763c` were sampled from the ITS logo's pixels (`--mccs-navy`, `--mccs-green`). Both are too dark for text on the dark theme, so the page uses lifted versions of the same hues: the accent (`--signal`, was orange) is MCCS green at 58% lightness, the heat scale is MCCS navy at 44%, and the backgrounds are tinted toward the navy hue. The header stripe is the raw green. Errors keep the red-orange `--alert` so they never look like the green accent. SDP's own status colors (the column swatches) are untouched. Nothing is smaller than 12px except the feed's "ago" label (11px, uppercase). Recheck the ratios when changing a token.
- **2026-10-15:** **Sound stays per browser** (`localStorage`), not in the saved settings, so a muted desk browser doesn't mute the wallboard signed in as the same person.
- **2026-10-15:** **Sign-in is limited by email domain (`ALLOWED_EMAIL_DOMAINS`).** Any Zoho account can approve our client, and before this, an outside account got a session and `/api/session` handed it the `.env` site, group and technician names (ticket queries failed with their token). Prod v2.0.0 had this gap until v2.1.0 was deployed on 2026-10-16. The callback now refuses other domains and revokes their token, `requireSession` re-checks every call so older cookies die too, and the server won't start without the setting. Students don't have Zoho accounts, so the domain is enough for now. Next step: check the user is an SDP technician at sign-in, which probably needs a users/technicians read scope.
- **2026-10-16:** **No ticket subjects, anywhere (FERPA).** Subjects can name students or describe their situation. The closed feed and the unassigned query no longer put `subject` in `fields_required`, and the objects the server returns are built field by field, so a subject SDP sends anyway (if it ever ignores `fields_required`) is dropped before it leaves the server. Feed cards show the ticket number, technician and closing time. Don't add `subject`, `description` or any other free-text field to a query or response without a new review; to identify a ticket, use its number and open it in SDP.
- **2026-10-15:** **`/api/names` makes the caller prove SDP access first** (a one-row requests query with their token) before returning the shared cache, so the names never go to a token SDP refuses.
- **2026-10-15:** **No published ports in compose.** Docker's port publishing bypasses ufw, which would skip NPM's IP access list. The container joins NPM's network instead.
- **2026-10-15:** **CI uses plain `docker` commands** on the self-hosted runner, not the buildx or login actions, so there are fewer moving parts on the runner. Pushes both the tag and `latest`.

## Known issues

- The status table scrolls sideways once there are more statuses than fit, because the page is capped at 1,320px. Consider widening the page or narrowing the sidebar.
- All statuses not in `STATUS_NAMES` count as open, including "Need Information" and "On Hold". This is intentional; people can hide those columns in Settings.
- If Zoho doesn't return an email at sign-in (the profile call failed), that session can't save settings and uses the defaults; signing in again fixes it.
- No screenshots: `images/` was removed on 2026-10-15 and new ones are due.