sdp-request-watcher/docs/NOTES.md

# Notes

## SDP v3 API gotchas

Docs: <https://www.manageengine.com/products/service-desk/sdpod-v3-api/getting-started/search-criteria.html>

- **Condition names have spaces:** `greater than`, `is not`, `lesser than`. `greater_than` isn't valid, and SDP silently ignored the **entire** search criteria because of it, returning every ticket (124,050). If a query's `total_count` is huge, suspect the criteria first.
- **Paging:** `row_count` is at most 100. Use `start_index`, and stop when `list_info.has_more_rows` is false. Ask for `get_total_count: true` on the first page. `getAllRequests` dedupes by ID and stops at 20 pages or when the total is unreachable.
- **Multiple values:** use a `values: [...]` array, and join criteria with `logical_operator: 'AND'` (see `allOf()` in `tickets.js`).
- **"Technician is empty":** written as `{ field: 'technician', condition: 'is' }` with no value. This isn't documented, since the docs only show the "is not" form, but it's verified to work.
- **Field names that work in filters:** `site.name`, `group.name`, `technician.name`, `status.name`, `last_updated_time`.
- **Group names:** the ManageEngine UI shows groups as "Customer Support « KVCC", but the API name is plain "Customer Support".
- **Sign-in scope:** `SDPOnDemand.requests.READ` is enough for every SDP query so far. `AaaServer.profile.READ` is added only for the signed-in user's name (`GET accounts.zoho.com/oauth/user/info`). If that call fails, sign-in still works. Changing the scopes means everyone signs in again.

## Zoho OAuth gotchas

- **Client type:** web sign-in needs a **Server-based Application** client. The old Non-Browser (device-code) client can't do redirects. The redirect URI must match `PUBLIC_URL/auth/callback` exactly.
- **Refresh tokens:** you only get one with `access_type=offline`, and `prompt=consent` makes Zoho issue one on every sign-in. Access tokens last about an hour; renew with `POST accounts.zoho.com/oauth/v2/token`, `grant_type=refresh_token`.
- **20 refresh tokens per user per client:** Zoho silently deletes the oldest when a 21st is issued. Each browser sign-in creates one, which is why Sign out revokes (`POST /oauth/v2/token/revoke?token=...`). A user signed in on more than 20 browsers would find the oldest one signed out.
- **Errors come back as HTTP 200:** a bad code or refresh token returns `{ "error": "invalid_code" }` with status 200. `auth.js` checks for `access_token` instead of relying on axios throwing.
- **Data centre:** everything points at `accounts.zoho.com` (US). The callback also gets `location` and `accounts-server` parameters, which would matter only for users in another Zoho DC.

## Deployment gotchas

- **NPM runs in Docker**, so `localhost` in a proxy host is NPM's own container. For a bare-metal test run, forward to the gateway of NPM's Docker network and allow that subnet through ufw. The production container is reached by name (`sdp-dashboard:3000`) because it joins NPM's network.
- **Port 3000 is Gitea's on the VPS.** Bare-metal test runs use `PORT=3005`. Never put `PORT` in the production `.env`; the container listens on 3000, and its health check and the NPM host expect that.
- **Cloudflare proxy and the NPM access list:** a proxied record can make NPM see Cloudflare's IPs, so the allow list blocks everyone. The fix is DNS only, or `real_ip_header CF-Connecting-IP;` in the proxy host's Advanced config. The live test on 2026-10-15 worked with the access list (an MCCS-listed IP got in, mobile data got a 403).

## Decisions

- **2026-10-15:** **MIT license.** Copyright Skylar Grant.
- **2026-10-15:** **OpenWeatherMap** for weather. The free tier covers current weather and the 5-day forecast, and results are cached for 10 minutes. The National Weather Service API was the alternative (free, no key), but it needs three calls per lookup.
- **2026-10-15:** **Unassigned counter** uses its own status exclusions (`UNASSIGNED_EXCLUDED_STATUSES`) to match the ManageEngine view. The debug list is kept but hidden (`SHOW_UNASSIGNED_LIST` in `index.html`).
- **2026-10-15:** **Per-user Zoho sign-in replaces the shared service token.** This reverses the earlier "one service token plus region config" decision, and drops the `ADMIN_KEY` and `data/token.json` from phase 1. Leadership's concern was that anyone who reached the page could see data while a shared token was live. Now every request needs the viewer's own sign-in, and NPM limits the site to MCCS addresses. The cost is that each open dashboard polls SDP itself; acceptable for a small team.
- **2026-10-15:** **Sessions in an encrypted cookie, not a server store.** AES-256-GCM with a key from `SESSION_SECRET`, `HttpOnly`, `SameSite=Lax` (Lax so the cookie survives the redirect back from Zoho), and `Secure` when `PUBLIC_URL` is https. The container needs no volume or database, and restarts don't sign anyone out. Renewing the access token rewrites the cookie but keeps `startedAt`, so `SESSION_DAYS` is a hard limit. Concurrent calls with an expired token share one renewal (`renewals` map in `session.js`).
- **2026-10-15:** **The browser runs the checks** and keeps the closed feed (first load asks for 12 hours, then 10-minute windows every minute). The server has no per-user timers or state. This also suits per-browser settings later: the browser will send its settings with each call.
- **2026-10-15:** **Per-user settings are stored on the server, keyed by Zoho email** (lowercased), in `DATA_DIR/settings.json`. Chosen over the planned `localStorage` so settings follow people between browsers and no export/import is needed. This gives the container state, so compose mounts the `sdp-data` volume at `/app/data`. Only values that differ from `.env` are stored, so `.env` changes still reach people who never changed that field. A plain JSON file, rewritten via temp file and rename, is enough for a small team with one process; move to SQLite if that stops being true.
- **2026-10-15:** **The server reads settings itself** rather than the browser sending them with each call, as first planned. The browser can only change them through `PUT /api/settings`, which validates everything (`LIMITS` in `settings.js`). That's for protecting SDP and the server, not access control: queries use the user's own token, and everyone at MCCS can see every site in SDP anyway.
- **2026-10-15:** **Name suggestions without a new scope.** `/api/names` runs one unscoped query (last 30 days, up to 500 tickets, names only) with the caller's token, at most hourly for everyone, and merges into `DATA_DIR/names.json` so the list grows over time. Typed names are still accepted; a name matching a suggestion except for case is saved with SDP's spelling, because the technician filters compare exactly. Proper pick lists from SDP's setup/users APIs would need another scope and a re-sign-in for everyone.
- **2026-10-15:** **Weather locations are a fixed list** (`src/places.js`), not free zip codes, so a browser can't make the server look up arbitrary places and the per-zip cache stays small. The `.env` zip is added to the list if it's missing.
- **2026-10-15:** **Status rules stay in `.env`** (`STATUS_NAMES`, `UNASSIGNED_EXCLUDED_STATUSES`). They're the help desk's definition of open and closed, not a personal preference. Hiding a status column is per user, but hidden columns still count toward Total open.
- **2026-10-19:** **The closed feed's statuses are per user (`feedStatuses`, v2.1.1).** This narrows the decision above: `STATUS_NAMES` still defines open and closed for the status table, Closed today and the unassigned count, and is the default for `feedStatuses`. Our SDP closes a resolved ticket on its own after 2 or 3 days if it isn't reopened, so watching only Closed gave the ding days late. When Resolved is in a user's list, the feed times a ticket by `resolved_time` before `completed_time`, so the automatic close (new `completed_time`, old `resolved_time`) falls outside the check window and doesn't ding again. A reopen and re-resolve gives a new `resolved_time`, which counts as a fresh close. An empty list is refused, because the feed would then take any status. Before this, an empty `STATUS_NAMES` made the feed take any status; now it falls back to `Closed,Resolved,Canceled` like the rest.
- **2026-10-19:** **The technician table sorts by clicking its headers, remembered per browser** (`localStorage` key `sdp-table-sort`, like the sound). This replaced a short-lived `tableSort` setting in the Settings dialog (never released): headers are easier to find and switch, and a wallboard and a desk on the same account can sort differently. The default is name A to Z; a new column starts A to Z for names and highest first for counts, and a second click reverses it. Ties always go A to Z. The arrow points toward the larger end, so ↓ means A to Z or lowest first and ↑ means highest first. Skylar found the usual ↑-for-ascending convention backwards, so don't "fix" it. A saved status sort whose column has no tickets right now just sorts everyone as 0 (A to Z).
- **2026-10-15:** **Sound stays per browser** (`localStorage`), not in the saved settings, so a muted desk browser doesn't mute the wallboard signed in as the same person.
- **2026-10-15:** **Sign-in is limited by email domain (`ALLOWED_EMAIL_DOMAINS`).** Any Zoho account can approve our client, and before this, an outside account got a session and `/api/session` handed it the `.env` site, group and technician names (ticket queries failed with their token). Prod v2.0.0 had this gap until v2.1.0 was deployed on 2026-10-16. The callback now refuses other domains and revokes their token, `requireSession` re-checks every call so older cookies die too, and the server won't start without the setting. Students don't have Zoho accounts, so the domain is enough for now. Next step: check the user is an SDP technician at sign-in, which probably needs a users/technicians read scope.
- **2026-10-16:** **No ticket subjects, anywhere (FERPA).** Subjects can name students or describe their situation. The closed feed and the unassigned query no longer put `subject` in `fields_required`, and the objects the server returns are built field by field, so a subject SDP sends anyway (if it ever ignores `fields_required`) is dropped before it leaves the server. Feed cards show the ticket number, technician and closing time. Don't add `subject`, `description` or any other free-text field to a query or response without a new review; to identify a ticket, use its number and open it in SDP.
- **2026-10-15:** **`/api/names` makes the caller prove SDP access first** (a one-row requests query with their token) before returning the shared cache, so the names never go to a token SDP refuses.
- **2026-10-15:** **No published ports in compose.** Docker's port publishing bypasses ufw, which would skip NPM's IP access list. The container joins NPM's network instead.
- **2026-10-15:** **CI uses plain `docker` commands** on the self-hosted runner, not the buildx or login actions, so there are fewer moving parts on the runner. Pushes both the tag and `latest`.

## Known issues

- The status table scrolls sideways once there are more statuses than fit, because the page is capped at 1,320px. Consider widening the page or narrowing the sidebar.
- All statuses not in `STATUS_NAMES` count as open, including "Need Information" and "On Hold". This is intentional; people can hide those columns in Settings.
- If Zoho doesn't return an email at sign-in (the profile call failed), that session can't save settings and uses the defaults; signing in again fixes it.
- No screenshots: `images/` was removed on 2026-10-15 and new ones are due.