sdp-request-watcher/docs/NOTES.md

Notes

SDP v3 API gotchas

Docs: https://www.manageengine.com/products/service-desk/sdpod-v3-api/getting-started/search-criteria.html

• Condition names have spaces: greater than, is not, lesser than. greater_than isn't valid, and SDP silently ignored the entire search criteria because of it, returning every ticket (124,050). If a query's total_count is huge, suspect the criteria first.
• Paging: row_count is at most 100. Use start_index, and stop when list_info.has_more_rows is false. Ask for get_total_count: true on the first page. getAllRequests dedupes by ID and stops at 20 pages or when the total is unreachable.
• Multiple values: use a values: [...] array, and join criteria with logical_operator: 'AND' (see allOf() in tickets.js).
• "Technician is empty": written as { field: 'technician', condition: 'is' } with no value. This isn't documented, since the docs only show the "is not" form, but it's verified to work.
• Field names that work in filters: site.name, group.name, technician.name, status.name, last_updated_time.
• Group names: the ManageEngine UI shows groups as "Customer Support « KVCC", but the API name is plain "Customer Support".
• Sign-in scope: SDPOnDemand.requests.READ is enough for every SDP query so far. AaaServer.profile.READ is added only for the signed-in user's name (GET accounts.zoho.com/oauth/user/info). If that call fails, sign-in still works. Changing the scopes means everyone signs in again.

Zoho OAuth gotchas

• Client type: web sign-in needs a Server-based Application client. The old Non-Browser (device-code) client can't do redirects. The redirect URI must match PUBLIC_URL/auth/callback exactly.
• Refresh tokens: you only get one with access_type=offline, and prompt=consent makes Zoho issue one on every sign-in. Access tokens last about an hour; renew with POST accounts.zoho.com/oauth/v2/token, grant_type=refresh_token.
• 20 refresh tokens per user per client: Zoho silently deletes the oldest when a 21st is issued. Each browser sign-in creates one, which is why Sign out revokes (POST /oauth/v2/token/revoke?token=...). A user signed in on more than 20 browsers would find the oldest one signed out.
• Errors come back as HTTP 200: a bad code or refresh token returns { "error": "invalid_code" } with status 200. auth.js checks for access_token instead of relying on axios throwing.
• Data centre: everything points at accounts.zoho.com (US). The callback also gets location and accounts-server parameters, which would matter only for users in another Zoho DC.

Deployment gotchas

• NPM runs in Docker, so localhost in a proxy host is NPM's own container. For a bare-metal test run, forward to the gateway of NPM's Docker network and allow that subnet through ufw. The production container is reached by name (sdp-dashboard:3000) because it joins NPM's network.
• Port 3000 is Gitea's on the VPS. Bare-metal test runs use PORT=3005. Never put PORT in the production .env; the container listens on 3000, and its health check and the NPM host expect that.
• Cloudflare proxy and the NPM access list: a proxied record can make NPM see Cloudflare's IPs, so the allow list blocks everyone. The fix is DNS only, or real_ip_header CF-Connecting-IP; in the proxy host's Advanced config. The live test on 2026-10-15 worked with the access list (an MCCS-listed IP got in, mobile data got a 403).

Decisions

• 2026-10-15: MIT license. Copyright Skylar Grant.
• 2026-10-15: OpenWeatherMap for weather. The free tier covers current weather and the 5-day forecast, and results are cached for 10 minutes. The National Weather Service API was the alternative (free, no key), but it needs three calls per lookup.
• 2026-10-15: Unassigned counter uses its own status exclusions (UNASSIGNED_EXCLUDED_STATUSES) to match the ManageEngine view. The debug list is kept but hidden (SHOW_UNASSIGNED_LIST in index.html).
• 2026-10-15: Per-user Zoho sign-in replaces the shared service token. This reverses the earlier "one service token plus region config" decision, and drops the ADMIN_KEY and data/token.json from phase 1. Leadership's concern was that anyone who reached the page could see data while a shared token was live. Now every request needs the viewer's own sign-in, and NPM limits the site to MCCS addresses. The cost is that each open dashboard polls SDP itself; acceptable for a small team.
• 2026-10-15: Sessions in an encrypted cookie, not a server store. AES-256-GCM with a key from SESSION_SECRET, HttpOnly, SameSite=Lax (Lax so the cookie survives the redirect back from Zoho), and Secure when PUBLIC_URL is https. The container needs no volume or database, and restarts don't sign anyone out. Renewing the access token rewrites the cookie but keeps startedAt, so SESSION_DAYS is a hard limit. Concurrent calls with an expired token share one renewal (renewals map in session.js).
• 2026-10-15: The browser runs the checks and keeps the closed feed (first load asks for 12 hours, then 10-minute windows every minute). The server has no per-user timers or state. This also suits per-browser settings later: the browser will send its settings with each call.
• 2026-10-15: Per-user settings are stored on the server, keyed by Zoho email (lowercased), in DATA_DIR/settings.json. Chosen over the planned localStorage so settings follow people between browsers and no export/import is needed. This gives the container state, so compose mounts the sdp-data volume at /app/data. Only values that differ from .env are stored, so .env changes still reach people who never changed that field. A plain JSON file, rewritten via temp file and rename, is enough for a small team with one process; move to SQLite if that stops being true.
• 2026-10-15: The server reads settings itself rather than the browser sending them with each call, as first planned. The browser can only change them through PUT /api/settings, which validates everything (LIMITS in settings.js). That's for protecting SDP and the server, not access control: queries use the user's own token, and everyone at MCCS can see every site in SDP anyway.
• 2026-10-15: Name suggestions without a new scope. /api/names runs one unscoped query (last 30 days, up to 500 tickets, names only) with the caller's token, at most hourly for everyone, and merges into DATA_DIR/names.json so the list grows over time. Typed names are still accepted; a name matching a suggestion except for case is saved with SDP's spelling, because the technician filters compare exactly. Proper pick lists from SDP's setup/users APIs would need another scope and a re-sign-in for everyone.
• 2026-10-15: Weather locations are a fixed list (src/places.js), not free zip codes, so a browser can't make the server look up arbitrary places and the per-zip cache stays small. The .env zip is added to the list if it's missing.
• 2026-10-15: Status rules stay in .env (STATUS_NAMES, UNASSIGNED_EXCLUDED_STATUSES). They're the help desk's definition of open and closed, not a personal preference. Hiding a status column is per user, but hidden columns still count toward Total open.
• 2026-10-19: The closed feed's statuses are per user (feedStatuses, v2.1.1). This narrows the decision above: STATUS_NAMES still defines open and closed for the status table, Closed today and the unassigned count, and is the default for feedStatuses. Our SDP closes a resolved ticket on its own after 2 or 3 days if it isn't reopened, so watching only Closed gave the ding days late. When Resolved is in a user's list, the feed times a ticket by resolved_time before completed_time, so the automatic close (new completed_time, old resolved_time) falls outside the check window and doesn't ding again. A reopen and re-resolve gives a new resolved_time, which counts as a fresh close. An empty list is refused, because the feed would then take any status. Before this, an empty STATUS_NAMES made the feed take any status; now it falls back to Closed,Resolved,Canceled like the rest.
• 2026-10-19: The technician table sorts by clicking its headers, remembered per browser (localStorage key sdp-table-sort, like the sound). This replaced a short-lived tableSort setting in the Settings dialog (never released): headers are easier to find and switch, and a wallboard and a desk on the same account can sort differently. The default is name A to Z; a new column starts A to Z for names and highest first for counts, and a second click reverses it. Ties always go A to Z. The arrow points toward the larger end, so ↓ means A to Z or lowest first and ↑ means highest first. Skylar found the usual ↑-for-ascending convention backwards, so don't "fix" it. A saved status sort whose column has no tickets right now just sorts everyone as 0 (A to Z).
• 2026-10-19: Contrast pass before any theme picker (v2.2.0). Feedback asked for higher contrast or theme options. We fixed the one dark theme first and will add a small per-browser picker (Dark, Light, High contrast) only if people still ask; a gallery of themes was ruled out because each one has to be rechecked against SDP's status colors, the heat scale and every new panel. Every color is now a token in :root in index.html (no inline or one-off colors), with text checked against --card and --card-raised for WCAG AA: --faint went from 2.9:1 to 6:1, --muted from 7:1 to 9:1. The table heat scale uses its own --heat-max (2.9:1 against the card, ink on it 5.2:1) with a floor of 25%, so a cell with one ticket is visible.
• 2026-10-19: MCCS colors (v2.2.0). There's no MCCS brand guide that we know of, so navy #1a3d6b and green #4f763c were sampled from the ITS logo's pixels (--mccs-navy, --mccs-green). Both are too dark for text on the dark theme, so the page uses lifted versions of the same hues: the accent (--signal, was orange) is MCCS green at 58% lightness, the heat scale is MCCS navy at 44%, and the backgrounds are tinted toward the navy hue. The header stripe is the raw green. Errors keep the red-orange --alert so they never look like the green accent. SDP's own status colors (the column swatches) are untouched. Nothing is smaller than 12px except the feed's "ago" label (11px, uppercase). Recheck the ratios when changing a token.
• 2026-10-15: Sound stays per browser (localStorage), not in the saved settings, so a muted desk browser doesn't mute the wallboard signed in as the same person.
• 2026-10-15: Sign-in is limited by email domain (ALLOWED_EMAIL_DOMAINS). Any Zoho account can approve our client, and before this, an outside account got a session and /api/session handed it the .env site, group and technician names (ticket queries failed with their token). Prod v2.0.0 had this gap until v2.1.0 was deployed on 2026-10-16. The callback now refuses other domains and revokes their token, requireSession re-checks every call so older cookies die too, and the server won't start without the setting. Students don't have Zoho accounts, so the domain is enough for now. Next step: check the user is an SDP technician at sign-in, which probably needs a users/technicians read scope.
• 2026-10-16: No ticket subjects, anywhere (FERPA). Subjects can name students or describe their situation. The closed feed and the unassigned query no longer put subject in fields_required, and the objects the server returns are built field by field, so a subject SDP sends anyway (if it ever ignores fields_required) is dropped before it leaves the server. Feed cards show the ticket number, technician and closing time. Don't add subject, description or any other free-text field to a query or response without a new review; to identify a ticket, use its number and open it in SDP.
• 2026-10-15: /api/names makes the caller prove SDP access first (a one-row requests query with their token) before returning the shared cache, so the names never go to a token SDP refuses.
• 2026-10-15: No published ports in compose. Docker's port publishing bypasses ufw, which would skip NPM's IP access list. The container joins NPM's network instead.
• 2026-10-15: CI uses plain docker commands on the self-hosted runner, not the buildx or login actions, so there are fewer moving parts on the runner. Pushes both the tag and latest.

Known issues

• The status table scrolls sideways once there are more statuses than fit, because the page is capped at 1,320px. Consider widening the page or narrowing the sidebar.
• All statuses not in STATUS_NAMES count as open, including "Need Information" and "On Hold". This is intentional; people can hide those columns in Settings.
• If Zoho doesn't return an email at sign-in (the profile call failed), that session can't save settings and uses the defaults; signing in again fixes it.
• No screenshots: images/ was removed on 2026-10-15 and new ones are due.