sdp-request-watcher/docs/NOTES.md

Notes

SDP v3 API gotchas

Docs: https://www.manageengine.com/products/service-desk/sdpod-v3-api/getting-started/search-criteria.html

• Condition names have spaces: greater than, is not, lesser than. greater_than isn't valid, and SDP silently ignored the entire search criteria because of it, returning every ticket (124,050). If a query's total_count is huge, suspect the criteria first.
• Paging: row_count is at most 100. Use start_index, and stop when list_info.has_more_rows is false. Ask for get_total_count: true on the first page. getAllRequests dedupes by ID and stops at 20 pages or when the total is unreachable.
• Multiple values: use a values: [...] array, and join criteria with logical_operator: 'AND' (see allOf() in tickets.js).
• "Technician is empty": written as { field: 'technician', condition: 'is' } with no value. This isn't documented, since the docs only show the "is not" form, but it's verified to work.
• Field names that work in filters: site.name, group.name, technician.name, status.name, last_updated_time.
• Group names: the ManageEngine UI shows groups as "Customer Support « KVCC", but the API name is plain "Customer Support".
• Sign-in scope: SDPOnDemand.requests.READ is enough for every SDP query so far. AaaServer.profile.READ is added only for the signed-in user's name (GET accounts.zoho.com/oauth/user/info). If that call fails, sign-in still works. Changing the scopes means everyone signs in again.

Zoho OAuth gotchas

• Client type: web sign-in needs a Server-based Application client. The old Non-Browser (device-code) client can't do redirects. The redirect URI must match PUBLIC_URL/auth/callback exactly.
• Refresh tokens: you only get one with access_type=offline, and prompt=consent makes Zoho issue one on every sign-in. Access tokens last about an hour; renew with POST accounts.zoho.com/oauth/v2/token, grant_type=refresh_token.
• 20 refresh tokens per user per client: Zoho silently deletes the oldest when a 21st is issued. Each browser sign-in creates one, which is why Sign out revokes (POST /oauth/v2/token/revoke?token=...). A user signed in on more than 20 browsers would find the oldest one signed out.
• Errors come back as HTTP 200: a bad code or refresh token returns { "error": "invalid_code" } with status 200. auth.js checks for access_token instead of relying on axios throwing.
• Data centre: everything points at accounts.zoho.com (US). The callback also gets location and accounts-server parameters, which would matter only for users in another Zoho DC.

Deployment gotchas

• NPM runs in Docker, so localhost in a proxy host is NPM's own container. For a bare-metal test run, forward to the gateway of NPM's Docker network and allow that subnet through ufw. The production container is reached by name (sdp-dashboard:3000) because it joins NPM's network.
• Port 3000 is Gitea's on the VPS. Bare-metal test runs use PORT=3005. Never put PORT in the production .env; the container listens on 3000, and its health check and the NPM host expect that.
• Cloudflare proxy and the NPM access list: a proxied record can make NPM see Cloudflare's IPs, so the allow list blocks everyone. The fix is DNS only, or real_ip_header CF-Connecting-IP; in the proxy host's Advanced config. The live test on 2026-10-15 worked with the access list (an MCCS-listed IP got in, mobile data got a 403).

Decisions

• 2026-10-15: MIT license. Copyright Skylar Grant.
• 2026-10-15: OpenWeatherMap for weather. The free tier covers current weather and the 5-day forecast, and results are cached for 10 minutes. The National Weather Service API was the alternative (free, no key), but it needs three calls per lookup.
• 2026-10-15: Unassigned counter uses its own status exclusions (UNASSIGNED_EXCLUDED_STATUSES) to match the ManageEngine view. The debug list is kept but hidden (SHOW_UNASSIGNED_LIST in index.html).
• 2026-10-15: Per-user Zoho sign-in replaces the shared service token. This reverses the earlier "one service token plus region config" decision, and drops the ADMIN_KEY and data/token.json from phase 1. Leadership's concern was that anyone who reached the page could see data while a shared token was live. Now every request needs the viewer's own sign-in, and NPM limits the site to MCCS addresses. The cost is that each open dashboard polls SDP itself; acceptable for a small team.
• 2026-10-15: Sessions in an encrypted cookie, not a server store. AES-256-GCM with a key from SESSION_SECRET, HttpOnly, SameSite=Lax (Lax so the cookie survives the redirect back from Zoho), and Secure when PUBLIC_URL is https. The container needs no volume or database, and restarts don't sign anyone out. Renewing the access token rewrites the cookie but keeps startedAt, so SESSION_DAYS is a hard limit. Concurrent calls with an expired token share one renewal (renewals map in session.js).
• 2026-10-15: The browser runs the checks and keeps the closed feed (first load asks for 12 hours, then 10-minute windows every minute). The server has no per-user timers or state. This also suits per-browser settings later: the browser will send its settings with each call.
• 2026-10-15: Per-user settings are stored on the server, keyed by Zoho email (lowercased), in DATA_DIR/settings.json. Chosen over the planned localStorage so settings follow people between browsers and no export/import is needed. This gives the container state, so compose mounts the sdp-data volume at /app/data. Only values that differ from .env are stored, so .env changes still reach people who never changed that field. A plain JSON file, rewritten via temp file and rename, is enough for a small team with one process; move to SQLite if that stops being true.
• 2026-10-15: The server reads settings itself rather than the browser sending them with each call, as first planned. The browser can only change them through PUT /api/settings, which validates everything (LIMITS in settings.js). That's for protecting SDP and the server, not access control: queries use the user's own token, and everyone at MCCS can see every site in SDP anyway.
• 2026-10-15: Name suggestions without a new scope. /api/names runs one unscoped query (last 30 days, up to 500 tickets, names only) with the caller's token, at most hourly for everyone, and merges into DATA_DIR/names.json so the list grows over time. Typed names are still accepted; a name matching a suggestion except for case is saved with SDP's spelling, because the technician filters compare exactly. Proper pick lists from SDP's setup/users APIs would need another scope and a re-sign-in for everyone.
• 2026-10-15: Weather locations are a fixed list (src/places.js), not free zip codes, so a browser can't make the server look up arbitrary places and the per-zip cache stays small. The .env zip is added to the list if it's missing.
• 2026-10-15: Status rules stay in .env (STATUS_NAMES, UNASSIGNED_EXCLUDED_STATUSES). They're the help desk's definition of open and closed, not a personal preference. Hiding a status column is per user, but hidden columns still count toward Total open.
• 2026-10-19: The closed feed's statuses are per user (feedStatuses, v2.1.1). This narrows the decision above: STATUS_NAMES still defines open and closed for the status table, Closed today and the unassigned count, and is the default for feedStatuses. Our SDP closes a resolved ticket on its own after 2 or 3 days if it isn't reopened, so watching only Closed gave the ding days late. When Resolved is in a user's list, the feed times a ticket by resolved_time before completed_time, so the automatic close (new completed_time, old resolved_time) falls outside the check window and doesn't ding again. A reopen and re-resolve gives a new resolved_time, which counts as a fresh close. An empty list is refused, because the feed would then take any status. Before this, an empty STATUS_NAMES made the feed take any status; now it falls back to Closed,Resolved,Canceled like the rest.
• 2026-10-19: Table order is a saved setting (tableSort), sorted in the browser. Values are list, name, open, closedToday or status:<name>; the server only validates the value. A status sort is saved by name and compared without case, so it survives that status having no tickets for a while (the rows then fall back to A to Z). Counts sort descending with ties by name, and there's no direction choice; add one if anyone asks. It's in Settings rather than clickable headers so a wallboard keeps its order after a reload.
• 2026-10-15: Sound stays per browser (localStorage), not in the saved settings, so a muted desk browser doesn't mute the wallboard signed in as the same person.
• 2026-10-15: Sign-in is limited by email domain (ALLOWED_EMAIL_DOMAINS). Any Zoho account can approve our client, and before this, an outside account got a session and /api/session handed it the .env site, group and technician names (ticket queries failed with their token). Prod v2.0.0 had this gap until v2.1.0 was deployed on 2026-10-16. The callback now refuses other domains and revokes their token, requireSession re-checks every call so older cookies die too, and the server won't start without the setting. Students don't have Zoho accounts, so the domain is enough for now. Next step: check the user is an SDP technician at sign-in, which probably needs a users/technicians read scope.
• 2026-10-16: No ticket subjects, anywhere (FERPA). Subjects can name students or describe their situation. The closed feed and the unassigned query no longer put subject in fields_required, and the objects the server returns are built field by field, so a subject SDP sends anyway (if it ever ignores fields_required) is dropped before it leaves the server. Feed cards show the ticket number, technician and closing time. Don't add subject, description or any other free-text field to a query or response without a new review; to identify a ticket, use its number and open it in SDP.
• 2026-10-15: /api/names makes the caller prove SDP access first (a one-row requests query with their token) before returning the shared cache, so the names never go to a token SDP refuses.
• 2026-10-15: No published ports in compose. Docker's port publishing bypasses ufw, which would skip NPM's IP access list. The container joins NPM's network instead.
• 2026-10-15: CI uses plain docker commands on the self-hosted runner, not the buildx or login actions, so there are fewer moving parts on the runner. Pushes both the tag and latest.

Known issues

• The status table scrolls sideways once there are more statuses than fit, because the page is capped at 1,320px. Consider widening the page or narrowing the sidebar.
• All statuses not in STATUS_NAMES count as open, including "Need Information" and "On Hold". This is intentional; people can hide those columns in Settings.
• If Zoho doesn't return an email at sign-in (the profile call failed), that session can't save settings and uses the defaults; signing in again fixes it.
• No screenshots: images/ was removed on 2026-10-15 and new ones are due.