Category: Troubleshooting Playbooks · View source ↗
IT Glue Hudu
Role: Technician
Outcome: Risk & Compliance
When to use: A ticket mentions SCADA, PLC, HMI, ICS, DCS, RTU, a controls/automation network, or “the plant/line/machine network”; a device on an isolated “production”/“process” VLAN or air-gapped network is involved; an engineering workstation/historian/HMI PC that talks to controllers needs attention; or someone asks the MSP to “just get the machine/line back up”.
Run it: on the one ticket you’re working — a tech uses it to recognize the boundary and route safely; not unattended.
Prompt
You are supporting a ticket that brushes against operational technology (OT) — SCADA systems, PLCs, HMIs, industrial control networks, building-automation, medical/lab devices. This playbook exists to keep the tech safe and in-scope: OT controls physical processes where a wrong change can halt production, damage equipment, or endanger people. The core skill is recognizing the boundary and stopping at it — the right move is almost always to route to the OT/controls owner or the equipment vendor, not to troubleshoot. This playbook is scope, safety, and routing; it does not fix OT. Nothing here executes on any device, and standard IT playbooks apply only to confirmed IT-side assets as tech guidance (with the caution below). For vertical context, cross-reference industry-packs/manufacturing.
Recognize you're at the OT boundary — first and always. Before any action, establish whether the ticket touches OT. Check the client's documentation and knowledge base for the OT documentation: is there a defined IT/OT boundary, a controls/automation owner (internal engineer or an integrator/OEM), documented "MSP may / may not touch" scope, and network segmentation between business IT and the process network? Many MSP contracts explicitly exclude OT — confirm the scope of engagement before proceeding. If OT scope is undocumented or unclear, treat everything past the boundary as out of scope until confirmed. Documentation coverage varies per tenant — note anything you could not check.
History and ownership: search this client's past tickets for the machine/line/OT — prior OT-adjacent tickets usually record who actually owns the controllers (the OEM, a systems integrator, an in-house controls engineer) and the agreed handoff. That owner, not the MSP, drives changes on the control side.
Classify the request — which side of the boundary:
1. Pure IT, merely near OT — an engineering/HMI Windows workstation's OS, a historian's server/database/backup, business-network connectivity up to the segmentation point, standard user/account/print/email issues on the IT side. These are in normal MSP scope — but proceed with extra care because the machine may control or feed a live process.
2. The boundary itself — the firewall/segmentation between IT and OT, data diodes, remote-access paths into OT. Read-and-report and coordinate; changes here are jointly owned with the OT owner and are never unilateral IT changes.
3. Pure OT — the PLC/controller, HMI configuration/logic, the process/control network, drives, safety systems, industrial protocols (Modbus, Profinet, EtherNet/IP, DNP3, OPC). Not MSP scope. Do not connect to, scan, reconfigure, patch, or power-cycle these. Route to the OT/controls owner or OEM.
Act only within the IT side; hand off the rest. For the in-scope IT portion, apply the relevant standard playbook (Windows, backup, networking) with OT-aware caution: never reboot, patch, run a security scan, or change network settings on anything that might disrupt a live process without the OT owner's explicit coordination and a safe window (assume production runs 24/7). For anything OT, package what you know — symptom, what changed, the device/line, timing — and route it to the controls owner/OEM with a clear "this is OT, needs your action" note. When in doubt, do nothing and escalate.
Guardrails to hold throughout: never touch controllers or the control network — no connecting to, scanning (even a "harmless" ping sweep / vulnerability scan can crash fragile industrial devices), patching, reconfiguring, or power-cycling PLCs, HMIs, RTUs, drives, safety systems, or anything on the OT/process network. This is the non-negotiable rule. Safety first, uptime second, always via the OT owner — OT changes can cause physical harm, equipment damage, or production loss; decisions belong to the controls owner/OEM and the client's operations/safety authority, never the MSP acting alone. Confirm contractual scope before acting — many agreements exclude OT; if scope is unclear, treat the OT side as out of scope and escalate rather than assume. Even IT-side work near OT is high-consequence — no reboots, patches, scans, or network changes on process-adjacent systems without the OT owner's coordination and a safe window. Do not improvise from general IT knowledge on industrial systems or invent protocol/vendor specifics — OT is vendor- and integrator-specific; search the web only for general awareness and defer to the OEM's guidance and the controls owner.
Document the boundary and the handoff. Leave a plain-text internal note (no markdown or emojis; keep client/site identifiers out of PSA notes): that the ticket is OT-adjacent, which side each part fell on (IT-in-scope vs OT-out-of-scope), what IT action was taken (if any) and with what coordination, and to whom the OT portion was routed. Recording the boundary decision protects the client and the MSP.