Christopher Engelhardt
Threat Watch is how Rainier IT notices a software flaw, works out whether it touches anything we look after, and sends the fix down the right road. Some of those roads now end with a patch installed the same day, with a person holding a veto rather than a to-do item.
This post walks through the whole pipeline with the real rules and the real code, trimmed for reading. It also says plainly which parts run by themselves today and which still wait for a person, because that difference is what you would want to know.
Why we built it
In October a serious PHP flaw nicknamed PHP2Shell made us realize that we need to implement a patching protocol that fixes exploits quickly. News sites and researchers were writing about it roughly nine hours before any CVE feed had it, and our old rule waited for proof of exploitation. So we rebuilt the watch around one question: how early can we know, without crying wolf?
Step one: polling
Every 15 minutes a job on our own server reads the sources below. Feeds are fetched with ETags, so an unchanged feed costs the publisher almost nothing.
- Known-exploited lists: CISA KEV and VulnCheck KEV, which give us proof of real-world attacks.
- CVE records: cve.org and NVD (every 30 minutes), for severity, often hours before NVD scores it.
- Vendor advisories: Chrome, Fortinet, PHP, WordPress and php-src on GitHub, for the vendor’s own word.
- News and research: CISA alerts, BleepingComputer, Krebs, SANS ISC and Qualys, for the earliest reporting, often with no CVE yet.
- Social sightings: Mastodon and Bluesky, for a CVE suddenly discussed by at least 4 accounts in 6 hours.
Each item then runs through plain rules. Here they are, from the code:
def rules(c):
"""
A on a known-exploited list (CISA KEV, VulnCheck KEV)
B a CISA alert
C the vendor, or a report, says exploited in the wild
D CVSS 9+ over the network with no login
E the same bug from 2+ independent sources within 6 h
K a Linux kernel CVE
P a critical security release announced in advance
S a CVE suddenly discussed by many accounts
"""Rules A and B, and a vendor saying it is exploited, count on their own. Everything else needs a second opinion from a small AI model (Claude Haiku, capped at $1.50 a day), which reads the headline and says whether a small office should care. It groups stories about the same bug too. It never starts anything by itself, as you will see next.
Step two: “under active attack” is a strict test
One label decides whether a clock can start, so the model’s opinion alone does not count:
def active_attack(c):
"""On a known-exploited list, the vendor says so,
or a report says so and triage agrees.
Triage's yes alone isn't enough: it only read a
headline, and this decides a prod release."""
r = c.get("rules", ())
return (c.get("kind") == "kev" or "A" in r
or bool(c.get("vendor_says"))
or ("C" in r and exploited(c)))Step three: does it touch anything we manage?
Most flaws do not. We read the installed software of every managed computer once a day and compare product names conservatively, so “Cisco Secure Client” never matches “Cisco Secure Firewall”. An early dry run produced 40 matches and about 12 real ones, which is why the matching got stricter. Our own services are listed in a separate file.
Critical and high alerts open a VULN ticket in our ticketing system, Zammad. Every 30 minutes a separate job, which we call the Archmage, picks up each open one. A model reads the advisory only to pull out product names, and read-only probes check the versions actually running on our hosts. Then ordinary code gives a verdict:
def verdict(adv, hits, probes, complete=(), inventory_ok=False,
threat_watch=None, confirmed=None):
"""A probe error is a match we cannot read:
unknown, never not_affected or patched.
Safe verdicts need every precondition: products named,
a fresh inventory, no Threat Watch match line for
not_affected, and for patched a probe key for every
probed product."""An old inventory or a failed probe gives “unknown”, and a person looks at it. Advisory text is treated as data and never reaches a command. An “affected” verdict creates one linked patch ticket.
Step four: routing
One alert can take several roads. Node.js might be on a service with staging and on one without. The sorting code has no AI in it:
- Fast path: client Windows computers and their apps, under active attack. Same-day patch, details below.
- Regular patching: the same computers, not known exploited. They wait for the client’s normal patch window.
- Staging: our own client portal, website and remote management platform. Staging first, release after that.
- A person decides: servers, Macs, Linux, client websites and anything with no staging. An alert goes to the owner, and nothing else happens.
Step five: the fast path
This is the auto-remediation part. Regular patching waits 5 days for Windows and 2 for apps, which is sensible until a flaw is being attacked. Each hour, per client, we read the vulnerability list from Action1, our patching platform, and keep the known-exploited rows that have a fix. Then:
1. post to Discord
client, CVE, product, computers, what will happen
2. wait 10 minutes
a check mark fires it now, a stop sign cancels it,
silence fires it
3. fire
one update, only the affected computers,
no forced restart
4. check
every 5 minutes against Action1's results
5. email the client
facts from a fixed template, once every computer
finishes or at 4 hoursThe safety rules matter more than the speed:
- The clock starts only once the post and its reactions are in place. If we cannot post, or cannot read the reactions, nothing fires, because a veto cannot be ruled out. We hear about it another way.
- Client, CVE and computer are claimed once, for good. A request with an unknown result is never repeated.
- Turning it on is not a burst. The first sweep of a client records what is already open and leaves it to regular patching. At most 5 new runs start per sweep.
- “Patched” has a strict meaning. Action1 also answers “Success” with “No updates are applicable”, which installed nothing. So we also need its install wording and no restart outstanding.
- No AI writes to clients here. The email is a template filled in with facts.
Our own services
Our own systems have a path too: a crew run builds the fix, staging and the regression gate test it, and Discord gets a post. For a flaw under active attack on the client portal, the release would go ahead 30 minutes after that post unless stopped, and only with every gate check green, a rollback ready and Discord working:
def all_green(checks):
"""Every gate check present and green. No checks is not green."""
return bool(checks) and all(c.get("ok") is True for c in checks)Today this path runs in dry-run mode. It logs what it would do, and every release of our own services still takes a person’s approval. Our remote management platform will always take one, whatever the checks say.
How it has done so far
Since Oct. 22, CISA has added 12 flaws to its known-exploited list, and Threat Watch alerted on 10. Eight of those alerts came before CISA’s listing date, by a median of about 18 hours across the 10. CISA lists by date rather than by hour, so we count from midday Eastern. Two got past us. It is a short window, so treat it as a first reading.
The fast path went live for managed clients on Nov. 1. On our test computer, a clean run posted at 12:34 p.m., waited just over 10 minutes with nobody touching it, installed, and sent the email. That test used a simulated known-exploited row pointing at a real missing update. The lab testing found five bugs, including a veto window that ran about 13 seconds short. All five are fixed. A real flaw through real detection is the next proof, and Patch Tuesday on Nov. 10 is the next good chance.
What this means for clients
- Known-exploited flaw on a covered Windows computer: we aim to patch it the same day a fix is available, then email you what we did.
- No forced restarts. If one is needed to finish, the email says so.
- Computer offline: it shows as still in progress until it comes back, and we follow up.
- No fix yet: we keep checking every hour. There is nothing to install, so you do not get an email about it from this path.
- Servers, Macs and Linux: a person decides, because a surprise patch on a server costs more than a few hours.
If you want this on your own computers, Business Watch can be ordered online, and our a la carte security plans let you pick individual protections. Not sure where you stand? The free security check is a good first step.
This article was drafted by Claude, an AI assistant, from Rainier IT’s own code and test records, and reviewed by Christopher Engelhardt before publication. Spot an error? Email [email protected].
Protected by reCAPTCHA. Google's Privacy Policy and Terms apply.