A Real Incident: How Service Fingerprinting Triggered a fail2ban Ban

We run SurfaceDiff on our own domains, the same as any customer would. This is the story of a feature that seemed simple, broke in a way we didn't expect, and took three separate attempts to actually get right - told straight, including the parts that didn't work.

The idea

Knowing a port is open only gets you so far. "22/tcp is open" is a fact; "this is OpenSSH 8.9p1 on Ubuntu" is evidence you can actually act on. We wanted to add real service and version fingerprinting - the same kind of banner-grab technique behind tools like nmap -sV- so an open-port finding could carry an actual answer to "what is this," not just "it's there."

The mistake

We built it as part of the regular scheduled scan and ran it for real, against our own infrastructure. Within hours, one of our own monitored domains threw a batch of incidents that had never happened before in that domain's entire history: several ports and web services suddenly reporting closed, all at once. Nothing had actually changed on the server. Our own scan had caused it.

What was actually happening

The target server ran fail2ban, and one of its jails was watching for exactly the connection pattern a version-detection probe produces: connect, read a banner, disconnect - without completing a full protocol handshake. Three of those from the same source within the jail's window (maxretry=3, findtime=600s- three strikes in ten minutes) triggered a real ban, and the ban's own reach was broader than the one service that tripped it - other, unrelated ports on the same host went unreachable too, for the length of the ban.

This is a completely reasonable thing for a mail server to do. It's also exactly the kind of hardening the customers we care most about - the ones running their own infrastructure carefully - are likely to have in place.

Two attempts that didn't work

Our first fix was spacing: probe one port at a time instead of all at once, with a delay between each. That still failed, because the ports in question were all served by the same underlying daemon, and even spaced by tens of minutes, three of them still landed inside a single ten-minute window somewhere in the sequence.

The second attempt added smarter scheduling on top - group ports by the daemon family most likely to be watching them, and interleave probes across groups so the same family never gets hit twice in a row. Better, but still not reliable enough: a single probe can occasionally register as more than one hit in the target's own logs, and once we tested the fix against the real, full scan pipeline (not just the fingerprinting step in isolation) it still tripped a ban.

What actually fixed it

The real problem wasn't the spacing math - it was running a whole domain's worth of fingerprinting in one sitting at all. We moved it out of the scheduled scan entirely and into an existing queue: the same on-demand mechanism a customer can already trigger by hand for one specific port now also gets fed automatically, once a day, with every open port on that domain. A worker drains that queue one probe at a time, and enforces one simple rule - never probe the same host again within an hour of the last probe to it.

That one rule is what actually holds: an hour is roughly six times longer than the jail window we measured, so no matter how a single probe happens to register in a target's own logs, there's no way for our own activity to cluster inside a typical detection window again. A domain's full port set gets fingerprinted gradually over the following day, in the background, without the scan itself ever needing to wait on it.

Why we're telling you this

A lot of scanning tools either don't think about this failure mode at all, or find out about it from an angry customer instead of their own testing. If you run any kind of active reconnaissance against real infrastructure - your own or a customer's - this is worth designing around deliberately, not discovering by accident. And if you're evaluating monitoring tools for infrastructure you actually care about staying up, it's a fair question to ask any of them: what happens when your scan runs into a customer's own security tooling?

Curious what a scan of your own domain would surface? Run a free scan - free, no card required, just a DNS record to prove you own the domain. Running a lot of domains on shared infrastructure? See how SurfaceDiff handles that.