Estimated reading time: 8 minutes
I had an early night planned. Instead, I got an SOS email from a client: their site was down, showing a cryptic error about something called “Redis.” A few minutes later, I realised a second client site — hosted on the same server — was down too.
Here’s the investigation that followed, what actually caused it, and what I changed as a result. If you run a website — or a few — I think there’s something useful in here for you too, because this isn’t really a story about my server. It’s a story about where web traffic is heading for everyone.
The message nobody wants to get
The error itself was almost helpful: “Error establishing a Redis connection. To disable Redis, delete the object-cache.php file.” Redis is a caching service that sits quietly in the background making WordPress sites faster. When it falls over, WordPress usually keeps limping along without it — except, this time, it took the sites down with it.
I deleted the file to get both sites back online immediately, and then went looking for why it happened.
Ruling things out
My first assumption was memory pressure — maybe Redis had simply been configured with too small a memory limit and got shut down when it hit the ceiling. I checked. It hadn’t. There was no memory cap on the service at all.
My second theory was the server’s own security software killing it for using too many resources. Checked that too — no record of it happening.
Then I found the actual kernel log entry, and it was unambiguous: the operating system’s out-of-memory killer had stepped in and force-killed several processes in the space of about thirty seconds — two background PHP workers, and then Redis itself. The server had run out of breathing room, however briefly, and started sacrificing processes to survive.
Which raised the obvious next question: why, all of a sudden, on an ordinary Wednesday?
The real culprit
I pulled the web server’s access logs for the exact two-minute window the crash happened in. Just over a thousand requests, on two sites, in about 120 seconds.
Almost none of it was human. When I broke down the traffic by user-agent, the list read like a who’s-who of AI companies: several different automated crawlers, all converging on the same two sites within the same couple of minutes — a mix of crawlers that train AI models on web content, and crawlers that AI assistants use to answer people’s questions in real time.
It wasn’t an attack. It wasn’t anything targeted at these two sites specifically. It was closer to bad luck — several unrelated companies’ crawl schedules happening to overlap, on a server that didn’t have any cushion (no swap space configured) to absorb the spike gracefully.
There was also a smaller, avoidable piece of the puzzle: one of the sites had an old staging/preview URL that was never meant to be public, and some of that bot traffic was hitting it too — essentially a duplicate copy of a small site, getting crawled for no reason at all. Worth checking your own setup for that kind of forgotten door left ajar.
Why a bot burst could take down the whole server
Here’s the detail that actually made this click for me. Every WordPress page a visitor — or a bot — loads spins up a background PHP process to build that page. On my server, each of those processes was eating roughly 200MB of memory.
Two of the affected sites had a worker-process limit left over from solving a completely unrelated problem months earlier — set high enough that, under a big enough burst, either site alone could spin up enough of these processes to claim the server’s entire memory allowance by itself. It didn’t matter that memory usage looked perfectly normal on an ordinary day; the ceiling for what a single bad moment could do was far too generous.
That’s a genuinely different fix from “buy a bigger server.” I brought those limits down to a sane number across every site on the box, so a burst on any one site now gets contained to a small slice of memory, instead of being able to swallow the whole server on its own.
Wait — is this happening to everyone?
Short answer: yes, and it’s accelerating. A few figures that stopped me in my tracks while I was reading around this afterwards:
- Automated (bot) traffic has reportedly overtaken human traffic on the web for the first time, according to recent measurements — a milestone that had been predicted for a year or two further out than it actually arrived.
- AI-specific crawler traffic in particular has been growing at a genuinely startling rate year over year — we’re talking multiples, not percentage points, in some measurements.
- Not all of these crawlers behave the same way. Some exist purely to feed AI training data and essentially never send a visitor back to your site. Others exist to let an AI assistant answer a live question by referencing your page — and can send you a real visitor, sometimes a very well-qualified one.
So the “ask ChatGPT” and “ask Gemini” ads you’ve probably noticed on your phone lately aren’t just marketing noise — they represent a real, measurable shift in how people find things online, and it comes with a real, measurable cost in server traffic for the rest of us to absorb.
What I actually changed
A few concrete things came out of this:
- Closed off the exposed staging URL so it can’t be crawled or indexed at all — a simple server-level redirect, invisible to real visitors.
- Split the crawler traffic deliberately in `robots.txt` — blocking the crawlers that only take (pure AI-training bots) while explicitly allowing the ones that can actually send a visitor or a citation back. Not a blanket “block all bots” approach, which would have thrown away the useful half along with the useless half.
- Tightened those per-site worker limits (see above) across every site on the server — the actual fix for the actual failure mode, and it cost nothing.
- Asked my hosting provider about adding a memory cushion (swap space) as extra insurance. Turns out that’s only available on their Dedicated server tier, not the VPS I’m on — so an upgrade was floated as the alternative. I’m holding off on spending more for now, since the worker-limit fix directly addresses what actually went wrong, and revisiting the upgrade conversation only if it turns out I’m wrong about that.
- Set up real uptime monitoring, because the honest truth is I only found out about the original outage because a client emailed me, not because I noticed first. That’s not a great system, and it’s now fixed — checks running on both affected sites, with SSL expiry monitoring added in for good measure.
If you run a website, here’s the actually useful bit
You don’t need to have gone through a server crash to take something from this:
- Check for exposed staging or preview URLs. If your host gives every site a temporary “yourdomain.hostname.com” style URL, check whether it’s still publicly reachable after your real domain went live. If it is, close it off — it does nothing for you and bots do find it.
- Think about your `robots.txt`, not just whether you have one. “Block everything” and “block nothing” are both blunt instruments. Blocking the crawlers that only extract value while allowing the ones that can send you a visitor is a more useful middle ground.
- Get some form of uptime monitoring, even a free tier. Waiting for a customer to tell you your site is down is a genuinely bad system, and it’s a cheap problem to fix.
- If you’re on shared or VPS hosting with more than one site, check each site’s PHP worker-process limit. If it’s set generously (or was never deliberately set at all), one site having a bad moment can take every other site on the same server down with it. Bringing that ceiling down to something sane costs nothing and contains the damage.
Where it stands now
Everything above is done — the exposed URL is closed, robots.txt is split sensibly, the worker limits are fixed across the whole server, and monitoring is live and watching. I’m not going to claim this is permanently solved forever, because no fix survives contact with the internet with a 100% guarantee. But I understand why it happened now, in a way I didn’t before, and the actual mechanism that let it happen is closed off. If it happens again, I’ll know within minutes instead of waiting for a client email — which, on its own, was worth this entire exercise.