But here's the terrifying part. Because wap-03 was "alive" according to basic ICMP pings, the cluster's consensus protocol had been treating it as a voting member. For six months, every time wap-03 choked on a null byte, it would delay the cluster's session replication by 400ms.
At 7:00 AM, Linda called. "Why are the morning graphs showing record throughput?"
Tonight was the night. I had a change ticket: CHG-0421 – Remove wap-03 from cluster and decommission.
I pulled the plug on wap-03 at 2:53 AM.
The remaining two WAPs ( wap-01 and wap-02 ) recalculated their session tables. CPU usage on wap-01 jumped from 18% to 32%. Well within limits. Memory stable. Error rate on the payment API… held steady at 0.01% (baseline noise).
The server went quiet. I held my breath.
But I knew the truth. wap-03 wasn't providing redundancy; it was providing uncertainty . Its TLS cipher suite was outdated (TLS 1.0, a compliance nightmare). Its network card had a known memory leak. And worst of all, the session persistence table would occasionally corrupt, silently dropping 0.5% of payment authorization requests. remove web application proxy server from cluster
At 2:17 AM, I drained the traffic. The F5 showed wap-03 's connection count dropping from 1,200 to 0. Beautiful.
For six months, wap-03 had been a source of low-grade anxiety. Every Tuesday at 4:00 PM, latency on that node would spike by 200ms. The logs showed a cryptic error: Event ID 1309 – Connection dropped by backend . Management refused to let me take it offline. "It's redundant," my boss, Linda, had said. "Redundancy means we keep it."
Instantly, the average response time for the payment API dropped from 340ms to 190ms. A 44% improvement. The error rate fell to 0.001%. But here's the terrifying part
I ran the stop command: Stop-WebApplicationProxy -Node wap-03
Or rather, two of the WAPs did the heavy lifting. The third one, wap-03.internal.stratus.com , was the problem child.
No alerts. No 500 errors. No angry emails from the night shift fraud team. At 7:00 AM, Linda called
As I prepared to shut down the virtual machine, I decided to tail the legacy logs one last time. tail -f /var/log/wap/traffic.log on wap-03 .