Luthfi Farabi Database engineer keeping a production cloud fleet alive —
and building the tools so it mostly keeps itself alive.

The page said "high memory," the cause was three things stacked. No connection pooler in front of a Puma app meant 498 idle connections each holding memory hostage. Long-running statements created lock contention. And a couple of hot queries were doing 23-second sequential scans on tables missing the right index.

The fix went in as a layered stack: PgBouncer in transaction-pooling mode to collapse the connection count, server-side idle timeouts to reap the stragglers, work_mem right-sized so sorts stopped spilling, and a targeted index plan to kill the seq-scans. Then I wrote the 5-Whys RCA in Confluence so the root cause — not just the symptom — is on record.

INSTANT DDL is magic right up until you hit the limits nobody mentions — the instant-add column budget, and the metadata lock that can still stall behind a long transaction. So before touching prod I verified the INSTANT headroom on the table and pre-flighted the metadata locks to be sure the ALTER wouldn't queue behind anything.

Result: the columns landed with no table rewrite and no downtime on a 1TB / 543M-row table, in under a second. The service integration shipped on schedule.

The instinct on a CPU-pinned cluster is to scale up. I scaled down — after finding why it was hot. 843,000 slow queries in 27 hours and 94–98% CPU weren't a capacity problem, they were a query problem. The worst offender was a LATERAL JOIN doing a filesort on every execution; the right composite index removed the sort entirely.

On top of that, over 99% of connections were idle — bloat, not load. With the queries fixed and the connection story cleaned up, the 3× 32vCPU/128GB footprint was wild overkill. Right-sizing the boxes took 40–55% off the bill.

You can't alert on what you've never looked at. I read all 33 panels across the 19-host fleet and wrote down what each one was actually telling me. That surfaced the host quietly running 13,000 QPS with 21 slow queries/sec, another near connection saturation, and 17 blind spots — signals we were graphing but not alerting on.

The output wasn't a complaint, it was a spec: the exact alerts, thresholds and owners needed to turn those blind spots into pages before they turn into incidents.

The thesis: a DBA's judgment can be encoded, and the cloud's control plane can act on it. The roadmap pairs Claude as the brain — anomaly detection, RCA reasoning, index recommendations — with Alibaba RDS as the body that executes scaling, backups and schema changes.

Ten projects, sequenced from safest to boldest: AI anomaly detection, index tuning with shadow validation (test the index against real traffic before it goes live), predictive scaling, automated backup/restore drills, schema-change CI gates, and auto-RCA. Doc first — now turning into running code (see $ now).

Auto-RCA agent — pulls the signals from Grafana, walks a 5-Whys analysis, and writes the draft RCA straight into Confluence via MCP, so the post-incident doc exists before the adrenaline wears off.

Slow-query monitor — a Bash watcher that catches slow queries, hands them to Claude for an explanation + fix suggestion, and drops the whole analysis into Google Chat. Daily standups — auto-generated from Jira via MCP, so the status update writes itself.

A lock-free schema change on a 100GB+ table tripped a known-nasty MySQL virtual generated-column index bug, and the instance started looping through failovers. The safe path: drop the offending index to stop the bleeding, rebuild the table cleanly via DTS, then swap it in with an atomic rename — all inside the agreed maintenance window so no one downstream felt it.

Same week, on the other engine: a MongoDB COLLSCAN on ApsaraDB tracked down and indexed. Different database, same instinct — find the missing index, stop the scan.