054: We're So Quacked

At 3:30 AM the day before Thanksgiving, Ben received an emergency page about a failing API end-point. Rushing to his desk, groggy-eyed and in various states of undress, he jumped into the #incident channel on Slack to see what was happening. What unfolded over the next 30-hours was the manifestation of Ben's worst nightmare. The moment he had been dreading for the last 4-years had finally come to pass: two of his database columns had run out of storage space! Using feature flags, emergency hot-fixes, shadow tables, and a database migration being performed over a transient and unstable terminal session, he and his team somehow made it through to the other side just in time to enjoy Thanksgiving turkey and pumpkin pie!

"Hug your data engineers - they are amazing people!" — Ben Nadel

Notes & Links

• Liquibase
• Percona Tooklit
• MySQL's Information Schema
• Datadog
• CFSearching: CFQueryparam Matrix for MySQL 5
• Ben Nadel: CAUTION: Silent Value Truncation In CFQueryParam Tag In Lucee CFML 5.3.7.47
• Ben Nadel: Recording Datadog / StatsD Gauges For Database Key Utilization
• Ben Nadel: Inspecting Primary And Secondary Index Key Utilization For MySQL

Follow the show and be sure to join the discussion on Discord! Our website is workingcode.dev and we're @WorkingCodePod on Twitter and Instagram. Or, leave us a message at (512) 253-2633 (that's 512-253-CODE). New episodes drop weekly on Wednesday.

And, if you're feeling the love, support us on Patreon.

With audio editing and engineering by ZCross Media.

Episode source