DEV Community

Sergey Shinder profile picture

Sergey Shinder

I am Sergey Shinder

Joined Joined on 
The ten minute outage that took six hours to finish

The ten minute outage that took six hours to finish

1
Comments
2 min read
The canary that never saw the traffic that would have broken it

The canary that never saw the traffic that would have broken it

Comments
2 min read
The order ID that changed by one digit between two services

The order ID that changed by one digit between two services

Comments
2 min read
The graph that stayed flat because nothing was reporting any more

The graph that stayed flat because nothing was reporting any more

Comments
2 min read
The CPU limit that throttled a service using a third of its quota

The CPU limit that throttled a service using a third of its quota

Comments 1
2 min read
The hotfix somebody made by hand that our nightly apply reverted

The hotfix somebody made by hand that our nightly apply reverted

Comments
2 min read
The required status check that never ran and blocked nothing

The required status check that never ran and blocked nothing

Comments
2 min read
The HTTP client with no timeout that took down four services

The HTTP client with no timeout that took down four services

1
Comments
2 min read
The migration that shipped with the code and locked the table for eleven minutes

The migration that shipped with the code and locked the table for eleven minutes

Comments
2 min read
The paginated export that silently skipped records every night

The paginated export that silently skipped records every night

Comments
2 min read
The one percent trace sampling that never captured a single failure

The one percent trace sampling that never captured a single failure

Comments
2 min read
The liveness probe that restarted every pod during a database slowdown

The liveness probe that restarted every pod during a database slowdown

Comments
2 min read
The list element we removed that recreated nine other resources

The list element we removed that recreated nine other resources

Comments
2 min read
The CI cache key that let one branch poison every build

The CI cache key that let one branch poison every build

1
Comments
2 min read
The load test that passed because it was too polite

The load test that passed because it was too polite

1
Comments
2 min read
The artifact we built twice and got two different binaries

The artifact we built twice and got two different binaries

1
Comments
2 min read
The certificate our partner rotated without telling anyone

The certificate our partner rotated without telling anyone

Comments
2 min read
The debug logging we left on in production for a month

The debug logging we left on in production for a month

Comments
2 min read
The container that ignored SIGTERM and dropped every in flight request

The container that ignored SIGTERM and dropped every in flight request

1
Comments
2 min read
The Terraform state file that stored our database password in plain text

The Terraform state file that stored our database password in plain text

Comments
2 min read
The pipeline that could deploy to production from any branch

The pipeline that could deploy to production from any branch

Comments
2 min read
The three nines we promised without measuring anything

The three nines we promised without measuring anything

Comments
2 min read
The deploy that needed one person who was on holiday

The deploy that needed one person who was on holiday

Comments
2 min read
The optional field every consumer treated as required

The optional field every consumer treated as required

Comments
2 min read
The metric label that cost more than the service

The metric label that cost more than the service

Comments
2 min read
The StatefulSet we kept treating like a Deployment

The StatefulSet we kept treating like a Deployment

1
Comments
2 min read
The Terraform plan that quietly said destroy

The Terraform plan that quietly said destroy

Comments
2 min read
The flaky test we retried instead of fixing

The flaky test we retried instead of fixing

Comments 2
2 min read
The Terraform module we made too flexible to use

The Terraform module we made too flexible to use

Comments
2 min read
The single point of failure we found by accident

The single point of failure we found by accident

Comments
2 min read
The rollback that could not undo the migration

The rollback that could not undo the migration

Comments
2 min read
The webhook that arrived twice and charged twice

The webhook that arrived twice and charged twice

Comments
2 min read
The request we could not follow across four services

The request we could not follow across four services

Comments
2 min read
The image tag that meant something different every hour

The image tag that meant something different every hour

Comments
2 min read
The build that only worked on the CI runner

The build that only worked on the CI runner

Comments
2 min read
The green build that shipped a broken release

The green build that shipped a broken release

1
Comments
2 min read
The backup that had never once been restored

The backup that had never once been restored

Comments
2 min read
The feature flag we forgot to delete

The feature flag we forgot to delete

Comments
2 min read
The retry storm we caused ourselves

The retry storm we caused ourselves

Comments
2 min read
What our averages were hiding from us

What our averages were hiding from us

Comments
2 min read
The rollout that took down the thing it depended on

The rollout that took down the thing it depended on

Comments
2 min read
Why we split one Terraform state into twenty

Why we split one Terraform state into twenty

Comments
2 min read
Der Rollback ist wichtiger als das Release

Der Rollback ist wichtiger als das Release

Comments
2 min read
Бюджет помилок навчив мене відпускати

Бюджет помилок навчив мене відпускати

Comments
1 min read
Автоматизуйте не деплой, а відкат

Автоматизуйте не деплой, а відкат

Comments
1 min read
Інтеграція, яка не падає разом із чужим сервісом

Інтеграція, яка не падає разом із чужим сервісом

Comments
1 min read
Метрики, які я перевіряю о третій ночі

Метрики, які я перевіряю о третій ночі

Comments
1 min read
Kubernetes не рятує від того, чого ви не продумали

Kubernetes не рятує від того, чого ви не продумали

Comments
1 min read
Terraform, який рятує вас у поганий день

Terraform, який рятує вас у поганий день

Comments
1 min read
Пайплайн, який пережив зміну команди

Пайплайн, який пережив зміну команди

Comments
1 min read
Ретроспектива без поиска виноватых спасла нам команду

Ретроспектива без поиска виноватых спасла нам команду

Comments
1 min read
Ночные релизы, от которых мы наконец отказались

Ночные релизы, от которых мы наконец отказались

Comments
1 min read
Ретраи, которые сложили внешний сервис вместо того чтобы спасти

Ретраи, которые сложили внешний сервис вместо того чтобы спасти

Comments
1 min read
Мы тонули в алертах и не видели настоящий пожар

Мы тонули в алертах и не видели настоящий пожар

Comments
1 min read
Под, который перезапускался каждые сорок секунд

Под, который перезапускался каждые сорок секунд

Comments
1 min read
История о том, как один terraform apply снёс базу

История о том, как один terraform apply снёс базу

Comments 1
1 min read
Как зелёный пайплайн однажды выкатил сломанный билд

Как зелёный пайплайн однажды выкатил сломанный билд

Comments
1 min read
La fiabilité se décide avant l'incident, pas pendant

La fiabilité se décide avant l'incident, pas pendant

Comments
2 min read
Le déploiement en un clic n'existe pas vraiment

Le déploiement en un clic n'existe pas vraiment

Comments
2 min read
L'intégration qui échoue en silence est la plus dangereuse

L'intégration qui échoue en silence est la plus dangereuse

Comments 1
2 min read
loading...