DEV Community

Cover image for Chaos engineering with Dynatrace: measure before you inject a failure
Span Observer
Span Observer

Posted on Originally published at span.observer

Chaos engineering with Dynatrace: measure before you inject a failure

Before scheduling a chaos engineering with Dynatrace experiment in production, one question matters: « if I slow down or cut this component, how will I know within ten minutes that the experiment worked, or that it must stop? » Without a numeric answer, a deliberate failure is just a failure.

This article starts from that operations decision. It does not describe a fault-injection tool: in the documentation and the vendor post I read, Dynatrace is used to measure (starting state, effects, cause), not to cause the failure. Injection stays with your usual tool. The original post (Build systems more reliably with Dynatrace: Chaos Engineering, August 2024, updated January 2025) defends the same general idea; the outline below is my own.

Scope: the Service-Level Objectives, Site Reliability Guardian (SRG) and Workflows apps of Dynatrace SaaS, built on Grail and DQL. I did not verify their availability on Managed, so assume nothing for that case.

Write the stop condition before the injection

A chaos experiment is a hypothesis test. Write it as a three-line contract: what you break, what you expect, and the threshold that triggers an immediate stop. If you cannot put numbers on the second and third lines, you are not ready.

YAML chaos experiment hypothesis file: target, injected failure, blast radius, expected outcome and stop thresholds on failure rate and response time

Three choices sit in this file. The blast radius is deliberately small (one zone, a share of traffic). The expected outcome is expressed as a service level objective. The stop thresholds are stricter than the objective, so you can interrupt before the error budget is gone. To pick good indicators, see how to choose SLIs in DQL.

Measure the starting state with DQL

The classic weak spot is a hypothesis built on a feeling (« this service is solid »). First record a baseline over a normal period, for example seven days, at a traffic level comparable to the experiment.

DQL timeseries query on service request metrics computing the failure rate as a percentage, per service, over seven days

The query is an example to adapt: check metric keys and dimensions in your environment before using it; I have not run it on a tenant. Keep the method: a failure rate and a response time per service, over a comparable period, recorded before any injection. Add the number of open problems and their usual duration, to know whether your system is already fragile that day. If the baseline is already bad, postpone the experiment.

Turn the hypothesis into checkable objectives

Two platform building blocks do this, and they answer different questions.

  • An SLO: the indicator (SLI) is a series normalised between 0 and 100% computed by a DQL query, compared with a target over an evaluation period. The error budget is the gap between the status and the target; the burn rate tells you how fast you are spending it. During the experiment, the burn rate says whether the test is costing too much. See burn-rate alerts.
  • An SRG guardian: a group of objectives, each fed by a DQL query and judged against warning and failure thresholds. The overall result is the most severe of the individual results.

For an experiment, the guardian is the referee. One objective per line of your hypothesis file: failure rate, response time, and a capacity indicator if you stress the CPU. Auto-adaptive thresholds need at least five past validations and return « info » until then: for a first experiment, set static thresholds taken from your baseline.

Automate the verdict with a workflow

A workflow can call the guardian action after the injection window, with an event trigger or an on-demand call. The result exposes a validation status (info, pass, warning, fail or error), an identifier and a link to the validation. You can derive a clear decision: pass, widen the blast radius; warning, repeat at the same scope; fail, stop and open the ticket. It is the same logic as a release gate, described in the article on SRG and release SLOs, applied here to a deliberate failure. For workflow basics, see module 7 of the SaaS course.

A word on vocabulary: the SRG documentation calls this step « Site Reliability Guardian » in the action picker, « guardian validation action » elsewhere, and « run validation » in generated workflows. Look for the match, not the exact label.

Chaos engineering in six steps

  1. Record the baseline and note the start time.
  2. Create the guardian and the workflow; test them first without any fault, the result must be pass.
  3. Announce the window to on-call and name the person who can abort.
  4. Inject the failure on the smallest scope, with your injection tool.
  5. Let the validation run and watch the burn rate; if the stop threshold is crossed, cut without debate.
  6. Analyse: if a problem opened, Davis root cause analysis shows whether it was attributed to the fault or to something else; fix, then widen by one notch.

The vendor post mentions, on its demo, request slowdowns and CPU stress followed by a root cause analysis. I did not replay that demo: treat it as a usage example, not a result measured on your side.

Best practices

  • Start in pre-production, then production on a tiny share of traffic.
  • One change at a time: no experiment during a deployment.
  • Keep stop thresholds stricter than the SLO, otherwise you stop too late.
  • Keep a record of each experiment: hypothesis, baseline, guardian result, fix.
  • Blameless tone: the experiment tests the system, not the team.

Frequently asked questions

Does Dynatrace inject the failures?

In the documentation pages and the post I read, no: they describe measurement, analysis and automating the decision. Injection belongs to another tool. Check your version’s release notes if you are looking for a native feature.

Do I need an SLO or a guardian?

Both help. The SLO follows the service over time and its error budget; the guardian gives a one-off verdict on a precise window, which suits an experiment.

Can I test directly in production?

Only with a reduced blast radius, a solid baseline and an automatic stop criterion. When in doubt, start in pre-production.

Conclusion

Chaos engineering with Dynatrace starts with measurement: a baseline, stop thresholds and an automated verdict. The failure comes after. To go further, error-budget logic is detailed in the article on the error budget as a release gate. Official sources: Site Reliability Guardian and Service-Level Objectives.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Dynatrace is a trademark of Dynatrace LLC. Features change: check them in the official documentation for your environment.


Originally published on span.observer. French version: https://span.observer/fr/chaos-engineering-dynatrace-mesurer-avant/.

Top comments (0)