Golden signals, alerts and end-to-end tracing

The four golden signals

Google's site reliability engineering guidance popularised four golden signals for any service. They map well onto mainframe workloads.

Batch work needs a slightly different view: whether each job stream will meet its SLA, elapsed time against normal, and abends. Latency for batch is 'will the critical path finish on time'.

Designing alerts people trust

The fastest way to ruin on-call is an alert that fires all night and means nothing. People learn to ignore it, and then they ignore the real one.

• Alert on symptoms first. 'Card authorisation latency above 500 ms for 5 minutes' matters to customers. 'CPU above 90%' may be normal at peak.
• Use duration and rate, not single samples. A spike of one interval is usually noise.
• Every alert needs an action. Link it to a runbook entry. If nobody would do anything, make it a dashboard item, not a page.
• Route to the owner. A DB2 lock alert should reach the DB2 or application team, not only operations.
• Review regularly. After each incident, ask which alert should have fired earlier and which ones were noise.

End-to-end transaction tracing

A distributed trace follows one request through many components. Each hop records a span with a start time, a duration and the same trace ID. Distributed systems pass the trace ID in HTTP headers; the W3C Trace Context standard defines a common header called traceparent.

Inside the mainframe, traditional tools record their own identifiers, such as the CICS task number and the DB2 correlation and accounting tokens. APM products with z/OS support, such as IBM Instana with IBM Z APM Connect, Dynatrace and others, connect these to the distributed trace so a single view shows time spent in each tier. Exact coverage depends on the product, version and subsystem.

Even without a full APM product, you can get much of the value by agreeing a correlation ID: generate one at the edge, pass it through the API and MQ message headers, and have the CICS or batch program log it. Then you can search every log for the same ID.

14:05:02.118Z  api-gw     corr=7f3a9c  POST /payments  start
14:05:02.131Z  zosconn    corr=7f3a9c  CICS PAY1 tran=PAYA
14:05:02.140Z  cics       corr=7f3a9c  PAYA task 51234 start
14:05:04.902Z  cics       corr=7f3a9c  PAYA task 51234 end  2.762s
14:05:04.915Z  api-gw     corr=7f3a9c  200  total 2.797s

Here the gateway and z/OS Connect added only milliseconds; nearly all the time was inside the CICS task. The next question is whether that time was CPU, DB2 waits or lock contention, which the CICS and DB2 monitors can answer.