Getting mainframe data into enterprise platforms

Most organisations run one observability or log platform for everything, such as Splunk, Elastic or Grafana. Mainframe data can feed it through streaming collectors, so one dashboard can show the whole service. Doing it well means choosing data carefully, handling formats, and respecting cost and sensitivity.

Why bring it together

A customer's payment might pass through a mobile app, an API gateway, cloud services, MQ, CICS and DB2. If each team looks only at its own console, an incident turns into a meeting where everyone shows a green screen. Putting mainframe data next to distributed data lets one team see where the time went.

How data moves

A typical streaming pipeline

SourcesSMF, SYSLOG, OPERLOG, logs

Collector on z/OSreads and filters

Transformdecode, convert, enrich

TransportTCP, Kafka, HTTP

PlatformSplunk, Elastic, Grafana stack

A collector runs on z/OS, often as a started task. It reads SMF records, log streams or files, filters them, and sends them off-platform. Examples include IBM Z Common Data Provider, Syncsort/Precisely Ironstream, and data forwarding built into product monitors. Some sites use vendor connectors to Apache Kafka as a hub so many consumers can read the same stream.

What has to be transformed

Binary records. SMF records are binary structures with packed fields, offsets and sections. They must be decoded by something that knows the record layout for that release.
Character encoding. Text is EBCDIC and must be converted, usually to UTF-8.
Timestamps. SMF and logs may hold local time or UTC; mixing them makes a timeline wrong by hours. Normalise to UTC and keep the source system name.
Volume. Some record types, such as per-transaction CICS or DB2 records, can produce millions of records an hour. Summarise or select before sending.

Choosing a platform role

Platform	Typical role
Splunk	Log and event search, security analytics, dashboards; widely used for SMF and SYSLOG
Elastic (Elasticsearch, Kibana)	Search and dashboards for logs and metrics, often self-managed
Prometheus and Grafana	Time-series metrics and dashboards; Grafana can also chart data from other stores
Vendor APM and observability suites	Traces, service maps and AI-assisted alerting across tiers

None of these is the right answer everywhere. Pick based on what the rest of the organisation already uses, so mainframe data lands where the incident responders already look.

OpenTelemetry at concept level

OpenTelemetry (OTel) is an open-source, vendor-neutral standard from the Cloud Native Computing Foundation for producing and shipping metrics, logs and traces. Its value is that instrumented software can send data to any compatible back end. IBM and other mainframe vendors have been adding OpenTelemetry support to some products; what is supported depends on the product and version, so check before you design around it.

Cost, security and ownership

Licensing and ingest cost. Many platforms charge by volume. Sending every SMF record can cost more than the insight is worth.
MIPS cost. Collection and conversion on z/OS uses CPU. Offload-eligible work may run on zIIP processors depending on the product.
Sensitive data. Records can contain user IDs, account numbers in transaction data, or dataset names that reveal business information. Apply masking and access control in the platform too.
Ownership. Decide who owns the pipeline. When it breaks silently, dashboards show 'all quiet' rather than an error.