Conversion pitfalls, quality, governance and the system of record
Mainframe data uses EBCDIC text and numeric formats such as packed decimal that other platforms do not understand. Converting it wrongly corrupts data silently. Once data is available elsewhere, quality checks, governance, security and a clear system of record keep the copies trustworthy.
Why conversion goes wrong
A mainframe record is a sequence of bytes whose meaning is given by its copybook. Text fields use EBCDIC, but numeric fields may be packed decimal, zoned decimal or binary. A tool that converts the whole record from EBCDIC to ASCII as if it were text will translate the numeric bytes too, and destroy them. Conversion must be field by field, driven by the copybook.
PIC X(5)PIC S9(5)PIC S9(5) COMP-3PIC S9(4) COMPCOMP-1 / COMP-2Classic symptoms
| Symptom on the target | Likely cause |
|---|---|
| Last digit of an amount shows as a letter, for example 12C or 12L | Zoned decimal sign treated as text (C for positive, D for negative in the zone) |
| Amounts are garbage or huge numbers | Packed or binary fields were translated as characters |
| A few symbols such as square brackets are wrong | Wrong EBCDIC code page used, for example 037 instead of 1047 |
| Lines shifted or records merged | Variable-length records or binary data containing line-end byte values |
| Records sort in a different order | EBCDIC and ASCII collating sequences differ |
PIC S9(5) COMP-3 value +12345 bytes 12 34 5C PIC S9(5) COMP-3 value -12345 bytes 12 34 5D PIC 9(5) COMP-3 value 12345 bytes 12 34 5F
A PIC S9(5) COMP-3 field contains hex 00 12 3D. What value is it?
Show a hint
Read the digits, then the last half-byte is the sign: C positive, D negative.
Show the solution
Digits 00123 with sign D, so the value is -123.
Data quality
Old files often contain values the programs tolerate but a database will reject: dates such as 00000000 or 99999999 used as markers, spaces in numeric fields, low-values (hex 00) as defaults, and codes no longer documented. Profile the data before migration, agree rules for each oddity with the business, and reconcile after every load: record counts, sums of amount fields and checks on key ranges should match source and target.
Governance and security
- Ownership: every data set has a named business owner who decides who may use it.
- Lineage: record where each target field came from and how it was transformed, so a wrong figure in a report can be traced. This is data lineage.
- Access control: on z/OS, RACF or an equivalent product protects the source; the target platform needs equal controls. Data copied off the mainframe is often protected less well.
- Sensitive data: mask or tokenise personal and card data before it reaches analytics or test environments (data masking), and encrypt in transit.
- Retention: copies must follow the same retention and deletion rules as the source, including legal requirements to erase personal data.
One system of record
For each piece of data there should be exactly one system of record: the place where it is created and changed, and whose value wins in a dispute. Everything else is a copy. During a migration the system of record can move, but only at a planned, single moment.
Common mistakes
Character conversion destroys packed, zoned and binary fields. Convert field by field using the copybook.
Different EBCDIC code pages differ for some characters. Confirm which one the data uses before converting.
If more than one place accepts updates, data diverges. Name one system of record and make copies read-only.
What you will see at work
- Developers supply copybooks and explain field formats to the teams building data pipelines.
- After each load, someone reconciles record counts and amount totals between source and target.
- Security and data governance teams approve which fields may leave the mainframe and in what masked form.
Key terms
Check your understanding.
Take this lesson's quiz and save your progress. Free.