DFSORT and ICETOOL in production

The sort product does most of the data manipulation in a typical batch schedule: sorting, filtering, reformatting, removing duplicates, counting and splitting. ICETOOL wraps several of these operations into one step with readable commands.

One product, many jobs

DFSORT is IBM's sort product. Many sites run Syncsort MFX from Precisely instead, which accepts largely the same control statements. Either way, JCL usually calls PGM=SORT and the site's sort product answers. Feature details and messages differ between products and releases, so check which one your site has before relying on a less common option.

How one DFSORT step processes records

SORTINinput records

INCLUDE / OMITkeep or drop

INRECreshape before sort

SORT / MERGE / COPYorder records

SUMcollapse equal keys

OUTREC / OUTFILshape and route output

The statements you will read most

Common DFSORT control statementsWhat it means

SORT FIELDS=(1,10,CH,A)

Sort on 10 bytes from position 1, character, ascending

OPTION COPY

No sort, just copy (with any filtering or reformatting)

INCLUDE COND=(15,2,CH,EQ,C'NY')

Keep only records with NY in positions 15-16

SUM FIELDS=NONE

Keep one record per sort key, removing duplicates

SUM FIELDS=(30,5,PD)

Add a packed-decimal amount across records with equal keys

OUTREC BUILD=(1,10,30,5)

Output only these fields, in this order

OUTFIL FNAMES=OUT1,INCLUDE=...

Write a subset of records to the DD named OUT1

Field positions and formats must match the record layout exactly. The most common sort bug is a key described as CH when it is packed decimal (PD) or binary (BI), or a position off by one because the copybook was misread. For variable-length records, remember the first four bytes are the record descriptor word, so data starts at position 5.

Splitting one file into several

Split by region in one pass (illustrative)

//SPLIT    EXEC PGM=SORT
//SYSOUT   DD SYSOUT=*
//SORTIN   DD DSN=PROD.SALES.DAILY,DISP=SHR
//NORTH    DD DSN=PROD.SALES.NORTH,DISP=(NEW,CATLG,DELETE),
//            SPACE=(CYL,(10,5),RLSE)
//SOUTH    DD DSN=PROD.SALES.SOUTH,DISP=(NEW,CATLG,DELETE),
//            SPACE=(CYL,(10,5),RLSE)
//SYSIN    DD *
  OPTION COPY
  OUTFIL FNAMES=NORTH,INCLUDE=(20,1,CH,EQ,C'N')
  OUTFIL FNAMES=SOUTH,INCLUDE=(20,1,CH,EQ,C'S')
/*

ICETOOL: several operations in one step

ICETOOL is a multi-purpose utility that comes with DFSORT. You list operators in TOOLIN; each names its input and output DDs. Messages go to TOOLMSG (ICETOOL's own) and DFSMSG (the underlying sort). Extra sort statements for an operator go in a DD named after a prefix you choose plus CNTL.

Operator	What it does
COPY / SORT	Copy or sort a file, optionally with control statements from a CNTL DD
COUNT	Count records, and optionally set a return code if the count meets a condition
SELECT	Keep records by key: FIRST, LAST, NODUPS (keys appearing once), ALLDUPS, and others
OCCUR	Report how often each key value occurs
STATS / DISPLAY	Print simple statistics or a formatted listing of fields
SPLICE	Join fields from records with matching keys

Fail-safe check and duplicate report (illustrative)

//CHECK    EXEC PGM=ICETOOL
//TOOLMSG  DD SYSOUT=*
//DFSMSG   DD SYSOUT=*
//IN       DD DSN=PROD.PAY.TRANS,DISP=SHR
//DUPS     DD DSN=PROD.PAY.TRANS.DUPKEYS,DISP=(NEW,CATLG,DELETE),
//            SPACE=(CYL,(1,1),RLSE)
//TOOLIN   DD *
  COUNT FROM(IN) EMPTY RC4
  SELECT FROM(IN) TO(DUPS) ON(1,12,CH) ALLDUPS
/*

COUNT ... EMPTY RC4 sets return code 4 when the input has no records, so the next step can skip processing instead of posting an empty day. SELECT ... ALLDUPS writes every record whose 12-byte key appears more than once, which is a quick way to find duplicate transactions before they reach a database load.

Return codes and failures

DFSORT ends RC 0 on success and RC 16 when it fails; a few options can produce RC 4 for warnings. The reason is an ICE message in SYSOUT.
ICETOOL's return code is the highest from its operators; RC 12 usually means an operator failed, and TOOLMSG says which one.
Sort work space: large sorts need work space, either dynamically allocated by the product or from SORTWKnn DDs. Running out is a classic failure on peak days such as month end.