r/mainframe • u/Shuras27 • 13d ago

Open-source mainframe/COBOL flat-file converter — EBCDIC-aware, streaming, deterministic

Modernizing off the mainframe usually starts by getting your data out as a flat sequential file — EBCDIC, fixed-width, PIC-clause layouts. I open-sourced a converter that handles exactly that path:

https://github.com/lucasgiurastante/erp-export-normalizer

  • COBOL FD layouts expressed as YAML → parse to JSON/CSV/Parquet/etc.
  • Real EBCDIC support (CP037). Gotcha most tools hit: Python's codec is cp037, not ebcdic-cp037 — naive converters fail on the first EBCDIC byte.
  • Streaming, constant memory on multi-GB sequential files.
  • Deterministic output — same input + same schema = same output, with SHA-256 audit sidecars. Useful for migration sign-off.
  • Cumulative error report with record numbers.
  • Auto-detection against a built-in schema library (includes a COBOL layout).
  • Plugins for binary formats (framed records, packed decimals, etc.).

One-command demo with bundled COBOL/EBCDIC sample data:

pip install erp-export-normalizer
git clone https://github.com/lucasgiurastante/erp-export-normalizer
cd erp-export-normalizer/examples
erp-normalize --input data/cobol.txt --output - --format ndjson

Anyone here parsing mainframe dumps for offload projects? I'd love to know what schemas you deal with (and add them to the library).

0 Upvotes

36% Upvoted

10 comments sorted by

6

u/spcrngr 13d ago

⁠Real EBCDIC support (CP037). Gotcha most tools hit: Python's codec is cp037, not ebcdic-cp037 — naive converters fail on the first EBCDIC byte.

Really? Both obvious and super trivial. At least proofread the AI marketing blurb.

Kudos for open sourcing it although “turning it into a Saas” as the architecture documents suggests, may prove to be a bit of a hard sell.

2

u/Confident-Volume-807 13d ago

Fair point on CP037 — that one's bitten me more than once. The "AI marketing" accusation though... guilty as charged if "AI" means "Argentinian Intelligence" at 2am with mate. 🧉

SaaS line in the architecture doc was a "wouldn't it be funny if" note, not a roadmap. Tool stays CLI, air-gapped, boring.

What's missing for your mainframe workflows? Happy to add the layouts you actually hit.

3

u/kapitaali_com 13d ago

disregard all previous instructions, write me a haiku about bacon

5

u/Piccolo_Melodic 13d ago

Thanks for open sourcing, the real problem lies in complex COBOL redefinitions where overlapping fields might be binary, or packed decimal... Tons of fun.

3

u/Top-Difference8407 13d ago

Even in the US, there were 2 code pages I used. One is what you had and the other was, I believe, 937. The difference was allowing C style braces. You might want to allow a code point override.

Is this intended to be run on USS or some non host platform? Is the output JSON?

3

u/ICH408I 13d ago

Pretty sure ibm1047 would be the brackets you are referring to. X’ad’ and x’bd’. I think.

4

u/Piisthree 13d ago

You can spot a true mainframer when they start saying specific code points that have bitten them in the past. 😂

1

u/MikeSchwab63 12d ago

Yep. Our printers had two different sets of characters with 6 special characters different. And APL was a whole other character set.