DFDL: Describing Binary and Text Data Formats

DFDL describes existing data

DFDL is designed for formats that already exist: fixed-width records, delimited text, binary telemetry, packed decimals and mixtures of those structures. Instead of inventing another serialization, a DFDL schema annotates XML Schema components with representation details. A processor can then parse physical bytes into an XML information set or unparse that information set back into the governed format. The schema becomes an executable description of layout, length, encoding and separators.

The schema and data-format layers

The XSD portion describes logical elements, sequences, choices and datatypes. DFDL annotations describe how those logical values appear on the wire. Separating the layers matters because two physical encodings can represent similar business information, while one physical record can contain padding and framing that applications should not treat as business data. Good models name the logical concepts first, then attach the narrowest representation rules needed to locate and decode them.

DFDL compared with ANTLR

ANTLR generates parsers from grammars intended primarily for programming languages and other token-oriented text. DFDL focuses on data interchange and provides vocabulary for lengths, delimiters, encodings, binary numbers, alignment, nil values and bidirectional unparsing. ANTLR offers direct control over grammar actions and parse trees; DFDL offers a declarative data model that schema-aware tools can inspect. Neither replaces the other universally: select according to the physical format and required output model.

Length, delimiter and encoding decisions

Every variable field needs an unambiguous boundary. DFDL can derive a length from an expression, consume a terminator or infer boundaries from enclosing structure. Delimiters require escaping rules when values may contain the delimiter itself. Character data also needs a declared encoding and error policy. These decisions should be tested with empty values, multibyte characters, maximum lengths and malformed records rather than demonstrated only with one clean sample.

Binary data and alignment

Binary formats introduce byte order, bit order, signed representation, floating-point conventions and alignment. Padding bytes may be meaningful at one layer and invisible at another. Write small fixtures whose byte sequences are independently known, and inspect both the parsed values and the exact bytes produced by unparsing. Round-trip success alone is insufficient because the same incorrect interpretation can be applied symmetrically in both directions.

Schema composition and reuse

Reusable format fragments can model common headers, timestamps and record envelopes, but hidden annotation defaults make reused components difficult to reason about. Keep format properties close to the declarations they govern, document inherited defaults and avoid long reference chains for fields that differ subtly between message versions. A catalog-controlled schema set also prevents production parsing from depending on network retrieval of imported resources.

Diagnostics and security

A parser should report the byte or character position, active schema component and failed representation rule. Bound recursion, lengths and allocations before accepting untrusted data; a declared size must not authorize an unlimited buffer. Disable uncontrolled external resource resolution and fuzz the parser with truncated, oversized and inconsistent inputs. DFDL describes a format, but deployment policy still determines whether parsing that format is safe.

A practical evaluation sequence

Begin with three representative records: ordinary, boundary-heavy and intentionally invalid. Model only the stable outer structure, verify parsing against an independent decoder, then add optional and repeated fields. Test unparsing only after parsing expectations are explicit. Before adoption, confirm processor support for every annotation used, measure throughput on realistic record sizes and record which schema release governs each stored or exchanged dataset.

Questions for a design review

Use this reference to make a review decision, not merely to recognize terminology. Record the concrete document, schema, processor or consumer being discussed; the language and processor versions; and the behavior that must remain compatible. A useful review produces fixtures and an owner for every unresolved assumption.

• What executable example or test demonstrates the intended behavior for dfdl describes existing data?
• What executable example or test demonstrates the intended behavior for the schema and data-format layers?
• What executable example or test demonstrates the intended behavior for dfdl compared with antlr?
• What executable example or test demonstrates the intended behavior for length, delimiter and encoding decisions?
• What executable example or test demonstrates the intended behavior for binary data and alignment?
• What executable example or test demonstrates the intended behavior for schema composition and reuse?

Include at least one ordinary case, one boundary case and one deliberately invalid or unsupported case. Check the result in the actual production toolchain, because parsers, validators, code generators and reasoners do not all implement the same optional features. Store the selected contract version with the test result, then repeat the review when a dependency, namespace, profile or public declaration changes.

Continue with a working reference

Data-contract layers · Grammars and parsers