feat(spec): key folding & path expansion (closes #4, #5)

Author: johannschopplich

CHANGELOG.md

@@ -5,6 +5,20 @@ All notable changes to the TOON specification will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.5] - 2025-11-08
+
+### Added
+
+- Optional key folding for encoders: `keyFolding="safe"` mode with `flattenDepth` control to collapse single-key object chains into dotted-path notation (§13.4)
+- Optional path expansion for decoders: `expandPaths="safe"` mode to split dotted keys into nested objects, with conflict resolution tied to `strict` option (§13.4, §14.5)
+- IdentifierSegment terminology and path separator definition (fixed to `"."` in v1.5) (§1.9)
+- Deep-merge semantics for path expansion: recursive merge for objects, error on conflict when `strict=true`, last-write-wins (LWW) when `strict=false` (§13.4)
+
+### Changed
+
+- Both new features default to OFF and are fully backward-compatible
+- Safe-mode folding requires IdentifierSegment validation, collision avoidance, and no quoting
+
 ## [1.4] - 2025-11-05
 
 ### Changed

README.md

@@ -1,6 +1,6 @@
 # TOON Format Specification
 
-[![SPEC v1.4](https://img.shields.io/badge/spec-v1.4-lightgrey)](./SPEC.md)
+[![SPEC v1.5](https://img.shields.io/badge/spec-v1.5-lightgrey)](./SPEC.md)
 [![Tests](https://img.shields.io/badge/tests-323-green)](./tests/fixtures/)
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](./LICENSE)
 
@@ -10,12 +10,24 @@ This repository contains the official specification for **Token-Oriented Object
 
 [→ Read the full specification (SPEC.md)](./SPEC.md)
 
-- **Version:** 1.4 (2025-11-05)
+- **Version:** 1.5 (2025-11-10)
 - **Status:** Working Draft
 - **License:** MIT
 
 The specification includes complete grammar (ABNF), encoding rules, validation requirements, and conformance criteria.
 
+### New in v1.5
+
+- **Key Folding** (encode): Collapse nested single-key objects into compact dotted paths
+  - `{"a": {"b": {"c": 1}}}` → `a.b.c: 1`
+  - Opt-in via `keyFolding="safe"` with `flattenDepth` control
+- **Path Expansion** (decode): Expand dotted keys back to nested objects
+  - `a.b.c: 1` → `{"a": {"b": {"c": 1}}}`
+  - Opt-in via `expandPaths="safe"` with deep-merge semantics
+
+> [!NOTE]
+> Both features are opt-in to maintain backward compatibility.
+
 ## What is TOON?
 
 **Token-Oriented Object Notation** is a compact, human-readable serialization format designed for passing structured data to Large Language Models with significantly reduced token usage. It's intended for LLM input, not output.

SPEC.md

@@ -2,9 +2,9 @@
 
 ## Token-Oriented Object Notation
 
-**Version:** 1.4
+**Version:** 1.5
 
-**Date:** 2025-11-05
+**Date:** 2025-11-10
 
 **Status:** Working Draft
 
@@ -189,6 +189,12 @@ Implementations that fail to conform to any MUST or REQUIRED level requirement a
 - Regular expressions appear in slash-delimited form.
 - ABNF snippets follow RFC 5234; HTAB means the U+0009 character.
 
+### 1.9 Key Folding and Path Expansion Terms
+
+- IdentifierSegment: A key segment eligible for safe folding and expansion, matching the pattern `^[A-Za-z_][A-Za-z0-9_]*$` (contains only letters, digits, and underscores; does not start with a digit; does not contain dots).
+- Path separator: The character used to join/split key segments during folding and expansion. Fixed to `"."` (U+002E, FULL STOP) in v1.5.
+- Note: Unquoted keys in TOON remain permissive per §7.3 (`^[A-Za-z_][A-Za-z0-9_.]*$`, allowing dots). IdentifierSegment is a stricter pattern used only for safe folding and expansion eligibility checks.
+
 ## 2. Data Model
 
 - TOON models data as:
@@ -351,6 +357,8 @@ Decoding requirements:
 - If a fields segment occurs between the bracket and the colon, parse field names using the active delimiter; quoted names MUST be unescaped per Section 7.1.
 - A colon MUST follow the bracket and optional fields; missing colon MUST error.
 
+Note: Key folding (§13.4) affects only the key prefix in headers. The header grammar remains unchanged. Example: `data.meta.items[2]{id,name}:` is a valid header with a folded key prefix `data.meta.items`, followed by a standard bracket segment, field list, and colon. Parsing treats folded keys as literal keys; see §13.4 for optional path expansion.
+
 ## 7. Strings and Keys
 
 ### 7.1 Escaping (Encoding and Decoding)
@@ -393,6 +401,8 @@ Object keys and tabular field names:
 
 Keys requiring quoting per the above rules MUST be quoted in all contexts, including array headers (e.g., "my-key"[N]:).
 
+Encoders MAY perform key folding when enabled (see §13.4 for complete folding rules and requirements).
+
 ### 7.4 Decoding Rules for Strings and Keys (Decoding)
 
 - Quoted strings and keys MUST be unescaped per Section 7.1; any other escape MUST error. Quoted primitives remain strings.
@@ -409,6 +419,7 @@ Keys requiring quoting per the above rules MUST be quoted in all contexts, inclu
   - Nested or empty objects: key: on its own line. If non-empty, nested fields appear at depth +1.
   - Key order: Implementations MUST preserve encounter order when emitting fields.
   - An empty object at the root yields an empty document (no lines).
+- Dotted keys (e.g., `user.name`) are valid literal keys in TOON. Decoders MUST treat them as single literal keys unless path expansion is explicitly enabled (see §13.4). This preserves backward compatibility and allows safe opt-in expansion behavior.
 - Decoding:
   - A line "key:" with nothing after the colon at depth d opens an object; subsequent lines at depth > d belong to that object until the depth decreases to ≤ d.
   - Lines "key: value" at the same depth are sibling fields.
@@ -582,12 +593,89 @@ Options:
   - indent (default: 2 spaces)
   - delimiter (document delimiter; default: comma; alternatives: tab, pipe)
   - lengthMarker (default: disabled)
+  - keyFolding (default: `"off"`; alternatives: `"safe"`)
+  - flattenDepth (default: Infinity when keyFolding is `"safe"`; non-negative integer ≥ 0; values 0 or 1 have no practical folding effect)
 - Decoder options:
   - indent (default: 2 spaces)
-  - strict (default: true)
+  - strict (default: `true`)
+  - expandPaths (default: `"off"`; alternatives: `"safe"`)
 
 Strict-mode errors are enumerated in §14; validators MAY add informative diagnostics for style and encoding invariants.
 
+### 13.4 Key Folding and Path Expansion
+
+Key folding and path expansion are optional transformations for compact dotted-path notation. Both default to `"off"`.
+
+#### Encoder: Key Folding
+
+Key folding allows encoders to collapse chains of single-key objects into dotted-path notation, reducing verbosity for deeply nested structures.
+
+Mode: `"off"` | `"safe"` (default: `"off"`)
+- `"off"`: No folding is performed. All objects are encoded with standard nesting.
+- `"safe"`: Fold eligible chains according to the rules below.
+
+flattenDepth: The maximum number of segments from K0 to include in the folded path (default: Infinity when keyFolding is `"safe"`; values less than 2 have no practical effect).
+- A value of 2 folds only two-segment chains: `{a: {b: val}}` → `a.b: val`.
+- A value of Infinity folds entire eligible chains: `{a: {b: {c: val}}}` → `a.b.c: val`.
+
+Foldable chain: A chain K0 → K1 → ... → Kn is foldable when:
+- Each Ki (where i = 0 to n−1) is an object with exactly one key Ki+1.
+- The chain stops at the first non-single-key object or when encountering a leaf value.
+- Arrays are not considered single-key objects; a chain stops at arrays.
+- The leaf value at Kn is either a primitive, an array, or an empty object.
+
+Safe mode requirements (all MUST hold for a chain to be folded):
+1. All folded segments K0 through K(d−1) (where d = min(chain length, flattenDepth)) MUST be IdentifierSegments (§1.9): matching `^[A-Za-z_][A-Za-z0-9_]*$`.
+2. No segment may contain the path separator (`.` in v1.5).
+3. The resulting folded key string MUST NOT equal any existing sibling literal key at the same object depth (collision avoidance).
+4. If any segment would require quoting per §7.3, the chain MUST NOT be folded.
+
+Folding process:
+- For a foldable chain of length n, determine d = min(n, flattenDepth).
+- Fold segments K0 through K(d−1) into a single key: `K0.K1.....K(d−1)`.
+- If d < n, emit the remaining structure (Kd through Kn) as normal nested objects.
+- The leaf value at Kn is encoded normally (primitive, array, or empty object).
+
+Examples:
+- `{a: {b: {c: 1}}}` with safe mode, depth=Infinity → `a.b.c: 1`
+- `{a: {b: {c: {d: 1}}}}` with safe mode, depth=2 → produces `a.b:` followed by nested `c:` and `d: 1` at appropriate depths
+- `{data: {"full-name": {x: 1}}}` → safe mode skips (segment `"full-name"` requires quoting); emits standard nested structure
+
+#### Decoder: Path Expansion
+
+Path expansion allows decoders to split dotted keys into nested object structures, enabling round-trip compatibility with folded encodings.
+
+Mode: `"off"` | `"safe"` (default: `"off"`)
+- `"off"`: Dotted keys are treated as literal keys. No expansion is performed.
+- `"safe"`: Expand eligible dotted keys according to the rules below.
+
+Safe mode behavior:
+- Any key containing the path separator (`.`) is considered for expansion.
+- Split the key into segments at each occurrence of `.`.
+- Only expand when ALL resulting segments are IdentifierSegments (§1.9) and none contain `.` after splitting.
+- Keys that do not meet the expansion criteria remain as literal keys.
+
+Deep merge semantics:
+When multiple expanded keys construct overlapping object paths, the decoder MUST merge them recursively:
+- Object + Object: Deep merge recursively (recurse into nested keys and apply these rules).
+- Object + Non-object (array or primitive): This is a conflict. Apply conflict resolution policy.
+- Array + Array or Primitive + Primitive: This is a conflict. Apply conflict resolution policy. Arrays are never merged element-wise.
+- Key ordering: During expansion, newly created keys are inserted in encounter order (the order they appear in the document). When merging creates nested keys, keys from later lines are appended after existing keys at the same depth. This ensures deterministic, predictable key order in the resulting object.
+
+Conflict resolution:
+- Conflict definition: A conflict occurs when expansion requires an object at a given path but finds a non-object value (array or primitive), or vice versa. A conflict also occurs when a final leaf key already exists with a non-object value that must be overwritten.
+- `strict=true` (default): Decoders MUST error on any conflict. This ensures data integrity and catches structural inconsistencies.
+- `strict=false`: Last-write-wins (LWW) conflict resolution: keys appearing later in document order (encounter order during parsing) overwrite earlier values. This provides deterministic behavior for lenient parsing.
+
+Application order: Path expansion is applied AFTER all base parsing rules (§4–12) have been applied and BEFORE the final decoded value is returned to the caller. Structural validations enumerated in §14 (strict-mode errors for array counts, indentation, etc.) operate on the pre-expanded structure and remain unaffected by expansion.
+
+Examples:
+- Input: `data.meta.items[2]: a,b` with `expandPaths="safe"` → Output: `{"data": {"meta": {"items": ["a", "b"]}}}`
+- Input: `user.name: Ada` with `expandPaths="off"` → Output: `{"user.name": "Ada"}`
+- Input: `a.b.c: 1` and `a.b.d: 2` and `a.e: 3` with `expandPaths="safe"` → Output: `{"a": {"b": {"c": 1, "d": 2}, "e": 3}}` (deep merge)
+- Input: `a.b: 1` then `a: 2` with `expandPaths="safe"` and `strict=true` → Error: "Expansion conflict at path 'a' (object vs primitive)"
+- Input: `a.b: 1` then `a: 2` with `expandPaths="safe"` and `strict=false` → Output: `{"a": 2}` (LWW)
+
 ### 13.1 Encoder Conformance Checklist
 
 Conforming encoders MUST:
@@ -601,6 +689,8 @@ Conforming encoders MUST:
 - [ ] Convert -0 to 0 (§2)
 - [ ] Convert NaN/±Infinity to null (§3)
 - [ ] Emit no trailing spaces or trailing newline (§12)
+- [ ] When `keyFolding="safe"`, folding MUST comply with §13.4 (IdentifierSegment validation, no separator in segments, collision avoidance, no quoting required)
+- [ ] When `flattenDepth` is set, folding MUST stop at the configured segment count (§13.4)
 
 ### 13.2 Decoder Conformance Checklist
 
@@ -609,9 +699,12 @@ Conforming decoders MUST:
 - [ ] Split inline arrays and tabular rows using active delimiter only (§11)
 - [ ] Unescape quoted strings with only valid escapes (§7.1)
 - [ ] Type unquoted primitives: true/false/null → booleans/null, numeric → number, else → string (§4)
-- [ ] Enforce strict-mode rules when strict=true (§14)
+- [ ] Enforce strict-mode rules when `strict=true` (§14)
 - [ ] Accept and ignore optional # length marker (§6)
 - [ ] Preserve array order and object key order (§2)
+- [ ] When `expandPaths="safe"`, expansion MUST follow §13.4 (IdentifierSegment-only segments, deep merge, conflict rules)
+- [ ] When `expandPaths="safe"` with `strict=true`, MUST error on expansion conflicts per §14.5
+- [ ] When `expandPaths="safe"` with `strict=false`, apply LWW conflict resolution (§13.4)
 
 ### 13.3 Validator Conformance Checklist
 
@@ -650,7 +743,17 @@ When strict mode is enabled (default), decoders MUST error on the following cond
 
 For root-form rules, including handling of empty documents, see §5.
 
-### 14.5 Recommended Error Messages and Validator Diagnostics (Informative)
+### 14.5 Path Expansion Conflicts
+
+When `expandPaths="safe"` is enabled:
+- With `strict=true` (default): Decoders MUST error on any expansion conflict.
+- With `strict=false`: Decoders MUST apply deterministic last-write-wins (LWW) resolution in document order. Implementations MUST resolve conflicts silently and MUST NOT emit diagnostics during normal decode operations.
+
+See §13.4 for complete conflict definitions, deep-merge semantics, and examples.
+
+Note (informative): Implementations MAY expose conflict diagnostics via out-of-band mechanisms (e.g., debug hooks, verbose CLI flags, or separate validation APIs), but such facilities are non-normative and MUST NOT affect default decode behavior or output.
+
+### 14.6 Recommended Error Messages and Validator Diagnostics (Informative)
 
 Validators SHOULD additionally report:
 - Trailing spaces, trailing newlines (encoding invariants).
@@ -972,6 +1075,74 @@ Quoted keys with arrays (keys requiring quoting per Section 7.3):
   - id: 2
 ```
 
+Key folding and path expansion (v1.5+):
+
+Encoding - basic folding (safe mode, depth=Infinity):
+
+Input: `{"a": {"b": {"c": 1}}}`
+```
+a.b.c: 1
+```
+
+Encoding - folding with inline array:
+
+Input: `{"data": {"meta": {"items": ["x", "y"]}}}`
+```
+data.meta.items[2]: x,y
+```
+
+Encoding - folding with tabular array:
+
+Input: `{"a": {"b": {"items": [{"id": 1, "name": "A"}, {"id": 2, "name": "B"}]}}}`
+```
+a.b.items[2]{id,name}:
+  1,A
+  2,B
+```
+
+Encoding - partial folding (flattenDepth=2):
+
+Input: `{"a": {"b": {"c": {"d": 1}}}}`
+```
+a.b:
+  c:
+    d: 1
+```
+
+Decoding - basic expansion (safe mode round-trip):
+
+Input: `data.meta.items[2]: a,b` with options `{expandPaths: "safe"}`
+
+Output: `{"data": {"meta": {"items": ["a", "b"]}}}`
+
+Decoding - deep merge (multiple expanded keys):
+
+Input with options `{expandPaths: "safe"}`:
+```
+a.b.c: 1
+a.b.d: 2
+a.e: 3
+```
+Output: `{"a": {"b": {"c": 1, "d": 2}, "e": 3}}`
+
+Decoding - conflict error (strict=true, default):
+
+Input with options `{expandPaths: "safe", strict: true}`:
+```
+a.b: 1
+a: 2
+```
+Result: Error - "Expansion conflict at path 'a' (object vs primitive)"
+
+Decoding - conflict LWW (strict=false):
+
+Input with options `{expandPaths: "safe", strict: false}`:
+```
+a.b: 1
+a: 2
+```
+Output: `{"a": 2}`
+
 ## Appendix B: Parsing Helpers (Informative)
 
 These sketches illustrate structure and common decoding helpers. They are informative; normative behavior is defined in Sections 4–12 and 14.
@@ -1060,6 +1231,15 @@ Note: Host-type normalization tests (e.g., BigInt, Date, Set, Map) are language-
 
 ## Appendix D: Document Changelog (Informative)
 
+### v1.5 (2025-11-08)
+
+- Added optional key folding for encoders: `keyFolding='safe'` mode with `flattenDepth` control (§13.4).
+- Added optional path expansion for decoders: `expandPaths='safe'` mode with conflict resolution tied to existing `strict` option (§13.4).
+- Defined safe-mode requirements for folding: IdentifierSegment validation, no path separator in segments, collision avoidance, no quoting required (§7.3, §13.4).
+- Specified deep-merge semantics for expansion: recursive merge for objects; conflict policy (error in strict mode, LWW when strict=false) for non-objects (§13.4).
+- Added strict-mode error category for path expansion conflicts (§14.5).
+- Both features default to OFF; fully backward-compatible.
+
 ### v1.4 (2025-11-05)
 
 - Removed JavaScript-specific normalization details; replaced with language-agnostic requirements (Section 3).
@@ -1249,6 +1429,7 @@ For a detailed version history, see Appendix D.
 
 - Backward-compatible evolutions SHOULD preserve current headers, quoting rules, and indentation semantics.
 - Reserved/structural characters (colon, brackets, braces, hyphen) MUST retain current meanings.
+- The path separator (see §1.9) is fixed to `"."` in v1.5; future versions MAY make this configurable.
 - Future work (non-normative): schemas, comments/annotations, additional delimiter profiles, optional \uXXXX escapes (if added, must be precisely defined).
 
 ## 21. Intellectual Property Considerations

VERSIONING.md

@@ -13,7 +13,7 @@ The TOON specification follows [Semantic Versioning](https://semver.org/) with a
 - **MAJOR version** - Incremented for breaking changes that are incompatible with previous versions
 - **MINOR version** - Incremented for backward-compatible additions, clarifications, or non-breaking changes
 
-**Example:** Moving from v1.3 to v1.4 means your implementation keeps working. Moving from v1.3 to v2.0 means you'll likely need to update your code.
+**Example:** Moving from v1.5 to v1.6 means your implementation keeps working. Moving from v1.5 to v2.0 means you'll likely need to update your code.
 
 ## What Constitutes a Breaking Change