Skip to content

pack-specification.md

Latest commit

 

History

History
629 lines (459 loc) · 15.7 KB

pack-specification.md

File metadata and controls

629 lines (459 loc) · 15.7 KB

senax-encoder Pack Format Specification

Version: 1.0
Date: 2025
Status: Draft

Table of Contents

  1. Overview
  2. Format Basics
  3. Data Type Specifications
  4. Struct and Enum Packing
  5. Optimization Features
  6. Implementation Notes

1. Overview

The senax-encoder pack format is designed for maximum compactness and performance when schema evolution is not required. Unlike the encode format, pack does not include field IDs for struct fields, resulting in significantly smaller binary size but requiring strict field order compatibility.

Key Design Principles

  • Maximum Compactness: No field IDs for struct fields, minimal overhead
  • High Performance: Direct binary representation without metadata
  • Order Dependent: Field order must match exactly between pack/unpack
  • Little Endian: Consistent byte order across platforms

Pack vs Encode Comparison

Feature Pack Format Encode Format
Binary Size Smaller Larger
Schema Evolution Not Supported Full Support
Field Order Must Match Independent
Performance Faster Moderate
Use Case High-performance, stable schemas Evolving APIs, long-term storage

2. Format Basics

2.1 Magic Numbers

The senax-encoder library uses magic numbers to distinguish between different serialization formats when using convenience functions:

Encode Format: 0xA55A (2 bytes, little-endian)

  • Used by encode() and encode_to() functions
  • Supports schema evolution with field IDs and type tags
  • Binary starts with: [0x5A, 0xA5]

Pack Format: 0xDADA (2 bytes, little-endian)

  • Used by pack() and pack_to() functions
  • Compact format without schema evolution support
  • Binary starts with: [0xDA, 0xDA]

Important Notes:

  • Magic numbers are only added by convenience functions (pack(), encode(), etc.)
  • Direct trait method calls (Packer::pack, Encoder::encode) do not include magic numbers
  • The unpack() and decode() functions validate and consume the magic numbers
  • This specification describes the format after magic number (raw pack format)

2.2 Byte Order

All multi-byte integers are encoded in little-endian format.

2.3 Basic Structure

Pack format uses direct binary representation with minimal tagging:

Primitives:

[DATA:variable]  // Direct encoding without type tags (except for optimizations)

Containers:

[LENGTH:variable] [DATA:variable]  // Length-prefixed for collections

Structs:

[HASH:u64_le] [FIELD1] [FIELD2] ...  // Named structs: structure hash + fields in order
[COUNT:variable] [FIELD1] [FIELD2] ...  // Tuple structs: field count + fields in order
// (no data)  // Unit structs: no additional data

Enums:

[VARIANT_ID:variable] [HASH:u64_le] [FIELD1] [FIELD2] ...  // Named variants
[VARIANT_ID:variable] [COUNT:variable] [FIELD1] [FIELD2] ...  // Tuple variants  
[VARIANT_ID:variable]  // Unit variants

2.4 Structure Hash

Named structs and enum variants include a CRC64 structure hash for validation:

  • Computed from type name, field names, and field types
  • Ensures pack/unpack compatibility
  • Detects structural mismatches at runtime

3. Data Type Specifications

3.1 Boolean

Pack Format:

  • false: 0x00
  • true: 0x01

Unpack Behavior:

  • 0x00 → false
  • Any non-zero value → true (relaxed validation)

3.2 Unsigned Integers (u16, u32, u64, u128, usize)

Pack Format (Variable-Length Encoding):

All unsigned integers except u8 use the same variable-length encoding as the encode format:

// Small values (0-127)
value -> [TAG_ZERO + value]  // Direct encoding in tag byte

// Medium values (128-383)  
value -> [TAG_U8] [value - 128]  // 2 bytes total

// Larger values use appropriate width tags
u16 -> [TAG_U16] [value:u16_le]  // if value > 383
u32 -> [TAG_U32] [value:u32_le]  // if value > 65535
u64 -> [TAG_U64] [value:u64_le]  // if value > 4294967295
u128 -> [TAG_U128] [value:u128_le]  // if value > 18446744073709551615
usize -> (same as u32 or u64 depending on platform)

Tags:

  • TAG_ZERO = 0x00 to TAG_U8_127 = 0x7F (values 0-127)
  • TAG_U8 = 0x83 (131)
  • TAG_U16 = 0x84 (132)
  • TAG_U32 = 0x85 (133)
  • TAG_U64 = 0x86 (134)
  • TAG_U128 = 0x87 (135)

Size Examples:

  • 0-127: 1 byte
  • 128-383: 2 bytes
  • 384-65535: 3 bytes (u16)
  • 65536-4294967295: 5 bytes (u32)
  • etc.

3.3 u8 (Special Case)

Pack Format:

u8 -> [value:u8]  // Direct 1-byte encoding without tags

Size: Always 1 byte (no tag overhead)

3.4 Signed Integers (i8, i16, i32, i64, i128, isize)

Pack Format (Variable-Length Encoding):

All signed integers use the same variable-length encoding as the encode format:

// Positive values (0 and above)
value -> (encoded as corresponding unsigned type)

// Negative values  
value -> [TAG_NEGATIVE] [encoded_inverted_value]

Where encoded_inverted_value is !(value as unsigned_type) encoded using the unsigned format.

Tag:

  • TAG_NEGATIVE = 0x88 (136)

Examples:

0     -> [0x00]                    // 1 byte
42    -> [0x2A]                    // 1 byte  
-1    -> [0x88] [0x00]             // 2 bytes (TAG_NEGATIVE + inverted 0)
-42   -> [0x88] [0x29]             // 2 bytes (TAG_NEGATIVE + inverted 41)
1000  -> [0x84] [0xE8, 0x03]       // 3 bytes (TAG_U16 + 1000 as u16_le)
-1000 -> [0x88] [0x84] [0xE7, 0x03] // 4 bytes (TAG_NEGATIVE + TAG_U16 + inverted 999)

3.5 Floating Point (f32, f64)

Pack Format with Optimization:

f32 -> [TAG_NONE]                    // if value == 0.0
    -> [TAG_F32] [value:f32_le]      // if value != 0.0

f64 -> [TAG_NONE]                    // if value == 0.0  
    -> [TAG_F64] [value:f64_le]      // if value != 0.0

Tags:

  • TAG_NONE = 0x80
  • TAG_F32 = 0x89
  • TAG_F64 = 0x8A

Size:

  • 0.0 values: 1 byte
  • Non-zero f32: 5 bytes (1 tag + 4 data)
  • Non-zero f64: 9 bytes (1 tag + 8 data)

3.6 Character (char)

Pack Format (Variable-Length Encoding):

char -> (encoded as u32 using variable-length format)

The Unicode code point is encoded using the same variable-length format as u32:

  • Small code points (0-127): 1 byte
  • Medium code points (128-383): 2 bytes
  • Larger code points: 3-5 bytes depending on value

Size: 1-5 bytes depending on Unicode code point value

3.7 String

Pack Format:

// Short strings (0-40 characters)
String -> [TAG_STRING_BASE + len] [utf8_bytes]

// Long strings (41+ characters)  
String -> [TAG_STRING_LONG] [len:variable] [utf8_bytes]

Tags:

  • TAG_STRING_BASE = 0x8B (139)
  • TAG_STRING_LONG = 0xB4 (180)

Examples:

""      -> [0x8B]                    // Empty string
"Hi"    -> [0x8D, 0x48, 0x69]        // 2-char string
"Long..." -> [0xB4, len, utf8_bytes] // Long string

3.8 Option

Pack Format:

None    -> [TAG_NONE]
Some(v) -> [TAG_SOME] [packed_value]

Tags:

  • TAG_NONE = 0x80
  • TAG_SOME = 0x81

3.9 Vec

Pack Format:

// Short vectors (0-5 elements)
Vec<T> -> [TAG_ARRAY_VEC_SET_BASE + len] [element1] [element2] ...

// Long vectors (6+ elements)
Vec<T> -> [TAG_ARRAY_VEC_SET_LONG] [len:variable] [element1] [element2] ...

Tags:

  • TAG_ARRAY_VEC_SET_BASE = 0xBC (188)
  • TAG_ARRAY_VEC_SET_LONG = 0xC2 (194)

3.10 Arrays [T; N]

Pack Format:

// Short arrays (0-5 elements)
[T; N] -> [TAG_ARRAY_VEC_SET_BASE + N] [element1] [element2] ... [elementN]

// Long arrays (6+ elements)
[T; N] -> [TAG_ARRAY_VEC_SET_LONG] [N:variable] [element1] [element2] ... [elementN]

Tags:

  • TAG_ARRAY_VEC_SET_BASE = 0xBC (188)
  • TAG_ARRAY_VEC_SET_LONG = 0xC2 (194)

Note: Array length N is encoded and validated against the expected size during unpack.

3.11 HashMap<K, V>

Pack Format:

HashMap -> [TAG_MAP] [len:variable] [key1] [value1] [key2] [value2] ...

Tag:

  • TAG_MAP = 0xC4 (196)

3.12 Tuples

Pack Format:

() -> [TAG_TUPLE] [0:variable]
(T1,) -> [TAG_TUPLE] [1:variable] [element1]
(T1, T2) -> [TAG_TUPLE] [2:variable] [element1] [element2]
...

Tag:

  • TAG_TUPLE = 0xC3 (195)

3.13 Bytes

Pack Format:

Bytes -> [TAG_BINARY] [len:variable] [byte_data]

Tag:

  • TAG_BINARY = 0xB5 (181)

3.14 Extended Types (Feature-Dependent)

DateTime (chrono feature)

Pack Format:

DateTime<Utc> -> [TAG_NONE]                          // if default value
              -> [TAG_CHRONO_DATETIME] [seconds:i64] [nanos:u32]  // if non-default

DateTime<Local> -> [TAG_NONE]                        // if default value  
                -> [TAG_CHRONO_DATETIME] [seconds:i64] [nanos:u32]  // if non-default

All DateTime types are normalized to UTC for storage.

Tags:

  • TAG_NONE = 0x80
  • TAG_CHRONO_DATETIME = 0xC5 (197)

NaiveDate (chrono feature)

Pack Format:

NaiveDate -> [TAG_CHRONO_NAIVE_DATE] [days_from_epoch:i64]

Tag:

  • TAG_CHRONO_NAIVE_DATE = 0xC6 (198)

Epoch: 1970-01-01

NaiveTime (chrono feature)

Pack Format:

NaiveTime -> [TAG_CHRONO_NAIVE_TIME] [seconds_from_midnight:u32] [nanoseconds:u32]

Tag:

  • TAG_CHRONO_NAIVE_TIME = 0xC7 (199)

NaiveDateTime (chrono feature)

Pack Format:

NaiveDateTime -> [TAG_NONE]                              // if default value
              -> [TAG_CHRONO_NAIVE_DATETIME] [seconds:i64] [nanos:u32]  // if non-default

Stores as seconds and nanoseconds since Unix epoch (1970-01-01 00:00:00 UTC).

Tags:

  • TAG_NONE = 0x80
  • TAG_CHRONO_NAIVE_DATETIME = 0xD0 (208)

Decimal (rust_decimal feature)

Pack Format:

Decimal -> [TAG_DECIMAL] [mantissa:i128] [scale:u32]

Tag:

  • TAG_DECIMAL = 0xC8 (200)

UUID/ULID (uuid/ulid features)

Pack Format:

UUID -> [TAG_NONE]               // if default/nil value
     -> [TAG_UUID] [value:u128_le]  // if non-default

ULID -> [TAG_NONE]               // if default/nil value  
     -> [TAG_UUID] [value:u128_le]  // if non-default

Tags:

  • TAG_NONE = 0x80
  • TAG_UUID = 0xC9 (201)

Note: UUID and ULID share the same tag and are binary compatible at the encoding level.

4. Struct and Enum Packing

4.1 Named Structs

Pack Format:

struct MyStruct {
    field1: T1,
    field2: T2,
} -> [structure_hash:u64_le] [packed_field1] [packed_field2]

Structure Hash: CRC64 of "type:MyStruct|struct|named|field1:T1|field2:T2"

Example:

struct User {
    id: u32,
    name: String,
}

// Packed as: [hash:u64] [id:u32] [name:String]

4.2 Tuple Structs

Pack Format:

struct MyStruct(T1, T2) -> [field_count:variable] [packed_field1] [packed_field2]

Example:

struct Point(f32, f32);

// Packed as: [2:variable] [x:f32] [y:f32]

4.3 Unit Structs

Pack Format:

struct MyStruct; -> // (no additional data)

4.4 Enums

Named Variants

enum MyEnum {
    Variant { field1: T1, field2: T2 }
} -> [variant_id:variable] [structure_hash:u64_le] [packed_field1] [packed_field2]

Tuple Variants

enum MyEnum {
    Variant(T1, T2)
} -> [variant_id:variable] [field_count:variable] [packed_field1] [packed_field2]

Unit Variants

enum MyEnum {
    Variant
} -> [variant_id:variable]

Variant IDs: CRC64 hash of variant name (same as encode format)

5. Optimization Features

5.1 Zero/Default Value Optimization

f32/f64: 0.0 values use single TAG_NONE byte instead of full representation.

chrono types: Default values use single TAG_NONE byte:

  • DateTime<Utc>::default() → [TAG_NONE]
  • DateTime<Local>::default() → [TAG_NONE]
  • NaiveDateTime::default() → [TAG_NONE]

UUID/ULID: Nil/default values use single TAG_NONE byte:

  • Uuid::nil() → [TAG_NONE]
  • Ulid::nil() → [TAG_NONE]

5.2 Short Collection Optimization

Vectors/Arrays/Sets: Collections with 0-5 elements encode length in the tag byte.

5.3 Short String Optimization

Strings: Strings with 0-40 characters encode length in the tag byte.

5.4 Direct Integer Encoding

Integers: Variable-length encoding with optimized representation for small values.

6. Implementation Notes

6.1 Field Order Dependency

Critical: Pack format requires exact field order matching between pack and unpack operations.

// This works
struct A { x: u32, y: String }
struct B { x: u32, y: String }  // Same order

// This fails  
struct A { x: u32, y: String }
struct C { y: String, x: u32 }  // Different order

6.2 Structure Hash Validation

Named structs and enum variants validate structure hash during unpack:

  • Prevents accidental type mismatches
  • Detects field reordering
  • Ensures pack/unpack compatibility

6.3 Cross-Type Compatibility

Pack format supports the same cross-type decoding as encode format:

  • Integer widening (u16 → u32)
  • Signed/unsigned conversion (where safe)

6.4 Performance Characteristics

Pack Advantages:

  • 20-50% smaller binary size (no field IDs)
  • Faster encoding/decoding (less metadata)
  • Direct memory mapping potential

Pack Limitations:

  • No schema evolution support
  • Field order dependency
  • Requires exact type matching

6.5 Use Case Guidelines

Use Pack When:

  • Maximum performance is required
  • Binary size is critical
  • Schema is stable and controlled
  • Both ends use the same code version

Use Encode When:

  • Schema evolution is needed
  • Long-term data storage
  • API versioning is required
  • Forward/backward compatibility is important

7. Examples

7.1 Simple Struct

#[derive(Pack, Unpack)]
struct Point {
    x: f32,
    y: f32,
}

let point = Point { x: 1.0, y: 2.0 };
// Packed: [hash:8] [TAG_F32][1.0:4] [TAG_F32][2.0:4] = 18 bytes

7.2 Enum with Mixed Variants

#[derive(Pack, Unpack)]
enum Message {
    #[senax(id=1)]
    Text(String),
    #[senax(id=2)]
    Data { id: u32, payload: Vec<u8> },
    Ping,
}

let msg = Message::Data { id: 42, payload: vec![1, 2, 3] };
// Packed: [variant_id] [hash:8] [42:1] [TAG_ARRAY+3][1][2][3] = 14 bytes

7.3 Optimization Examples

// Floating point optimization
let zero_f32 = 0.0f32;        // Packed: [TAG_NONE] = 1 byte
let nonzero_f32 = 3.14f32;    // Packed: [TAG_F32][3.14:4] = 5 bytes

// Collection optimization
let empty_vec: Vec<u8> = vec![];     // Packed: [TAG_ARRAY+0] = 1 byte  
let small_vec = vec![1u8, 2, 3];     // Packed: [TAG_ARRAY+3][1][2][3] = 4 bytes
let large_vec = vec![0u8; 100];      // Packed: [TAG_ARRAY_LONG][100][data] = ~103 bytes

// String optimization
let short_str = "Hi";                // Packed: [TAG_STRING+2][Hi] = 3 bytes
let empty_str = "";                  // Packed: [TAG_STRING+0] = 1 byte

// Chrono type optimization
use chrono::{DateTime, NaiveDateTime, Utc};
let default_dt = DateTime::<Utc>::default();  // Packed: [TAG_NONE] = 1 byte
let actual_dt = DateTime::from_timestamp(1640995200, 0).unwrap();  
// Packed: [TAG_CHRONO_DATETIME][seconds:i64][nanos:u32] = 13 bytes

let default_naive = NaiveDateTime::default();  // Packed: [TAG_NONE] = 1 byte
let actual_naive = DateTime::from_timestamp(1640995200, 123456789).unwrap().naive_utc();
// Packed: [TAG_CHRONO_NAIVE_DATETIME][seconds:i64][nanos:u32] = 13 bytes

// UUID optimization  
use uuid::Uuid;
let nil_uuid = Uuid::nil();          // Packed: [TAG_NONE] = 1 byte
let actual_uuid = Uuid::new_v4();    // Packed: [TAG_UUID][value:u128] = 17 bytes

8. Migration Guide

8.1 From Encode to Pack

  1. Ensure schema stability
  2. Verify field order consistency
  3. Test with actual data
  4. Measure size/performance gains

8.2 From Pack to Encode

  1. Add field IDs if needed
  2. Update error handling for unknown fields
  3. Test schema evolution scenarios
  4. Consider migration strategy for existing data