Website: introduce future work with p2p evoluation and persistence

Author: dannywillems

website/docs/developers/future-work.md

@@ -0,0 +1,84 @@
+---
+sidebar_position: 1
+title: Overview
+description:
+  Overview of planned enhancements and design proposals for the Mina Rust node
+slug: /developers/future-work
+---
+
+# Future Work
+
+This section outlines planned enhancements and design proposals for the Mina
+Rust node that are not yet implemented but are critical for the project's
+continued development and mainnet readiness.
+
+## Overview
+
+The Mina Rust node is an active development project with several key areas
+identified for future enhancement. These proposals represent thoughtful designs
+for addressing current limitations and preparing the node for production
+deployment at scale.
+
+## Key Areas
+
+### P2P Networking Evolution
+
+The [P2P Evolution Plan](p2p-evolution) builds on the successful pull-based
+design already implemented for webnodes. Key planned improvements include:
+
+- **QUIC Transport Integration**: Adding QUIC as an alternative transport to
+  WebRTC for improved performance and reduced complexity
+- **Block Propagation Optimization**: Reducing bandwidth usage by sending only
+  block headers and missing data
+- **OCaml Node Integration**: Potential unification of networking layers across
+  Mina implementations
+
+### Persistent Storage
+
+The [Persistence Design](persistence-design) addresses one of the most critical
+requirements for mainnet readiness. Current challenges include:
+
+- **Memory Usage**: The entire ledger is currently kept in memory, creating
+  scalability issues
+- **Startup Time**: Nodes must reconstruct the full ledger, which is
+  time-consuming
+- **SNARK Verification**: Redundant verification work across blocks and pools
+
+## Implementation Status
+
+**Important**: All items in this section are design proposals only and are not
+yet implemented. These documents serve as:
+
+- **Technical Specifications**: Detailed designs for future implementation
+- **Discussion Starting Points**: Basis for technical discussions and
+  refinements
+- **Roadmap Guidance**: Priority areas for development effort
+
+## Contribution
+
+These design proposals benefit from community review and input. Developers
+interested in contributing to these areas should:
+
+1. **Review the designs** thoroughly to understand the proposed approaches
+2. **Check the
+   [project dashboard](https://github.com/orgs/o1-labs/projects/24/)** for
+   current development status and active work items
+3. **Provide feedback** on technical feasibility and implementation details
+4. **Participate in discussions** about priorities and trade-offs
+5. **Contribute to implementation** when development begins
+
+The designs may evolve based on community feedback, technical constraints, and
+changing requirements as the Mina ecosystem develops.
+
+## Timeline Considerations
+
+While these documents provide detailed implementation plans, actual development
+timelines depend on:
+
+- **Resource allocation** and team priorities
+- **Community needs** and feedback
+- **Technical dependencies** and prerequisites
+- **Coordination requirements** with other Mina implementations
+
+The goal is to ensure these enhancements are implemented thoughtfully and with
+proper consideration for the broader Mina ecosystem.

website/docs/developers/getting-started.mdx

@@ -23,6 +23,12 @@ import NixDevelop from "!!raw-loader!./scripts/setup/nix-develop.sh";
 Welcome to Mina Rust Node development! This guide will help you set up your
 development environment and build the Mina Rust Node from source.
 
+This documentation covers the current implementation, architecture, and
+networking approaches. The [**Future Work**](future-work) section outlines
+planned enhancements and design proposals for upcoming features like persistent
+storage and P2P protocol evolution that are not yet implemented but are critical
+for the node's continued development and mainnet readiness.
+
 ## Prerequisites
 
 ### System Requirements

website/docs/developers/p2p-evolution.md

@@ -0,0 +1,293 @@
+---
+sidebar_position: 5
+title: P2P Evolution Plan
+description: Evolution plan for Mina's P2P networking layer
+slug: /developers/p2p-evolution
+---
+
+# P2P Layer Evolution Plan
+
+This document outlines the evolution plan for Mina's P2P networking layer,
+building on the successful pull-based design already implemented for the Mina
+Rust node webnodes. The idea of using QUIC as a transport was originally
+proposed by George in his "Networking layer 2.0" document.
+
+**Status**: The pull-based P2P protocol is implemented and operational. This
+document proposes enhancements including QUIC transport, block propagation
+optimizations, and integration with the OCaml node to create a unified
+networking layer across all Mina implementations. Coordination with OCaml Mina
+team required for ecosystem-wide adoption.
+
+## Current State
+
+### The Problem: Divergent P2P Architectures
+
+The Mina ecosystem currently has divergent P2P implementations:
+
+1. **Mina (OCaml) nodes**
+   - Use libp2p exclusively via external Golang helper process (`libp2p_helper`)
+   - Push-based GossipSub protocol
+   - Known weaknesses in network performance and scalability
+
+2. **The Mina Rust node**
+   - Support both libp2p (for OCaml compatibility) AND pull-based WebRTC
+   - Must internally normalize between push and pull models, adding complexity
+   - Webnodes use WebRTC exclusively and require Rust nodes as bridges to libp2p
+     network
+   - Maintenance burden of supporting two different protocol designs
+
+This creates significant complexity:
+
+- The Mina Rust node maintains two protocol implementations
+- Webnodes cannot directly communicate with OCaml nodes
+- Different security and performance characteristics
+- Inconsistent behavior and debugging challenges
+
+## Vision: Unified Pull-Based P2P Layer
+
+The goal is to evolve the Mina Rust node's pull-based P2P design to improve
+webnode networking immediately and potentially become the universal networking
+layer for all Mina nodes (both Rust and OCaml), with multiple transport options.
+Full ecosystem adoption would require coordination and agreement with the OCaml
+Mina team.
+
+### Core Design Principles
+
+The pull-based model addresses fundamental problems in traditional push-based
+systems:
+
+#### Problems with Push-Based Systems
+
+- **Resource Exhaustion**: Message queues grow unboundedly during high traffic
+- **Message Loss**: Dropped messages break eventual consistency
+- **DDOS Vulnerability**: Attackers can flood nodes with messages
+- **Fairness Issues**: Some peers can monopolize resources
+
+#### Pull-Based Advantages
+
+- **Flow Control**: Recipients control message flow through permits
+- **Resource Protection**: Processing required before requesting next message
+- **Eventual Consistency**: Guaranteed message delivery and processing
+- **Fairness**: Equal resource allocation across peers
+
+## Evolution Phases
+
+### Phase 1: Enhanced WebRTC Implementation (Current)
+
+**Status**: ✅ Complete
+
+- Pull-based messaging with WebRTC transport
+- Multiple signaling methods (HTTP, relay-based)
+- Channel isolation per protocol type
+- Efficient pool propagation
+- NAT traversal and encryption
+
+### Phase 2: QUIC Transport Integration
+
+**Goals**:
+
+- Add QUIC as alternative transport to WebRTC
+- Maintain pull-based protocol semantics
+- Improve performance and reduce complexity
+
+**Benefits**:
+
+- **Simplified NAT Traversal**: QUIC handles NAT better than WebRTC setup
+- **Lower Latency**: Reduced connection establishment time
+- **Better Multiplexing**: Native stream multiplexing without complex setup
+- **Standardized Protocol**: Well-defined, battle-tested transport
+
+**Implementation**:
+
+- QUIC streams map to current WebRTC data channels
+- Same pull-based messaging protocol
+- Gradual rollout alongside existing WebRTC
+
+### Phase 3: Block Propagation Optimization
+
+**Current Challenge**: Blocks contain redundant data (transactions, SNARKs)
+already in local pools.
+
+**Solution**:
+
+- Send block headers + merkle proofs + missing data only
+- Nodes reconstruct full blocks from local pools
+- Dramatic reduction in block transmission size
+- Faster propagation across network
+
+**Benefits**:
+
+- Reduced bandwidth usage
+- Lower memory overhead
+- Faster block propagation
+- Improved scalability
+
+### Phase 4: OCaml Node Integration (Future)
+
+**Vision**: Enable OCaml nodes to use pull-based protocol
+
+**Approach Options**:
+
+1. **FFI Integration**
+   - Bind Rust P2P implementation to OCaml
+   - Gradual migration from libp2p
+   - Maintains OCaml node architecture
+
+2. **Protocol Standardization**
+   - Define language-agnostic pull-based protocol specification
+   - OCaml native implementation
+   - Both implementations interoperate
+
+3. **Hybrid Bridge**
+   - Enhanced bridge between protocols
+   - Improved push-to-pull translation
+   - Maintains backward compatibility
+
+## Technical Implementation Details
+
+### Transport Layer Abstraction
+
+```rust
+trait Transport {
+    async fn connect(&self, addr: Address) -> Result<Connection>;
+    async fn listen(&self, addr: Address) -> Result<Listener>;
+}
+
+impl Transport for WebRtcTransport { ... }
+impl Transport for QuicTransport { ... }
+```
+
+### Protocol Compatibility
+
+Pull-based protocol remains transport-agnostic:
+
+- Same message formats
+- Same flow control semantics
+- Same channel abstractions
+- Transport selection via configuration
+
+### Migration Strategy
+
+1. **Parallel Operation**: Run both transports simultaneously
+2. **Gradual Adoption**: Nodes advertise transport capabilities
+3. **Preference System**: Prefer QUIC when both peers support it
+4. **Fallback Support**: Maintain WebRTC for compatibility
+
+## Performance Expectations
+
+### QUIC Benefits Over WebRTC
+
+- **Connection Time**: ~50% reduction in handshake time
+- **Memory Usage**: Lower per-connection overhead
+- **CPU Usage**: Reduced encryption/decryption overhead
+- **Multiplexing**: More efficient stream management
+
+### Block Propagation Improvements
+
+- **Size Reduction**: 60-80% smaller block messages
+- **Propagation Speed**: 2-3x faster across network
+- **Resource Usage**: Significant reduction in bandwidth and parsing
+
+## Ecosystem Integration
+
+### Webnode Improvements
+
+- Direct QUIC connections without complex WebRTC setup
+- Better performance behind restrictive networks
+- Simplified debugging and monitoring
+
+### OCaml Node Benefits (Future)
+
+- Access to optimized pull-based protocol
+- Improved network performance
+- Unified P2P behavior across implementations
+
+### Network-Wide Effects
+
+- More efficient resource utilization
+- Better resistance to network attacks
+- Improved consistency guarantees
+- Enhanced scalability
+
+## Implementation Timeline
+
+### Immediate (Current Release Cycle)
+
+- ✅ WebRTC pull-based implementation
+- ✅ Multi-transport abstraction foundation
+
+### Short Term (Next 2-3 Releases)
+
+- QUIC transport implementation
+- Block propagation optimization
+- Performance benchmarking
+
+### Medium Term (6-12 Months)
+
+- Production QUIC deployment
+- Advanced block reconstruction
+- Protocol refinements based on real-world usage
+
+### Long Term (12+ Months)
+
+- OCaml integration exploration
+- Protocol standardization
+- Ecosystem-wide adoption planning
+
+## Success Metrics
+
+### Technical Metrics
+
+- Connection establishment time reduction
+- Block propagation latency improvement
+- Bandwidth usage reduction
+- Memory and CPU usage optimization
+
+### Network Health
+
+- Improved consensus convergence time
+- Reduced network partitions
+- Better handling of high-traffic periods
+- Enhanced resistance to attacks
+
+### Developer Experience
+
+- Simplified debugging
+- Unified protocol behavior
+- Better monitoring and observability
+- Reduced maintenance burden
+
+## Risks and Mitigation
+
+### Technical Risks
+
+- **QUIC Implementation Complexity**: Mitigate with gradual rollout and
+  extensive testing
+- **Transport Compatibility**: Maintain WebRTC fallback during transition
+- **Protocol Changes**: Ensure backward compatibility during evolution
+
+### Ecosystem Risks
+
+- **Adoption Resistance**: Demonstrate clear benefits before proposing ecosystem
+  changes
+- **Fragmentation**: Maintain compatibility with existing implementations
+- **Coordination Complexity**: Start with Mina Rust node-only improvements
+
+### Mitigation Strategies
+
+- Incremental rollout with feature flags
+- Comprehensive testing across different network conditions
+- Close coordination with stakeholders
+- Clear migration paths and documentation
+
+## Conclusion
+
+The P2P layer evolution builds on the Mina Rust node's successful pull-based
+design to create a more efficient, secure, and unified networking layer for the
+Mina ecosystem. While immediate improvements benefit the Mina Rust node and
+webnodes, the long-term vision of ecosystem-wide adoption would require
+coordination with the OCaml Mina team and careful migration planning.
+
+The phased approach allows for immediate improvements while keeping future
+integration possibilities open, ensuring that the Mina network can evolve toward
+better performance and consistency regardless of implementation language.