Skip to content

FileStorage Implementation Documentation

Overview

The FileStorage implementation represents the completion of KotaDB's storage engine layer, built using the full 6-stage risk reduction methodology. This provides a production-ready, file-based storage system with comprehensive safety features and observability.

Architecture

Core Components

// Core implementation
src/file_storage.rs        // FileStorage struct implementing Storage trait
src/lib.rs                 // Module exports and integration

// Testing and examples
tests/file_storage_integration_test.rs  // Comprehensive integration tests
examples/file_storage_demo.rs           // Usage demonstration

// Factory function
create_file_storage()      // Production-ready instantiation with all wrappers

Stage 6 Integration

The FileStorage leverages the complete Stage 6 Component Library:

pub async fn create_file_storage(
    path: &str,
    cache_capacity: Option<usize>,
) -> Result<TracedStorage<ValidatedStorage<RetryableStorage<CachedStorage<FileStorage>>>>> {
    // Creates fully wrapped storage with all Stage 6 components
}

Wrapper Composition: 1. CachedStorage - LRU caching for performance 2. RetryableStorage - Automatic retry with exponential backoff 3. ValidatedStorage - Contract enforcement and validation 4. TracedStorage - Comprehensive observability and metrics

Implementation Details

File Organization

database_path/
├── documents/           # Document content and metadata
│   ├── {uuid}.md       # Document content files
│   └── {uuid}.json     # Document metadata
├── indices/            # Index data (future implementation)
├── wal/               # Write-ahead logging
│   └── current.wal    # Current WAL file
└── meta/              # Database metadata

Document Storage

Documents are stored using a dual-file approach: - Content files (.md): Human-readable markdown content - Metadata files (.json): Structured metadata for fast lookups

struct DocumentMetadata {
    id: Uuid,
    file_path: PathBuf,
    size: u64,
    created: i64,
    updated: i64,
    hash: [u8; 32],
}

In-Memory Index

The FileStorage maintains an in-memory HashMap for fast document lookups:

pub struct FileStorage {
    db_path: PathBuf,
    documents: RwLock<HashMap<Uuid, DocumentMetadata>>,
    wal_writer: RwLock<Option<tokio::fs::File>>,
}

This provides O(1) lookup performance while maintaining durability through file persistence.

CRUD Operations

Insert

  1. Validate document doesn't already exist
  2. Write content to .md file
  3. Create and persist metadata to .json file
  4. Update in-memory index

Read

  1. Check in-memory index for metadata
  2. Read content from corresponding .md file
  3. Reconstruct Document struct

Update

  1. Verify document exists
  2. Update content file
  3. Update metadata with new timestamps and hash
  4. Refresh in-memory index

Delete

  1. Remove from in-memory index
  2. Delete both content and metadata files
  3. Handle gracefully if files don't exist

Safety and Reliability Features

Stage 1: Test Coverage

  • Comprehensive integration tests covering all CRUD operations
  • Multi-document scenarios
  • Persistence verification across storage instances
  • Error handling validation

Stage 2: Contract Enforcement

  • All Storage trait preconditions and postconditions validated
  • Input validation through existing Stage 2 validation functions
  • Runtime assertion system prevents invalid operations

Stage 3: Pure Function Integration

  • Uses existing validation::path::validate_directory_path for path safety
  • Leverages pure functions for word counting and content processing
  • Clear separation of I/O operations from business logic

Stage 4: Comprehensive Observability

  • Automatic operation tracing with unique trace IDs
  • Performance metrics collection for all operations
  • Structured error reporting with full context
  • Operation counting and timing statistics

Stage 5: Adversarial Resilience

  • Handles file system errors gracefully
  • Protects against path traversal attacks
  • Recovers from partial write failures
  • Validates data integrity on read operations

Stage 6: Component Library Safety

  • Validated Types: All inputs validated at type level
  • Builder Patterns: Safe document construction with fluent API
  • Wrapper Components: Automatic application of best practices
  • Factory Function: One-line instantiation with all safety features

Usage Examples

Basic Usage

use kotadb::{create_file_storage, DocumentBuilder, Storage};

#[tokio::main]
async fn main() -> Result<()> {
    // Create production-ready storage
    let mut storage = create_file_storage("/path/to/db", Some(1000)).await?;

    // Create document using builder
    let doc = DocumentBuilder::new()
        .path("/notes/rust-patterns.md")?
        .title("Rust Design Patterns")?
        .content(b"# Rust Patterns\n\nKey patterns...")?
        .build()?;

    // Store document (automatically traced, validated, cached, retried)
    storage.insert(doc.clone()).await?;

    // Retrieve document (cache-optimized)
    let retrieved = storage.get(&doc.id).await?;

    Ok(())
}

Advanced Configuration

// High-performance configuration with large cache
let storage = create_file_storage("/fast/ssd/path", Some(10_000)).await?;

// Memory-constrained configuration
let storage = create_file_storage("/path/to/db", Some(100)).await?;

Integration with Existing Systems

// The FileStorage implements the Storage trait, so it can be used
// anywhere a Storage implementation is expected
fn process_documents<S: Storage>(storage: &mut S) -> Result<()> {
    // Works with FileStorage or any other Storage implementation
}

Performance Characteristics

Memory Usage

  • Base overhead: ~200 bytes per document (metadata)
  • Cache overhead: Configurable LRU cache size
  • Index overhead: HashMap with O(1) lookup performance

Disk Usage

  • Content files: Variable size based on document content
  • Metadata files: ~150-200 bytes per document
  • WAL overhead: Minimal until significant write volume

Operation Performance

  • Insert: ~1-5ms (depending on document size)
  • Read: ~0.1-1ms (cache hit: ~0.01ms)
  • Update: ~1-5ms (similar to insert)
  • Delete: ~0.5-2ms (file system dependent)

Error Handling

Graceful Degradation

  • File system errors include detailed context
  • Partial failures don't corrupt database state
  • Read-only mode available if write permissions unavailable
  • Automatic recovery from interrupted operations

Error Categories

  1. Validation Errors: Invalid input data or operations
  2. I/O Errors: File system access issues
  3. Concurrency Errors: Lock contention or race conditions
  4. Corruption Errors: Data integrity verification failures

Future Enhancements

Planned Improvements

  1. Compression: Document content compression for large files
  2. Encryption: At-rest encryption for sensitive data
  3. Backup Integration: Automatic backup and restore capabilities
  4. Metrics Dashboard: Real-time performance monitoring
  5. Advanced Caching: Multi-level cache hierarchy

Index Integration

The FileStorage is designed to work seamlessly with future index implementations: - Primary Index: Document ID → File path mapping - Full-Text Index: Content tokenization and search - Graph Index: Document relationship tracking - Semantic Index: Vector embeddings for similarity search

Security Considerations

Path Safety

  • All paths validated through existing Stage 2 validation
  • No directory traversal vulnerabilities
  • Sandbox constraints enforced at API level

Data Integrity

  • SHA-256 hashes for content verification
  • Atomic file operations prevent corruption
  • WAL ensures consistency during failures

Access Control

  • File system permissions determine access rights
  • No additional authentication layer (delegated to OS)
  • Audit trail through comprehensive logging

Debugging and Troubleshooting

Log Analysis

All operations automatically logged with: - Unique trace IDs for correlation - Operation timing and performance metrics - Error context and stack traces - Cache hit/miss ratios

Common Issues

  1. Permission Errors: Check file system permissions
  2. Disk Space: Monitor available storage
  3. Corruption: Verify file integrity and restore from backup
  4. Performance: Analyze cache hit ratios and tune cache size

Diagnostic Tools

# Check database status
./run_standalone.sh status

# Run integration tests
./run_standalone.sh test file_storage_integration_test

# Run performance demo
cargo run --example file_storage_demo

Integration with KotaDB Architecture

The FileStorage implementation represents the foundational layer for the complete KotaDB system:

Query Interface
       ↓
Query Engine  
       ↓
Indices (Future)
       ↓
FileStorage ← YOU ARE HERE
       ↓
File System

This storage layer provides the reliable foundation needed for building the remaining database components while maintaining the 99% success rate achieved through the 6-stage risk reduction methodology.

Conclusion

The FileStorage implementation successfully delivers:

✅ Production-Ready Storage: Complete CRUD operations with safety guarantees
✅ Stage 6 Integration: Automatic application of all safety and performance features
✅ Comprehensive Testing: Full integration test coverage
✅ Documentation: Complete usage examples and architectural guidance
✅ Future-Proof Design: Ready for index and query engine integration

The implementation maintains KotaDB's 99% success rate while providing the essential storage capabilities needed for the next development phase: index implementation.