StorageFileImportExt: Filesystem Import

Loading…

StorageFileImportExt: Filesystem Import

Relevant source files

• extensions/src/lib.rs
• extensions/src/storage_file_import_ext.rs
• extensions/tests/storage_file_import_tests.rs

The StorageFileImportExt trait extends the DataStore with capabilities to ingest entire directory trees from the local filesystem into the append-only store. It specifically addresses the need for bulk data migration and streaming file access, using relative paths as keys to maintain directory structure within the schema-less storage engine.

Overview and Purpose

The primary goal of StorageFileImportExt is to provide a high-level API for synchronizing filesystem assets with a DataStore extensions/src/storage_file_import_ext.rs:12-59 It abstracts the complexity of recursive directory walking and ensures that file contents are written using efficient streaming I/O rather than loading entire files into memory.

Key features include:

• Recursive Ingestion : Automatically walks subdirectories using the walkdir crate extensions/src/storage_file_import_ext.rs:77-80
• Path-to-Key Mapping : Converts OS-specific paths into normalized, Unix-style binary keys (e.g., subdir/file.txt) extensions/src/storage_file_import_ext.rs:117-124
• Namespacing : Supports optional NamespaceHasher integration to prevent collisions when importing multiple directories into the same store extensions/src/storage_file_import_ext.rs:126-129
• Streaming Reads : Provides EntryStream wrappers for zero-copy access to stored file data extensions/src/storage_file_import_ext.rs:54-58

Sources : extensions/src/storage_file_import_ext.rs:1-59 extensions/src/storage_file_import_ext.rs:117-130

Implementation Detail: Recursive Import

The import_dir_recursively function serves as the entry point for filesystem ingestion. It validates the source directory using standard std::path::Path checks before initiating the walk extensions/src/storage_file_import_ext.rs:68-75

Data Flow: Filesystem to DataStore

The following diagram illustrates how a file on disk is transformed into a namespaced entry within the DataStore.

Import Pipeline: Path Normalization and Streaming Write

graph TD
    subgraph "Local Filesystem"
        A["File: ./assets/images/logo.png"]
end

    subgraph "StorageFileImportExt.import_dir_recursively()"
        B["walkdir::WalkDir"]
C["strip_prefix()"]
D["to_namespaced_key()"]
E["Unix-style Path: 'images/logo.png'"]
end

    subgraph "DataStore Logic"
        F["NamespaceHasher (Optional)"]
G["DataStore.write_stream()"]
end

 
   A --> B
 
   B --> C
 
   C --> E
 
   E --> D
 
   D --> F
 
   F --> G
 
   G --> H[("DataStore (.bin file)")]

Sources : extensions/src/storage_file_import_ext.rs:62-96 extensions/src/storage_file_import_ext.rs:117-130

Key Functions

Sources : extensions/src/storage_file_import_ext.rs:61-130

Streaming and Zero-Copy Access

A critical performance feature of this extension is its use of streaming for both ingestion and retrieval.

Streaming Write

Instead of reading a file into a buffer and passing it to DataStore.write(), import_dir_recursively opens a std::fs::File and passes it to DataStore.write_stream() extensions/src/storage_file_import_ext.rs:90-91 This allows the core engine to pipe data directly from the filesystem into the memory-mapped store, minimizing heap allocations.

Streaming Read (EntryStream)

When a file is retrieved via open_file_stream(), the system returns an EntryStream extensions/src/storage_file_import_ext.rs113 This object wraps an EntryHandle, which itself is a view into the memory-mapped storage file.

Code Entity Relationship: Read Streaming

classDiagram
    class DataStore {+read(key) EntryHandle}
    class StorageFileImportExt {+import_dir_recursively(base_dir, namespace)\n+open_file_stream(rel_path, namespace) EntryStream}
    class EntryHandle {
        +as_slice() &[u8]
    }
    class EntryStream {-handle: EntryHandle\n-pos: usize\n+read(buf) io::Result}

    DataStore <|-- StorageFileImportExt : implements
    StorageFileImportExt ..> EntryHandle : retrieves
    EntryStream o-- EntryHandle : wraps
    EntryStream ..|> "std::io::Read" : implements

Sources : extensions/src/storage_file_import_ext.rs:61-115 extensions/src/storage_file_import_ext.rs:2-3

Verification and Testing

The extension is verified through comprehensive integration tests that simulate directory imports and content validation.

• Content Integrity : Tests ensure that read_file_entry and open_file_stream return bytes identical to the original filesystem source extensions/tests/storage_file_import_tests.rs:17-61 extensions/tests/storage_file_import_tests.rs:131-168
• Path Normalization : Verification that keys are stored as Unix-style paths regardless of the host OS by converting the key back to UTF-8 and replacing separators during comparison extensions/tests/storage_file_import_tests.rs:41-42
• Error Cases : Validation that the system correctly rejects non-existent directories or file paths passed as directories extensions/tests/storage_file_import_tests.rs:171-193

Sources : extensions/tests/storage_file_import_tests.rs:1-193

Error Handling

• NotFound : Returned if the base_dir provided to import_dir_recursively does not exist or is not a directory extensions/src/storage_file_import_ext.rs:70-75
• IO Errors : Standard filesystem errors (permissions, disk full) encountered during WalkDir or File::open are propagated to the caller extensions/src/storage_file_import_ext.rs:90-91
• Key Collisions : Because DataStore is append-only, importing the same directory twice will result in new entries that mask the old ones in the KeyIndexer extensions/src/storage_file_import_ext.rs91

Sources : extensions/src/storage_file_import_ext.rs:62-96