HeapFile 1.0.0

HeapFile Library Documentation

The code provided is meticulously crafted by hand, ensuring precision and reliability. This accompanying markdown documentation, however, is generated by AI and, while informative, should be approached with a discerning eye for potential inaccuracies.

The HeapFile library stands as a refined solution for managing and storing data efficiently in a compact file format, drawing upon the principles of heap memory management. This documentation is designed to elucidate the foundational functionalities of the HeapFile library, encompassing creation, reading, updating, and deletion of data within heap files. Through illustrative examples, we aim to showcase the library's capabilities, underlining its applicability in scenarios demanding efficient data management.

Introduction

Leveraging the heap memory concept, the HeapFile library facilitates the creation and manipulation of data files in a compact manner. This approach is analogous to data handling in formats such as JSON, XML, or databases but is distinguished by its space-saving and performance-enhancing characteristics. Data is bifurcated into two files: the header file ({filename}.hpf.header), which stores pointers for rapid data access upon subsequent loads, and the data file ({filename}.hpf), which contains the actual data payload. This dual-file methodology simplifies data operations, particularly for extensive datasets.

Getting Started

Preliminaries

Prior to delving into the HeapFile library, ensure its integration into your project. This guide presupposes a working knowledge of C# and fundamental file operations.

Core Operations

File Initialization

To embark on utilizing heap files, instantiate a Fmem object with a designated file name. If an existing file needs to be overwritten, it must first be removed.

using HeapFile;

// Instantiate the Fmem object with a file name
Fmem fmem = new Fmem("File." + Fmem.FILE_EXTENSION);

Memory Allocation and Identifiers

Memory allocation is performed through Alloc, AllocString, or AllocArray methods, which return a Fpointer for subsequent data access. An identifier is employed to facilitate the association of pointers with their respective data types or structures, simplifying data retrieval upon program restarts.

// Allocate memory with identifiers for easy future reference
Fpointer pointerToInt = fmem.Alloc<int>(identifier); // For integers
Fpointer pointerToString = fmem.AllocString(ref someString, identifier); // For strings
Fpointer pointerToArray = fmem.Alloc<char>(arrayLength, identifier); // For arrays

Loading Existing Pointers

To facilitate seamless data retrieval across sessions, the HeapFile library provides a mechanism to load existing pointers from the header file. This is accomplished using the GetAllOldPointers method, which populates an array of Fpointer objects, each corresponding to previously allocated memory spaces. This feature ensures that data can be efficiently accessed without the need for re-allocation, thereby maintaining the integrity and continuity of your data management operations.

// Load existing pointers
Fpointer[] pointers;
fmem.GetAllOldPointers(out pointers);

// Example usage of loaded pointers, checking if pointers were previously allocated
if (pointers.Length > 0)
{
    // Pointers exist, proceed with using the pointers for data manipulation
}
else
{
    // No pointers loaded, proceed with initial allocation
}

Data Manipulation

Data is written to allocated memory spaces using Write, WriteString, or WriteArray methods. Reading data is similarly straightforward, employing Read, ReadString, or ReadArray methods.

// Writing data
fmem.Write<int>(pointerToInt, 100); // Integer
fmem.WriteString(pointerToString, "Hello, World!"); // String
fmem.WriteArray<char>(pointerToArray, new char[] {'H', 'i'}); // Array

// Reading data
int intValue = fmem.Read<int>(pointerToInt);
string stringValue = fmem.ReadString(pointerToString);
char[] arrayValue = fmem.ReadArray<char>(pointerToArray, arrayLength);

Memory Management

The Free method does not delete data but rather marks the allocated space as available for future allocations, analogous to the behavior of free in low-level memory management.

// Mark memory as free for future use
fmem.Free(pointerToInt); // Does not delete data but frees allocated space

File Size Optimization

To effectively reduce the file size, particularly after marking memory spaces as free using the Free method, the Shrink method can be applied to eliminate unused, freed memory at the end of the file. This action compacts the file without altering pointers or affecting the structure of stored data, ensuring the file remains as compact as possible while maintaining data integrity and access efficiency.

fmem.Shrink(); // Compact the file to optimize storage

Closure

Ensuring the closure of the Fmem object is crucial for data integrity, as it finalizes all pending operations and securely saves changes to disk.

fmem.Close(); // Finalize operations and close the file

Advanced Usage

Large Dataset Management

The HeapFile library is particularly adept at handling large datasets, thanks to its efficient memory allocation and data access methodologies, making it a suitable choice for high-performance applications.

Security Aspects

While the HeapFile library offers robust data management capabilities, it mirrors the traditional heap memory model, inheriting its potential vulnerabilities. Specifically, when dealing with sensitive data, it's crucial to manually overwrite the data with zeros or another neutral value before marking the memory space as free using the Free method. This precaution prevents data remnants from being accessible after the space is freed, addressing one of the primary security concerns associated with heap memory management. Furthermore, improper management of pointers can lead to issues such as dangling pointers or memory leaks, which are well-known risks in low-level programming environments. Users of the HeapFile library, especially those accustomed to the managed environment of C#, should be vigilant in applying these heap memory principles to safeguard against unintended data exposure and ensure proper memory hygiene.

Conclusion

Embracing the HeapFile library offers a scalable and efficient mechanism for data management within C# applications. By leveraging its comprehensive feature set, developers can significantly enhance data handling efficiency and application performance.