NewLineReader 1.0.12

NewLineReader

NewLineReader is a C# package that provides efficient and flexible line-by-line reading capabilities for large text files. It's designed to handle files of any size while maintaining low memory usage.

Features

• Efficient line-by-line reading of text files
• Support for random access to lines by index
• Low memory footprint, suitable for large files
• Customizable line ending detection

Installation

[Add installation instructions here, e.g., NuGet package information]

Usage

The main class in this package is PositionalNewLineReader. Here's how to use it:

using NewLineReader;

var filePath = "path/sample3.txt";

long position = 0;
var numLinesToRead = 1; // Specify the number of lines to read

for (var i = 0; i < 21; i++)
{
    var (success, result, lines, nextPosition) =
        PositionalNewLineReader.TryReadNextLines(filePath, position, numLinesToRead);

    switch (result)
    {
        case ReadResult.Success:
            Console.WriteLine("Lines:");
            foreach (var line in lines) Console.WriteLine($"-----xx-----{line}");
            Console.WriteLine($"Next Position: {nextPosition}");
            position = nextPosition;
            break;
        case ReadResult.NotAtLineStart:
            Console.WriteLine(
                $"Error: The provided position {position} is not at the start of a line. Please provide a position at the start of a line.");
            break;
        case ReadResult.NegativePosition:
            Console.WriteLine($"Error: Negative position {nextPosition} is invalid.");
            break;
        case ReadResult.EndOfFile:
            Console.WriteLine($"Error: The provided position {nextPosition} is at the end of the file.");
            break;
        case ReadResult.BeyondEndOfFile:
            Console.WriteLine(
                $"Error: The provided position {position} is beyond the end of the file. File ends at {nextPosition}.");
            break;
    }
}

Console.WriteLine(new FileInfo(filePath).Length + " bytes in total.");

Console.WriteLine("First New Line = {0}",
    PositionalNewLineReader.TryGetNextNewLine(filePath, 0).line);

Console.WriteLine("First New Line Position = {0}",
    PositionalNewLineReader.TryGetNextNewLinePosition(filePath, 0).nextPosition);

Performance Considerations

NewLineReader is designed with performance in mind, especially when dealing with large files:

1. Efficient Memory Usage: Instead of loading the entire file into memory, NewLineReader reads only the requested lines, making it suitable for very large files.

2. Random Access: The TryReadNextLines method allows for efficient random access to any part of the file by specifying the starting position.

3. Batched Reading: By allowing users to specify the number of lines to read in each call, NewLineReader enables efficient batched processing of file contents.

4. No Index Building: Unlike some other line readers, NewLineReader doesn't build an index of all line positions upfront. This means it can start reading from files instantly, without any preprocessing time.

5. Minimal Seeking: The method returns the next position after reading, allowing for sequential reads with minimal file seeking operations.

6. Error Handling: The use of a tuple return value with a success flag and error message allows for efficient error handling without exceptions.

When using NewLineReader, consider the following for optimal performance:

• Choose an appropriate batch size (numLinesToRead) based on your specific use case and available memory.
• For sequential reading of the entire file, reuse the nextPosition returned from each call to minimize seeking operations.
• If you need to repeatedly access the same file, consider caching line positions for frequently accessed lines to avoid re-reading from the beginning.

Maximum File Size Supported

NewLineReader utilizes the long data type in .NET 6 to handle file positions and sizes. This allows for extremely large file support. The maximum value of a long in C# is 9,223,372,036,854,775,807, which translates to a maximum file size of approximately 9.22 exabytes (EB).

To put this size into perspective:

The maximum supported file size of 9.22 EB is an incredibly large amount of data, far exceeding the capacity of current storage systems. In practice, NewLineReader can handle any realistically sized file you're likely to encounter.

Note: While NewLineReader can theoretically handle files up to this size, other system limitations (such as available storage, file system constraints, or memory) may become limiting factors before reaching this theoretical maximum.

For any questions or clarifications about file size handling in NewLineReader, please feel free to open an issue or contact the maintainers.

Contributing

Contributions to NewLineReader are welcome! If you'd like to contribute, please follow these steps:

1. Fork the repository
2. Create a new branch for your feature or bug fix
3. Make your changes and commit them with clear, descriptive messages
4. Push your changes to your fork
5. Submit a pull request to the main repository

Please ensure your code adheres to the existing style and include appropriate tests for new features or bug fixes.

License

NewLineReader is released under the MIT License.