FilePilot 1.5.0

Introduction

FilePilot provides the ability to read and write CSV and fixed-width content in files. FilePilot also includes an efficient file reader component with cursor/position management.

Mapping

To map between file contents and objects in code, a template needs to be created which represents the fields in a file. The 'PilotField' Data Annotation needs to be added to each field, along with the respective index of that field in the field.

For CSV files, only the index of each field needs to be specified. The zero-based index represents the position of the field in the CSV line.

    public class DemoFileMap
    {
        [PilotField(0)]
        public string SomeField { get; set; }
        
        [PilotField(1)]
        public string AnotherField { get; set; }
    }

For fixed-width files, both the start index and length of each field needs to be specified. The zero-based start index represents the string index position of a field in a line.

Note that 'gaps' can exist between defined fields.

    public class DemoFileMap
    {
        [PilotField(0, 10)]
        public string SomeField { get; set; }
        
        [PilotField(10, 20)]
        public string AnotherField { get; set; }
    }

Fields can also be decorated with the 'PilotFieldFormat' data annotation to specify certain standard properties, such as padding. Fields can also be nullable and represented by different primitive data types.

The below example demonstrates the 'PilotFieldFormat' usage with all of the available options set to their defaults.

    public class DemoFileMap
    {
        [PilotField(10, 20)]
        [PilotFieldFormat(trim: false, paddingType: PaddingType.None, paddingChar: ' ', allowTruncate: false)]
        public string SampleField { get; set; }        
    }

The code required to map to and from objects is shown below.

    //Create a mock record
    var item = new DemoFileMap { TextField = "ABCDEFG" };

    //A FilePilot mapper instance is required
    IPilotMapper mapper = new PilotMapper();
    
    //Convert from an object to file contents
    var contentForFile = mapper.MapToString(item, MappingType.FixedWidth);
    
    //Convert back from a string to an object
    var objectFromFile = mapper.MapToObject<DemoFileMap>(contentForFile, MappingType.FixedWidth);

Care should be taken when dealing with fields such as DateTime, because the default ToString() and Parse() will be used when mapping to and from a file and may result in unexpected results. One approach to deal with the predictability of field values is to make a field a 'string' and manage the conversions yourself. A second approach is to use a custom converter. A custom converter named 'PilotFieldYyyymmdd' has been included with FilePilot, as shown below.

    public class DemoFileMap
    {
        [PilotField(0, 8)]
        [PilotFieldYyyymmdd]
        public DateTime? SomeDate { get; set; }
    }

To create your own converter, you'll need to create a class and inherit from the PilotFieldConverter class. The below custom converter converts text to upper case.

    public class MyUppercaseField : PilotFieldConverter
    {
        public override object ConvertRead(string value)
        {
            if (string.IsNullOrWhiteSpace(value))
                return value;
            return value.ToUpper();
        }

        public override string ConvertWrite(object value)
        {
            if (value == null)
                return string.Empty;
            return value.ToString().ToUpper();
        }
    }

Reader

The File Pilot reader (PilotReader) safely manages the iteration of lines in a file, without loading the entire file into memory. There are also some additional useful properties returned, as demonstrated in the code sample below.

    var reader = new PilotReader();
    var result = reader.ReadNextLine(sampleFilePath);
    if (!result.DidRead)
    {
        Console.WriteLine("End of file");
    }
    else
    {
        Console.WriteLine($"Line Number: {result.LineNumber}");
        Console.WriteLine($"Line Content: {result.Content}");
        Console.WriteLine($"Is Last Line: {result.IsLastLine}");
        Console.WriteLine($"Is First Line: {result.IsFirstLine}");
    }

The FilePilot reader comes coupled with a cursor manager (PilotCursorManager). The reader will create its own internal cursor manager if an explicit instance isn't provided as an overload. Any active instance of a file cursor will keep a file lock open on the file being read.

The overload parameter for the cursor manager should be used if there is a requirement to extend the lifetime of internally managed read positions. A large file is usually read in batches, where each batch is processed in a scoped lifetime, but a global/singleton cursor manager is required to improve performance when continuing from the last position read in a file.

    var reader = new PilotReader();
    var cursorManager = new PilotCursorManager();
    
    var lastLineRead = 0; //Fetch lastLineRead from persistence
    var endOfFile = false;
    for (var count = 1; count <= 100; count++) //Read 100 lines at a time
    {
        var result = reader.ReadNextLine(sampleFilePath, lastLineRead + 1, cursorManager);
        if (!result.DidRead)
        {
            endOfFile = true;
            break;
        }

        lastLineRead = result.LineNumber;
    }
    
    //Save lastLineRead to persistence

    //If a user provided PilotCursorManager is alive for extended periods of time,
    //then it's best to expire in-memory cursors that never terminated successfully
    cursorManager.ExpireCursors(CursorExpiryOption.Accessed, TimeSpan.FromMinutes(5));

If a PilotReader's ReadNextLine() method is used without supplying an explicit PilotCursorManager instance, then an internal PilotCursorManager instance will be created within the PilotReader instance. To access an internal cursor manager instance, you can use the method GetLocalCursorManager().

The PilotReader is disposable and can be used with a 'using' statement. If no explicit PilotCursorManager is used when reading from a file, then the default internal cursor manager will be disposed of when the PilotReader is disposed of.

    using (var reader = new PilotReader())
    {
        //Your logic here
    }

Read cursors will be disposed of automatically under the following circumstances:

• When the end of a file is reached when using ReadNextLine()
• When a cursor is removed using ExpireCursors() on the PilotCursorManager instance

A cursor can also be disposed of manually by calling RemoveCursor() on the cursor manager instance.

Fetching Files

The PilotReader class has a method named FetchFiles(), which fetches all files in a directory as well as some additional information relating to each file. The below properties can be found on the results returned from the FetchFiles (and LoadFile) methods.

    public string Name { get; }
    public string Extension { get; }
    public string Directory { get; }
    public string FullPath { get; }
    public Guid ContentHash { get; }
    public Guid FullPathHash { get; }
    public DateTime LastAccessedAt { get; }
    public DateTime LastModifiedAt { get; }
    public DateTime CreatedAt { get; }
    public bool FileLocked { get; }

An MD5 hash is computed on both the full path of the file and another MD5 hash is computed from the contents of each file. These hashes will be returned in the FullPathHash and ContentHash properties. If a file is locked, then the ContentHash property will be an empty Guid, and the FileLocked property will be true.