GrindCore 0.9.0

GrindCore

An AOT-compatible compression and hashing library built in the System.IO.Compression style—providing a managed .NET wrapper around the GrindCore Native library.

Published on NuGet as GrindCore with support for multiple platforms.

Table of Contents

• Quick Start
• Overview
• Key Features
• Compression
• Hashing
• Continuous Integration (CI) Status
• Key Project Integrations
• Why GrindCore?
• To Do
• Conclusion

Quick Start

For usage examples and API patterns, see the Getting Started Guide.

For more in-depth information, see .

NuGet

dotnet add package GrindCore

Overview

GrindCore delivers maintainable, high-performance compression and hashing for .NET by unifying multiple native C codecs into a single multiplatform library. It uses the same CMake build system as the dotnet runtime, preserving exact algorithm versions for byte-perfect, deterministic output where required.

Key Features

• Stream Position Correction: Advanced buffer management for precise stream rewinding when overreading occurs
• Multi-Framework Support: Compatible with .NET Framework 2.0 through .NET 10 net10.0;net9.0;net8.0;net7.0;net6.0;net5.0;netcoreapp3.1;netstandard2.1;netstandard2.0;net48;net47;net46;net45;net40;net35;net20
• AOT Compatible: Fully supports Ahead-of-Time compilation
• Native Performance: Native C libraries compiled from source as part of this project — no third-party binary dependencies. See GrindCore for the native build.
• Seekable Compression: ZStd seekable format for random access into compressed archives

Compression

GrindCore implements compression in two forms: Stream-based and Block-based.

• Stream-based compression follows the standard .NET approach, enabling seamless integration with existing workflows.
• Block-based compression is designed for one-shot buffer compression, providing efficient, high-performance processing for specific use cases.

All compression streams inherit from the CompressionStream class, ensuring consistent behavior and shared functionality across implementations.
Similarly, all block-based compression implementations inherit from CompressionBlock, maintaining structured handling of compression operations.

Additionally, GrindCore provides seekable compression via ZStdSeekableStream, enabling random access decompression of ZStd archives. Data is organized into independently decompressible frames with a seek table, allowing efficient seeking to any byte offset without scanning the entire archive. See the Getting Started Guide for details.

To simplify instance creation, GrindCore provides:

• CompressionStreamFactory, allowing easy instantiation of stream-based compression classes.
• CompressionBlockFactory, offering a straightforward mechanism for initializing block-based compression instances.

Supported Compression Algorithms

All native compression algorithms are directly built into the GrindCore Native project—no third-party binaries are used or required. The following algorithms are compiled from source as part of the native library, ensuring full integration, security, and maintainability:

• Brotli v1.1.0 (from .NET 9.0)
• BZip2 v1.0.8 (from bzip2) — supports multi-stream (concatenated) decompression
• Copy (no compression - direct stream copy)
• LZ4 v1.10.0 (from LZ4)
• LZMA v25.1.0 (from 7Zip App)
• LZMA2 v25.1.0 (from 7Zip App)
• Fast-LZMA2 v1.0.1 (from 7Zip-mcmilk)
• ZLib v1.3.1 (GZip, ZLib, Deflate - from .NET 8.0)
• ZLib-NG v2.2.1 (GZip, ZLib, Deflate - from .NET 9.0)
• ZStd v1.5.7 & v1.5.2 (from Facebook) — includes seekable format, skippable frame support, multithreaded compression, and dictionary support

Notes:

• Multiple versions of some algorithms (e.g., ZStd, ZLib/ZLib-NG) are included to support applications that require pinned or frozen versions, most commonly for scenarios demanding byte-perfect, deterministic outputs.
• ZStd supports multithreaded compression via ThreadCount in CompressionOptions, enabling parallel compression using multiple worker threads for significantly higher throughput on multicore systems.
• ZStd supports pre-trained dictionaries via InitProperties in CompressionOptions for improved compression of small data.
• The set of supported algorithms will continue to expand.
• Both blocking and asynchronous methods are available, allowing flexible compression workflows.
• Compression streams expose .Position (compressed) and .PositionFullSize (uncompressed) properties for accurate progress tracking.

Hashing

Hashes inherit from HashAlgorithm allowing them to be used with CryptoStream for standard .Net use.

• Blake3, Blake2sp
• MD5, MD4, MD2
• SHA1
• SHA2 [SHA256, SHA384, SHA512]
• SHA3 [SHA3-224, SHA3-256, SHA3-384, SHA3-512]
• XXHash [XXH32, XXH64]

Continuous Integration (CI) Status

A comprehensive list of test statuses for various platforms is available below. For a quick summary: GrindCore is being actively tested across major platforms including Linux ARM64, Linux ARM, Linux x64, macOS x64, macOS ARM64, Windows x64, Windows x86, and Windows ARM64.

Detailed Test Status

Platform	Unit Test Status
Linux ARM64
Linux ARM
Linux x64
macOS x64
macOS ARM64
Windows x86
Windows x64
Windows ARM64

Key Project Integrations

GrindCore integrates robust solutions from several key projects:

• dotnet Runtime GitHub Repository:
  • Provides a foundation with multiplatform C compilation based on CMake and C, ensuring seamless integration across different platforms.
  • Supplies zlib/deflate and Brotli from the dotnet 8 code, combined with C# wrappers, to offer efficient and reliable compression algorithms.
• ZStd Facebook GitHub Repository:
  • Multiplatform zstandard direct from the source.
• bzip2 Official Repository:
  • The official bzip2/libbzip2 1.0.8 source, providing block-sorting compression with streaming and one-shot block APIs.
• 7zip mcmilk GitHub Repository:
  • Contributes a comprehensive suite of hash functions, including SHA-1, SHA-2, SHA-3, MD2, MD4, MD5, and XXHash (32 and 64). More compression and hashing algorithms will be ported, benefiting from a uniform Make project structure that simplifies integration.
• 7-Zip Official:
  • LZMA and LZMA2 compression from the official 7-Zip source, along with Fast-LZMA2 from the mcmilk fork.
• GrindCore.SharpCompress:
  • Enhanced fork of SharpCompress leveraging GrindCore's native streams for improved performance and additional features like LZMA/2 level support.

Why GrindCore?

• Performance — Native C outperforms C# ports; GrindCore gives you native speed with a managed API.
• Up-to-date — Built from well-maintained upstream sources (dotnet runtime, Facebook ZStd, 7zip mcmilk) that are easy to update.
• Cross-platform — Multiplatform C compilation via dotnet's CMake system. The managed layer abstracts this so it works like System.IO.Compression.
• Deterministic — Preserves exact algorithm versions for byte-perfect checksummed output.
• Complete — Exposes functionality missing from other libraries (e.g. compress2 from zlib, LZMA levels, ZStd dictionaries and seekable format).

To Do

Several enhancements and additional features could be introduced to further improve GrindCore. While these may be addressed over time, listing them here serves to communicate known gaps and encourage community contributions:

• Multi-language support.
• Custom Dictionaries and training where supported. ✅ ZStd dictionary support added.
• Progress updates raised from C library.
• Update native compression versions.
• Expanded compression algorithm capabilities.
  • If you identify missing features, feel free to raise issues or submit pull requests.
  • Any unimplemented methods from the C source can be exposed upon request.

Conclusion

GrindCore is on a journey to create a more reliable and efficient compression solution for dotnet. The community's contributions and collaboration are welcomed.