dtpipe 1.10.0

DtPipe

A self-contained CLI for streaming, transforming, and anonymizing data across databases and file formats.

DtPipe reads from a source, applies row and columnar transformations in batches, and writes to a destination with no intermediate staging. It is designed for automation and CI/CD workflows where repeatable, observable data pipelines matter.

📚 Start with the documentation → docs/

Install, a first pipeline, the concepts, and a guide per task — anonymization, joins and fan-out, incremental loading, YAML jobs, secrets, and running in a .NET estate.

Install

dotnet tool install -g dtpipe        # or download a binary from the releases page
dtpipe --help

Six platforms per release — Linux, macOS and Windows, x64 and arm64 — each a single executable with the .NET runtime, the DuckDB engine and every database driver inside it. Nothing else to install. Full instructions, including building from source: docs/install.md.

One example

Anonymize a table on the way out, without ever writing the clear values to disk:

dtpipe \
  -i "pg:Host=localhost;Database=prod;Username=postgres" \
  --query "SELECT * FROM users" \
  --fake "email:internet.email" \
  --fake "name:name.fullName" \
  --mask "phone:###-****" \
  --null "ssn" \
  -o anonymized_users.parquet

More, from a first pipeline to a DAG: docs/quickstart.md.

Studio

dtpipe studio ./jobs opens a local page over the YAML jobs of a folder: each job as cards you edit and sample on the real engine, with the target never written. It is part of the CLI and needs no coordinator; --coordinator only adds the bricks and the pipeline library of a distributed setup. Details in REFERENCE.md.

Not everything in this release is stable. The distributed pipeline (coordinator, pipeline nodes, runner) and dtpipe studio are new: they may change or disappear without notice, and the distributed projects ship in no package. Do not build a workload on them yet. What was there before, reading, transforming and writing through the CLI, a job file or the MCP server, is versioned as it always was. The distributed family has not been run on Windows; see docs/experimentation.md.

What it connects to

dtpipe providers lists what your binary actually carries. The connection catalog, with the string format and the rules that apply to every provider, is at docs/connections/.

Is DuckDB alone enough? Often, yes — and then use it directly. DtPipe embeds it as its SQL engine and adds what a transfer needs around it: anonymization in transit, concurrent fan-out, write strategies (upsert, auto-migrate, bulk), Oracle / SQL Server / XML sources, and repeatable YAML jobs with secret management.

Documentation

Contributing

See EXTENDING.md for the adapter and transformer patterns.

License

MIT