FileSync 4.0.5

FileSync

A cross-platform command-line file-syncer, which can sync files from a local source directory, with glob filters, and backup to either another directory, or to cloud storage, namely to Azure blob storage. (Though stalled, it should be trivial in the future to expanded this to Amazon S3 and hopefully even to FTP)

Many years prior I named this tool FileUp.

dotnet tool

Currently the source for this tool is not released. Till then however this gist / readme provides needed documentation.

See nuget for latest version, but installation is as simple as:

dotnet tool install --global FileSync --version 4.0.2

Use

Preliminary notes

1. For better or for worse, FileSync (FS) runs continuously until you close the console window running it. I think I regret this now, to do it all over I think I would prefer to have it run and immediately close like most command-line apps.

Help

Run input help, current output:

help                    Display help info / CLI commands
new                     Create a new `.filesync` settings file. In working directory, or abs or rel directory that follows, e.g.: `new ../foo/bar/` If one already exists in location, will NOT overwrite but will make a randomly generated sample file prefix
sync                    Runs an update / sync
check                   Runs sync but without doing the actual upload / syncing, to 'check' what would be updated if `sync` is run
clean                   Deletes any orphaned files on destination that don't exist any longer in source. By default a prompt is given after files to delete are found.
clean!                  Same as `clean` but ignores `promptOnDelete` if it was enabled
clean-trial             Same as `clean` but stops before carrying out the clean, allowing one to see a display of orphaned files on dest
watch                   Starts a file-system watch to auto-upload / sync changed files


verbose                 To turn on verbose
verbose-off             To turn off verbose
-since                      examples:
                               `-since 1h`    = 1 hour
                               `-since 10m`   = 10 minutes
                               `-since 2.5d`  = 2.5 days
                                When set causes any **source** files to be ignored / excluded from
                                updates whose Last-Modified file time is older than current-time minus
                                specified span. This might be useful when for instance, you have 10,000 files in source,
                                and only are concerned about updating files that you're currently
                                actively working on. Examples (the following values work identically when
                                set on `since` property in `.filesync`, e.g. `"since": "1h"`:

+++ Settings Init +++   Set these (only allowed once at statup) to setup FileSync via a single
                           command-line command. Otherwise must send in a settings.json
-type                   Sync type, currently: `directory`, `azure` or `amazon` (`default` defaults to `directory`)
-src                    Source directory path. Absolute, or relative to `root-dir`. Root however often will be set automatically by command-line call.
-dest                   Destination path. For cloud storage, dest could be left null, otherwise this value will be set as the subdirectory path following `container` value.
-filters                One or more glob patterns (`;` **semicolon** separated if more than one) to filter what `src` files get backed up. Filters can be one or more glob patterns (https://w.wiki/4Wsx). MAY be [bracketed], individual values MAY be in 'quotes' or "quotes": all these will be trimmed regardless
-ex-filters             One or more glob patterns to *exclude* from any matching `src` files.
-creds                  A dynamic type: For azure can be their storage connection string (which is just semicolon separated key-value pairs). It can also be anything of what `creds-path` accepts, see documentation elsewhere. This value does NOT have to be set also if credential values were input elsewhere (this is true for the other following settings).
-creds-path             In order to allow the storage key to be out of source control while keeping `.filesync` settings in, this setting allows a path to be entered for account name/key. This path can be relative to `root-dir`. When read, the file contents will be set to `creds`. The contents can be for instance an azure storage credentials connection string, or some other simpler things, like it can simply be a single line of the key, see further documentation elsewhere
-parallel               Set true or false to do syncs in parallel
-container              For cloud storage: the container / bucket name
-account-name           For cloud storage, the storage account name (this can also be set elsewhere)
-account-key            For cloud storage: The access key for this storage account (this can also be set elsewhere)
-root-dir               If set, other paths can be relative to it, such as `src` and `dest`

Initialization / Setup

1. At startup, file-sync must be initialized with settings, specifying source directory location, destination location, type of syncing to be done, etc. These settings can be input either as command-line input arguments (see help), or more preferably often, with a json file (ideally but not necessarily with extension .filesync or simply named that. Run new to generate a sample .filesync settings file. Here's a sample file that will be created (within the working root directory):

{
  "type": "directory",
  "src": "src",
  "dest": "dist",
  "filters": "**/*.{txt,css,js,jpg,png}",
  "exFilters": "**/*.{gif,xml}",
  "container": null,
  "creds": null,
  "credsPath": null,
  "parallel": null,
  "accountName": null,
  "accountKey": null,
  "promptOnDelete": null,
  "root": null,
  "sync": null,
  "watch": null,
  "since": null,
  "notes": "Filters can be one or more glob patterns (https://w.wiki/4Wsx), semicolon-separated. MAY be [bracketed], individual values MAY be in 'quotes' or \"quotes\": all these will be trimmed regardless"
}

Here's a fuller example .filesync file:

{
  "type": "azure",
  "src": "alpha",
  "dest": "dist",
  "filters": "**/*.{txt,css,js,jpg,png,xml}",
  "exFilters": "**/*.{gif,xml}",
  "container": "foo",
  "creds": "foobar;aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==",
}

Credential settings

When syncing to Azure / cloud for Azure's blob storage, there needs to be set:

• accountName
• accountKey
• container name.

So consider an Azure blob storage account credentials string:

DefaultEndpointsProtocol=https;AccountName=foobar;AccountKey=aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==;EndpointSuffix=core.windows.net

Note how these are just key-value pairs with an = equals sign, and kvs separated by semicolons ;. To sync to a storage account you need the three things listed above. So these values can be set directly in the file settings above, by setting:

• "container", "accountName", and "accountKey" properties. But you can also set the creds property with the full Azure connection string instead (in which case "container" still needs to be set). However we allow also a "credsPath" property, allowing a path (if relative from root) to a file that contains credential information. This allows source control to save a .filesync settings file, but without saving the key. Acceptable values in that file (or in the creds property value) are as follows:

"credsPath" can contain the following content:

1. A single line representing the storage key

aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==

2. Two lines with the top line being accountName and second line accountKey

foobar
aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==

Though in all cases new lines can work instead of semicolons for key-value pair separators. In processing, semicolons are just replaced first with \n new lines. So above (and in all cases below) could be as well:

foobar;aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==

3. A full azure storage credentials string (as showed above), note again new-lines would also work:

DefaultEndpointsProtocol=https;AccountName=foobar;AccountKey=aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==;EndpointSuffix=core.windows.net

Or with new-lines:

DefaultEndpointsProtocol=https
AccountName=foobar
AccountKey=aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==
EndpointSuffix=core.windows.net

In fact we are just parsing out key-value pairs for this, so other key names will just be ignored, and order doesn't matter. In fact we ignore DefaultEndpointsProtocol and EndpointSuffix right now when connecting to Azure. Also for AccountName instead name can be used, and instead of AccountKey can be used key, e.g.:

name=foobar
key=aaaa+bbbbbcccccdddddeeeeefffffggggghhhhhiiiiijjjjjkkkkklllllmmmmmnnnnn0123456789uvwxyz==
container=foo

We also allow an additional key / value type, container (see above), if you wanted the container to come through here.

If settings were established a message something like the following will be shown:

+++ Settings established +++
  * type: Azure
  * src : "C:/foo/bar"
  * glob filter(s)   : "**/*.{txt,css,js,jpg,png,xml}"
  * exclude filter(s): "**/*.{gif,xml}"
 ++ destination ++
  * account name: "foobar"
  * container   : "foo"
  * dest        : "dist"

Install as global dotnet tool

For installation, the easiest way is probably with dotnet global tools, see here and here

Maintainers: Pack project to a nuget package (.nupkg)

See here to pack a project. Note that PackAsTool is set in FileSync.csproj, as well as <PackageOutputPath>./../nupkg</PackageOutputPath>. Or (at least in some cases) you can just right click the project in VStudio and select "Pack" to have it build to that location.

Install

Example installation where final nupkg directory has the built nuget package file FileSync.2.4.0.nupkg:

dotnet tool install filesync -g --add-source 'C:\<path>\FileSync\nupkg\'

For --add-source, it has to be a directory, which is quite annoying, as you have remove all but one built .nupkg.

Some other commands to be aware of:

--verbosity detailed "q[uiet], m[inimal], n[ormal], d[etailed], and diag[nostic]."

--ignore-failed-sources "Treat package source failures as warnings" ... This has proved important before, if any nuget public sources conflict, etc.