The `monitor-delay-interval` has been added to the DB config so that
users can change the time period between WAL checks after a file
change notification has occurred. This can be useful to batch up
changes in larger files in the shadow WAL or to reduce or eliminate
the delay in propagating changes during read replication.
Setting the interval to zero or less will disable it.
Previously, the index & offsets were encoded as 8-character hex
strings, however, this limits the maximum value to a `uint32`. This
is normally not an issue, however, indices could go over the maximum
value of 4 billion over time and the offset could exceed this value
for an especially large WAL update. For safety, these encodings have
been updated to 16-character hex encodings.
Unfortunately, I don't have the expertise or bandwidth to maintain
the Windows support in Litestream. I'm open to re-adding support in
the future but right now it is hindering development and is not
well-tested or well-used.
Previously, Litestream would avoid closing the SQLite3 connection
in order to ensure that the WAL file was not cleaned up by the
database if it was the last connection. This commit changes the
behavior by introducing a file control call to perform the same
action. This allows us to close the database file normally in all
cases.
This commit refactors out the complexity of downloading ordered WAL
files in parallel to a type called `WALDownloader`. This makes it
easier to test the restore separately from the download.
By default, the snapshots command seems to output in alphabetical order of hash, which isn't meaningful, as far as I can tell.
This change modifies the order of the command output so that ./litestream snapshots returns snapshots from newest to oldest.
This commit adds the ability to run a subcommand through Litestream.
Shutting down the subcommand will cause Litestream to gracefully
shutdown. Litestream will forward interrupt signals and wait for
the subprocess to shutdown.
This commit adds a flag to `litestream restore` to skip the restore
if the database file already exists. It is useful when using the
official Litestream Docker image and you don't have the ability to
add a script to check for the existence of the file.
Previously, S3 would default to sync every 10s when using a config
file but only every 1s when using the replica URL on the command
line. Obviously, this is confusing.
A sync interval of 1s could incur a cost of $1.30/month, however,
in practice applications are not receiving a constant stream of
writes so the cost is typically only a few pennies. This lower
sync interval will provide a smaller window for data loss in the
event of a catastrophic failure at a neglible cost.
This commit adds optional `LITESTREAM_ACCESS_KEY_ID` and
`LITESTREAM_SECRET_ACCESS_KEY` environment variables that can
be used instead of their `AWS` counterparts. The AWS-prefixed
variables have caused some confusion with users who were not
using AWS S3.
This commit changes the restore to download multiple WAL files to
the local disk in parallel while another goroutine applies those
files in order. Downloading & applying the WAL files in serial
reduces the total throughput as WAL files are typically made up of
multiple small files.
This commit changes the `-index` flag on the `restore` command by
parsing it as a hex number instead of a decimal number. This is
done because the index is represented in hex form everywhere else
in the application.
This commit adds simple variable expansion using either `$FOO`
or `${FOO}` when evaluating the config file. This can be disabled
by any command by using the `-no-expand-env` flag.
This commit adds a `skip-verify` flag to the replica configuration
so that it can be used with self-signed certificates. This is useful
when running a local instance of MinIO with TLS for testing.
This commit changes the signal handler for `replicate` to catch
`syscall.SIGTERM` for non-Windows installations. It also adds some
logging to indicat when a shutdown has been initiated and when it
has finished.
This commit adds a flag to change the exit code when restoring
from a replica where there is no existing backup. When set,
finding no backup will return a `0` exit code. The command will
still fail if other errors occur.
This commit changes the replica configuration behavior to default
the `force-path-style` field to `true` when an `endpoint` is set.
This works because the only service that does not use the path
style is AWS S3 which does not use an endpoint.
This commit exposes the monitor interval, checkpoint interval,
minimum checkpoint page count, and maximum checkpoint page count
via the YAML configuration file.
This commit fixes a bug where the bind address is not reported
correctly in the log if a hostname is specified. Previously it
would always report the host as "localhost" even if a host was
specified (such as "0.0.0.0:9090").
This commit also adds validation to require the port to be
specified and only specifying a hostname will return an error.
Previously, if a replica URL was specified then the `-config` flag
was silently ignored. This commit changes this behavior so that
specifying both the URL & config flag will now return an error.
This commit adds the ability to periodically perform snapshots on
an interval that is separate from retention. For example, this lets
you retain backups for 24 hours but you can snapshot your database
every six hours to improve recovery time.
This commits adds support for non-AWS S3-compatible storage such as
MinIO, Backblaze B2, & Google Cloud Storage (GCS). Other backends
should also work but some code has been added to make URL-based
configurations work more easily.
This commit changes the default sync interval from 10s to 1s
when replicating using the inline replica URL. This approach is
used when users are first testing the software so a faster
replication interval makes it easier to see results.
This commit removes the verbose flag (`-v`) and replaces it with
the trace flag (`-trace PATH`). This moves tracing to a separate
file instead of writing to STDOUT.
Previously, there were excessive log messages for checkpoints and
retention. These have been removed or combined into a single log
message where appropriate.
