rkrtechnology/litestream
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
369 Commits 6 Branches 29 Tags
46597ab22fff45cc35dd76ce5e15c76fc7d3befc
Commit Graph

369 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Hiroaki Nakamura
46597ab22f Fix wal internal error log 2022-05-13 15:25:43 -06:00
Hiroaki Nakamura
e6f7c6052d Add two environments for overriding endpoint and region 
export LITESTREAM_ACCESS_KEY_ID=your_key_id
export LITESTREAM_SECRET_ACCESS_KEY=your_access_key
export LITESTREAM_ENDPOINT=your_endpoint
export LITESTREAM_REGION=your_region
litestream replicate fruits.db s3://mybkt/fruits.db
 2022-05-10 08:48:48 -06:00
Ben Johnson
7d8b8c6ec0 Remove verbose flag from restore docs 2022-05-03 06:33:53 -07:00
Michael Lynch
88737d7164 Add a unit test for internal.MD5Hash 2022-04-17 15:02:37 -06:00
Michael Lynch
6763e9218c Fix path to coverage file 2022-04-17 15:02:32 -06:00
Michael Lynch
301e1172fd Add Go code coverage to CI 2022-04-17 15:02:32 -06:00
Ben Johnson
ca07137d32 Re-add point-in-time restore 2022-04-14 20:03:52 -06:00
Ben Johnson
80f8de4d9e Fix release workflow v0.4.0-beta.2 2022-04-09 10:34:26 -06:00
Ben Johnson
5d394bbc57 Document -addr flag on replicate command v0.4.0-beta.1 2022-04-05 13:48:15 -06:00
Ben Johnson
f53857e1ad Add minimum shadow WAL retention 2022-04-04 21:25:31 -06:00
Ben Johnson
44662022fa Allow read replication recovery from last position 2022-04-04 20:19:02 -06:00
Ben Johnson
2c3e28c786 Improve http error logging 2022-04-03 11:55:42 -06:00
Ben Johnson
46888530b2 Default upstream path if not specified 2022-04-03 09:15:54 -06:00
Ben Johnson
6aba416656 Remove CI task for executing long running test runner on each build 2022-04-02 11:53:22 -06:00
Ben Johnson
8d10881278 Use database page size in read replication 2022-04-02 11:50:30 -06:00
Ben Johnson
00bad4308d Set permission on file replica client on init 2022-03-06 08:38:07 -07:00
Ben Johnson
d5792c42b9 Prevent double-close for SFTP client 2022-03-05 11:33:34 -07:00
Ben Johnson
07d220028a Rename 'gcs' to 'gs' for consistency 2022-03-05 11:17:42 -07:00
Ben Johnson
8ee5fcb591 Read config file from present working directory, if present 2022-03-05 11:07:49 -07:00
Ben Johnson
7fe79d3883 Add -addr flag to replicate command 2022-03-05 09:55:15 -07:00
Ben Johnson
14026421b2 Disable dependabot 2022-03-05 08:55:50 -07:00
Ben Johnson
59de3a01ba Upgrade mattn/go-sqlite3 to v1.14.12 2022-03-05 08:53:03 -07:00
Ben Johnson
c435b6b672 Pass first DB path to child process 2022-03-05 08:44:11 -07:00
Ben Johnson
62e301afd0 Change dependabot from weekly to monthly 2022-02-26 08:45:34 -07:00
Tobias Nießen
06ea1b13c1 Improve iterator Next() descriptions 2022-02-26 08:41:30 -07:00
Ben Johnson
a090706421 Implement live read replication 
This commit adds an http server and client for streaming snapshots
and WAL pages from an upstream Litestream primary to a read-only
replica.
 2022-02-19 09:06:49 -07:00
Ben Johnson
4898fc2fc1 Remove Docker linux/arm64 for PR builds 2022-02-18 14:39:59 -07:00
Ben Johnson
6f8cd5a9c4 Configurable monitor-delay-interval 
The `monitor-delay-interval` has been added to the DB config so that
users can change the time period between WAL checks after a file
change notification has occurred. This can be useful to batch up
changes in larger files in the shadow WAL or to reduce or eliminate
the delay in propagating changes during read replication.

Setting the interval to zero or less will disable it.
 2022-02-18 14:38:50 -07:00
Ben Johnson
4027c87a02 Fix Docker arch mismatch 2022-02-15 16:02:02 -07:00
Ben Johnson
fde17d0e62 Upgrade dependencies 2022-02-15 13:53:59 -07:00
Ben Johnson
fc42576e47 Add Docker arm/v7 to CI 2022-02-15 12:31:39 -07:00
Campbell Vertesi
1a630aed04 Add docker multiarch build and push to release 
Co-authored-by: Ben Johnson <benbjohnson@yahoo.com>
 2022-02-15 12:08:21 -07:00
Ben Johnson
8589111717 Implement streaming WAL segment iterator 
Currently, WALSegmentIterator implementations read to the end of
the end of their list of segments and return EOF. This commit adds
the ability to push additional segments to in-process iterators and
notify their callers that new segments are available. This is only
implemented for the file-based iterator but other segment iterators
may get this implementation in the future or have a wrapping
iterator provide a polling-based implementation.
 2022-02-11 13:50:44 -07:00
Ben Johnson
006e4b7155 Update index & offset encoding 
Previously, the index & offsets were encoded as 8-character hex
strings, however, this limits the maximum value to a `uint32`. This
is normally not an issue, however, indices could go over the maximum
value of 4 billion over time and the offset could exceed this value
for an especially large WAL update. For safety, these encodings have
been updated to 16-character hex encodings.
 2022-02-08 13:14:49 -07:00
Ben Johnson
54f3b94d3f Upgrade dependencies 
- github.com/aws/aws-sdk-go v1.42.44 => v1.42.48
- cloud.google.com/go/storage v1.19.0 => v1.20.0
- github.com/pierrec/lz4/v4 v4.1.12 => v4.1.14
- google.golang.org/api v0.66.0 => v0.67.0
 2022-02-07 14:21:52 -07:00
Ben Johnson
30a8d07a81 Add WAL overrun validation 
Under high write load, it is possible for write transactions from
another process to overrun the WAL between the time when Litestream
performs a RESTART checkpoint and when it obtains the write lock
immediately after. This change adds validation that an overrun has
not occurred and, if it has, it will start a new generation.
 2022-02-07 13:35:20 -07:00
Ben Johnson
76e53dc6ea Remove built-in validation option 
Previously, Litestream had a validator that worked most of the time
but also caused some false positives. It is difficult to provide
validation from with Litestream without controlling outside processes
that can also affect the database. As such, validation has been moved
out to the external CI test runner which provides a more consistent
validation process.
 2022-02-06 11:37:06 -07:00
Ben Johnson
762c7ae531 Implement FileWatcher 2022-02-06 09:51:04 -07:00
Ben Johnson
8009bcf654 Remove Windows support 
Unfortunately, I don't have the expertise or bandwidth to maintain
the Windows support in Litestream. I'm open to re-adding support in
the future but right now it is hindering development and is not
well-tested or well-used.
 2022-02-05 08:19:31 -07:00
Ben Johnson
4349398ff5 Remove shadow WAL iterator 
This commit removes the shadow WAL iterator and replaces it with a
fileWalSegmentIterator instead. This works since the shadow WAL now
has the same structure as the replica WAL. This reduces duplicate
code and will make it so read replication can be daisy chained in
the future.
 2022-01-31 16:09:02 -07:00
dependabot[bot]
89560c8632 Bump github.com/prometheus/client_golang from 1.12.0 to 1.12.1 
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 14:27:38 -07:00
dependabot[bot]
5f38134032 Bump cloud.google.com/go/storage from 1.18.2 to 1.19.0 
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.18.2 to 1.19.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.18.2...spanner/v1.19.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:55:52 -07:00
dependabot[bot]
d5c15593bb Bump google.golang.org/api from 0.65.0 to 0.66.0 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.65.0 to 0.66.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.65.0...v0.66.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:50:18 -07:00
dependabot[bot]
fb3a3d904f Bump github.com/aws/aws-sdk-go from 1.42.40 to 1.42.44 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.40 to 1.42.44.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.40...v1.42.44)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:15:13 -07:00
Ben Johnson
ee77592d7e Skip dependabot CI using branches 2022-01-31 13:01:24 -07:00
Ben Johnson
a2cf2e260b Skip some CI jobs for dependabot 2022-01-31 12:46:06 -07:00
Ben Johnson
5d811f2e39 Fix golangci-lint issues 2022-01-31 09:21:20 -07:00
Ben Johnson
e84994ad95 Add golangci-lint to CI 2022-01-31 09:21:20 -07:00
Ben Johnson
f6c859061b Fix CodeQL warnings 2022-01-31 08:53:21 -07:00
Ben Johnson
0dfa5f98d1 Re-enable SFTP integration tests 2022-01-30 09:15:31 -07:00