rkrtechnology/litestream
2
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
339 Commits 6 Branches 29 Tags
fc42576e47a2e5c5151f4a53048e08f79134a5b9
Commit Graph

339 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Ben Johnson
fc42576e47 Add Docker arm/v7 to CI 2022-02-15 12:31:39 -07:00
Campbell Vertesi
1a630aed04 Add docker multiarch build and push to release 
Co-authored-by: Ben Johnson <benbjohnson@yahoo.com>
 2022-02-15 12:08:21 -07:00
Ben Johnson
8589111717 Implement streaming WAL segment iterator 
Currently, WALSegmentIterator implementations read to the end of
the end of their list of segments and return EOF. This commit adds
the ability to push additional segments to in-process iterators and
notify their callers that new segments are available. This is only
implemented for the file-based iterator but other segment iterators
may get this implementation in the future or have a wrapping
iterator provide a polling-based implementation.
 2022-02-11 13:50:44 -07:00
Ben Johnson
006e4b7155 Update index & offset encoding 
Previously, the index & offsets were encoded as 8-character hex
strings, however, this limits the maximum value to a `uint32`. This
is normally not an issue, however, indices could go over the maximum
value of 4 billion over time and the offset could exceed this value
for an especially large WAL update. For safety, these encodings have
been updated to 16-character hex encodings.
 2022-02-08 13:14:49 -07:00
Ben Johnson
54f3b94d3f Upgrade dependencies 
- github.com/aws/aws-sdk-go v1.42.44 => v1.42.48
- cloud.google.com/go/storage v1.19.0 => v1.20.0
- github.com/pierrec/lz4/v4 v4.1.12 => v4.1.14
- google.golang.org/api v0.66.0 => v0.67.0
 2022-02-07 14:21:52 -07:00
Ben Johnson
30a8d07a81 Add WAL overrun validation 
Under high write load, it is possible for write transactions from
another process to overrun the WAL between the time when Litestream
performs a RESTART checkpoint and when it obtains the write lock
immediately after. This change adds validation that an overrun has
not occurred and, if it has, it will start a new generation.
 2022-02-07 13:35:20 -07:00
Ben Johnson
76e53dc6ea Remove built-in validation option 
Previously, Litestream had a validator that worked most of the time
but also caused some false positives. It is difficult to provide
validation from with Litestream without controlling outside processes
that can also affect the database. As such, validation has been moved
out to the external CI test runner which provides a more consistent
validation process.
 2022-02-06 11:37:06 -07:00
Ben Johnson
762c7ae531 Implement FileWatcher 2022-02-06 09:51:04 -07:00
Ben Johnson
8009bcf654 Remove Windows support 
Unfortunately, I don't have the expertise or bandwidth to maintain
the Windows support in Litestream. I'm open to re-adding support in
the future but right now it is hindering development and is not
well-tested or well-used.
 2022-02-05 08:19:31 -07:00
Ben Johnson
4349398ff5 Remove shadow WAL iterator 
This commit removes the shadow WAL iterator and replaces it with a
fileWalSegmentIterator instead. This works since the shadow WAL now
has the same structure as the replica WAL. This reduces duplicate
code and will make it so read replication can be daisy chained in
the future.
 2022-01-31 16:09:02 -07:00
dependabot[bot]
89560c8632 Bump github.com/prometheus/client_golang from 1.12.0 to 1.12.1 
Bumps [github.com/prometheus/client_golang](https://github.com/prometheus/client_golang) from 1.12.0 to 1.12.1.
- [Release notes](https://github.com/prometheus/client_golang/releases)
- [Changelog](https://github.com/prometheus/client_golang/blob/main/CHANGELOG.md)
- [Commits](https://github.com/prometheus/client_golang/compare/v1.12.0...v1.12.1)

---
updated-dependencies:
- dependency-name: github.com/prometheus/client_golang
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 14:27:38 -07:00
dependabot[bot]
5f38134032 Bump cloud.google.com/go/storage from 1.18.2 to 1.19.0 
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.18.2 to 1.19.0.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/storage/v1.18.2...spanner/v1.19.0)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:55:52 -07:00
dependabot[bot]
d5c15593bb Bump google.golang.org/api from 0.65.0 to 0.66.0 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.65.0 to 0.66.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.65.0...v0.66.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:50:18 -07:00
dependabot[bot]
fb3a3d904f Bump github.com/aws/aws-sdk-go from 1.42.40 to 1.42.44 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.40 to 1.42.44.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.40...v1.42.44)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-31 13:15:13 -07:00
Ben Johnson
ee77592d7e Skip dependabot CI using branches 2022-01-31 13:01:24 -07:00
Ben Johnson
a2cf2e260b Skip some CI jobs for dependabot 2022-01-31 12:46:06 -07:00
Ben Johnson
5d811f2e39 Fix golangci-lint issues 2022-01-31 09:21:20 -07:00
Ben Johnson
e84994ad95 Add golangci-lint to CI 2022-01-31 09:21:20 -07:00
Ben Johnson
f6c859061b Fix CodeQL warnings 2022-01-31 08:53:21 -07:00
Ben Johnson
0dfa5f98d1 Re-enable SFTP integration tests 2022-01-30 09:15:31 -07:00
Ben Johnson
906ed9b3ca Revert "Add test runner request action" 
This reverts commit 26f219da1d.
 2022-01-30 08:57:46 -07:00
Ben Johnson
26f219da1d Add test runner request action 2022-01-30 08:51:55 -07:00
Ben Johnson
f8382cfa15 Dispatch test runner in CI 2022-01-28 15:59:02 -07:00
Ben Johnson
dbdde21341 Use sqlite3_file_control(SQLITE_FCNTL_PERSIST_WAL) to persist WAL 
Previously, Litestream would avoid closing the SQLite3 connection
in order to ensure that the WAL file was not cleaned up by the
database if it was the last connection. This commit changes the
behavior by introducing a file control call to perform the same
action. This allows us to close the database file normally in all
cases.
 2022-01-28 15:12:43 -07:00
Ben Johnson
1741c82839 Produce build for every pull request 2022-01-26 16:10:39 -07:00
Ben Johnson
ffaba87b40 Separate out GitHub Actions 2022-01-25 16:20:38 -07:00
dependabot[bot]
8d759bb0b8 Bump github.com/aws/aws-sdk-go from 1.42.39 to 1.42.40 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.42.39 to 1.42.40.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.42.39...v1.42.40)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-24 13:10:29 -07:00
Ben Johnson
8950de8f7e Update dependabot.yml 2022-01-24 13:09:32 -07:00
Ben Johnson
39114502f3 Create codeql-analysis.yml 2022-01-23 09:30:17 -07:00
Ben Johnson
5d24f91ea7 Upgrade github.com/prometheus/client_golang@v1.12.0 2022-01-23 09:30:03 -07:00
Ben Johnson
55c475e3fe Upgrade github.com/pkg/sftp@v1.13.4 2022-01-23 09:24:32 -07:00
Ben Johnson
500cfd8bf4 Upgrade shellwords, golang.org/x 2022-01-23 09:19:10 -07:00
Ben Johnson
90715ef8f3 Upgrade azure-storage-blob-go to v0.14.0 2022-01-23 09:08:21 -07:00
Matt Joiner
79b50c6944 Update sqlite 3.36 2022-01-23 08:39:22 -07:00
dependabot[bot]
d045b7bef0 Bump google.golang.org/api from 0.45.0 to 0.65.0 
Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.45.0 to 0.65.0.
- [Release notes](https://github.com/googleapis/google-api-go-client/releases)
- [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-api-go-client/compare/v0.45.0...v0.65.0)

---
updated-dependencies:
- dependency-name: google.golang.org/api
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-22 11:34:08 -07:00
dependabot[bot]
6c5fb2c446 Bump cloud.google.com/go/storage from 1.15.0 to 1.18.2 
Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.15.0 to 1.18.2.
- [Release notes](https://github.com/googleapis/google-cloud-go/releases)
- [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md)
- [Commits](https://github.com/googleapis/google-cloud-go/compare/pubsub/v1.15.0...storage/v1.18.2)

---
updated-dependencies:
- dependency-name: cloud.google.com/go/storage
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-22 11:28:33 -07:00
dependabot[bot]
0b533e5d7b Bump github.com/aws/aws-sdk-go from 1.27.0 to 1.42.39 
Bumps [github.com/aws/aws-sdk-go](https://github.com/aws/aws-sdk-go) from 1.27.0 to 1.42.39.
- [Release notes](https://github.com/aws/aws-sdk-go/releases)
- [Changelog](https://github.com/aws/aws-sdk-go/blob/main/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-go/compare/v1.27.0...v1.42.39)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-22 11:24:24 -07:00
dependabot[bot]
17831c7025 Bump github.com/pierrec/lz4/v4 from 4.1.3 to 4.1.12 
Bumps [github.com/pierrec/lz4/v4](https://github.com/pierrec/lz4) from 4.1.3 to 4.1.12.
- [Release notes](https://github.com/pierrec/lz4/releases)
- [Commits](https://github.com/pierrec/lz4/compare/v4.1.3...v4.1.12)

---
updated-dependencies:
- dependency-name: github.com/pierrec/lz4/v4
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
 2022-01-22 11:19:48 -07:00
Ben Johnson
b8d04957a2 Update CI 2022-01-22 11:14:59 -07:00
Ben Johnson
0a6474fb28 Restrict CI jobs 2022-01-22 11:00:38 -07:00
Ben Johnson
c7aa3635fd Parallelize GitHub Actions 2022-01-22 10:28:57 -07:00
Ben Johnson
b8536fa4f3 dependabot.yml 2022-01-22 10:02:32 -07:00
Ben Johnson
84d08f547a Add end-to-end replication/restore testing 2022-01-15 09:05:46 -07:00
Ben Johnson
f308e0b154 CLI test coverage 2022-01-11 13:29:20 -07:00
Ben Johnson
3f0ec9fa9f Refactor Restore() 
This commit refactors out the complexity of downloading ordered WAL
files in parallel to a type called `WALDownloader`. This makes it
easier to test the restore separately from the download.
 2022-01-04 15:03:59 -07:00
Ben Johnson
531e19ed6f Refactor checksum calculation; improve test coverage 2021-12-12 10:25:20 -07:00
Michael Lynch
ba6e13b5d0 Sort output of snapshots in descending timestamp order 
By default, the snapshots command seems to output in alphabetical order of hash, which isn't meaningful, as far as I can tell.

This change modifies the order of the command output so that ./litestream snapshots returns snapshots from newest to oldest.
 2021-12-08 18:52:49 -07:00
Ben Johnson
d09f4ef618 Fix FindMinSnapshotByGeneration() loop ref bug 
This commit fixes an issue where the reference is taken
on the loop variable rather than the slice element when
computing the minimum snapshot within a generation so
it can cause the wrong snapshot to be chosen.
 2021-12-08 18:51:10 -07:00
Ben Johnson
61c80cbfc2 README 2021-12-05 08:44:19 -07:00
Ben Johnson
755f54f4d9 Update CONTRIBUTING & remove pull request template 2021-10-10 08:35:29 -06:00