Commit Graph

231 Commits

Author SHA1 Message Date
Ben Johnson c8b72bf16b Fix release action v0.3.0 2021-01-21 14:39:23 -07:00
Ben Johnson 9c4de6c520 Debian release action 2021-01-21 14:34:42 -07:00
Ben Johnson 94411923a7 Fix unit test 2021-01-21 13:52:35 -07:00
Ben Johnson e92db9ef4b Enforce stricter validation on restart.
Previously, the sync would validate the last page written to ensure
that replication picked up from the last position. However, a large
WAL file followed by a series of shorter checkpointed WAL files means
that the last page could be the same even if multiple checkpoints
have occurred.

To fix this, the WAL header must match the shadow WAL header when
starting litestream since there are no guarantees about checkpoints.
2021-01-21 13:44:05 -07:00
Ben Johnson 031a526b9a Only copy committed WAL pages 2021-01-21 12:44:11 -07:00
Ben Johnson 2244be885d README 2021-01-20 17:05:47 -07:00
Ben Johnson 95bcaa5927 Fix file replica compression 2021-01-19 09:25:38 -07:00
Ben Johnson 1935ebd6f0 Fix S3 GET bytes metric 2021-01-19 06:46:13 -07:00
Ben Johnson 7fb98df240 cleanup 2021-01-18 15:58:49 -07:00
Ben Johnson f31c22af62 Remove s3 bucket lookup log 2021-01-18 15:27:16 -07:00
Ben Johnson 139d836d7a Fix file/dir mode 2021-01-18 15:23:28 -07:00
Ben Johnson 14dad1fd5a Switch from gzip to lz4 2021-01-18 14:45:12 -07:00
Ben Johnson 35d755e7f2 Remove debugging code 2021-01-18 10:33:30 -07:00
Ben Johnson 358dcd4650 Copy shadow WAL immediately after init 2021-01-18 10:01:16 -07:00
Ben Johnson 2ce4052300 Remove write lock during db checksum 2021-01-18 07:05:27 -07:00
Ben Johnson 44af75fa98 Fix missing WAL reader error 2021-01-18 07:05:17 -07:00
Ben Johnson 3c4fd152c9 Add more checksum logging 2021-01-18 06:38:03 -07:00
Ben Johnson d259d9b9e3 Fix checksum logging 2021-01-17 10:19:39 -07:00
Ben Johnson 90a1d959d4 Remove size from s3 filenames 2021-01-17 10:02:06 -07:00
Ben Johnson 04d75507e3 Fix checksum hex padding 2021-01-17 09:52:09 -07:00
Ben Johnson 4b65e6a88f Log validation position 2021-01-17 07:38:13 -07:00
Ben Johnson 07a65cbac7 Fix crc64 unit test 2021-01-16 10:04:03 -07:00
Ben Johnson 6ac6a8536d Obtain write lock during validation. 2021-01-16 09:27:43 -07:00
Ben Johnson 71ab15e50a Fix S3 GET stats 2021-01-16 09:22:33 -07:00
Ben Johnson b4e5079760 Add .deb packaging 2021-01-16 09:22:02 -07:00
Ben Johnson 78563f821d Do not require databases when starting replication 2021-01-16 09:15:16 -07:00
Ben Johnson e65536f81d Stop waiting for replica if generation changes 2021-01-16 07:47:02 -07:00
Ben Johnson 25fec29e1a Clear last position on replica sync error 2021-01-16 07:45:08 -07:00
Ben Johnson cbc2dce6dc Add busy timeout 2021-01-16 07:33:32 -07:00
Ben Johnson 1b8cfc8a41 Add validation interval 2021-01-15 16:37:04 -07:00
Ben Johnson 290e06e60d Reduce s3 LIST operations 2021-01-15 13:31:04 -07:00
Ben Johnson b94ee366e5 Fix snapshot only restore 2021-01-15 13:12:15 -07:00
Ben Johnson 743aeb83e1 Revert gzip compression level, fix s3 wal upload 2021-01-15 13:04:21 -07:00
Ben Johnson a7ec05ad7a Allow global AWS settings in config. 2021-01-15 12:27:41 -07:00
Ben Johnson 28dd7b564e Lookup s3 bucket region if not specified 2021-01-15 12:18:07 -07:00
Ben Johnson 43dda4315f Allow URLs for replica config path 2021-01-15 12:04:23 -07:00
Ben Johnson 0655bf420a Fix unit tests 2021-01-14 16:13:19 -07:00
Ben Johnson 8c113cf260 Add file & s3 replica metrics 2021-01-14 16:10:02 -07:00
Ben Johnson daa74f87b4 Add file replica metrics 2021-01-14 15:47:58 -07:00
Ben Johnson e1c9e09161 Update wal segment naming 2021-01-14 15:26:29 -07:00
Ben Johnson 1e4e9633cc Add s3 sync interval 2021-01-14 15:04:26 -07:00
Ben Johnson 294846cce2 Add context to s3 2021-01-13 16:38:00 -07:00
Ben Johnson 9eb7bd41c2 S3 reader & retention enforcement 2021-01-13 16:21:42 -07:00
Ben Johnson 1ac4adb272 Basic s3 replication working 2021-01-13 14:23:41 -07:00
Ben Johnson a42f83f3cb Add LITESTREAM_CONFIG env var 2021-01-13 13:17:38 -07:00
Ben Johnson 57a02a8628 S3 replica 2021-01-13 10:14:54 -07:00
Ben Johnson faa5765745 Add retention policy, remove WAL subdir 2021-01-12 15:22:37 -07:00
Ben Johnson 1fa1313b0b Add trace logging. 2021-01-11 11:04:29 -07:00
Ben Johnson bcdb553267 Use database owner/group 2021-01-11 09:39:08 -07:00
Ben Johnson 9828b4c1dd Rename 'databases' to 'dbs' in config 2021-01-10 10:07:07 -07:00