Commit Graph

11 Commits

Author SHA1 Message Date
Toni Spets
e4254bbf69 Never directly dereference AWS SDK returned pointers (#557) 2024-01-07 06:18:32 +02:00
guangwu
c1ae968188 fix: typo (#509) 2023-10-19 18:29:04 -06:00
Markus Schanz
1bfcaa4a17 Recognize Scaleway S3 replica URLs (#499) 2023-08-13 18:07:19 -06:00
Erik Kristensen
3368b7cf44 fix: remove debug code 2023-05-02 16:03:31 -06:00
Erik Kristensen
ae670b0d27 fix: aws credential chain by using aws.Config 2023-05-02 16:03:31 -06:00
Lincoln Stoll
5afd0bf161 Handle errors when deleting objects from S3
I recently noticed that the cost for ListBucket calls was increasing for an
application that was using Litestream. After investigating it seemed that the
bucket had retained the entire history of data, while Litestream was
continually logging that it was deleting the same data:

```
2022-10-30T12:00:27Z (s3): wal segmented deleted before 0792d3393bf79ced/00000233: n=1428
<snip>
2022-10-30T13:00:24Z (s3): wal segmented deleted before 0792d3393bf79ced/00000233: n=1428
```

This is occuring because the DeleteObjects call is a batch item, that returns
the individual object deletion errors in the response[1]. The S3 replica client
discards the response, and only handles errors in the original API call. I had
a misconfigured IAM policy that meant all deletes were failing, but this never
actually bubbled up as a real error.

To fix this, I added a check for the response body to handle any errors the
operation might have encountered. Because this may include a large number of
errors (in this case 1428 of them), the output is summarized to avoid an overly
large error message. When items are not found, they will not return an error[2]
- they will still be marked as deleted, so this change should be in-line with
the original intentions of this code.

1: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html#API_DeleteObjects_Example_2
2: https://docs.aws.amazon.com/AmazonS3/latest/API/API_DeleteObjects.html
2023-05-02 16:03:31 -06:00
Ben Johnson
c06997789b Add support for filebase replica URL 2021-06-02 15:09:08 -06:00
Ben Johnson
1c0c69a5ab Unify replica client metrics 2021-06-01 18:19:56 -06:00
Ben Johnson
ac32e8e089 Add Google Cloud Storage replica 2021-05-22 07:16:14 -06:00
Ben Johnson
6c865e37f1 Move path functions to litestream package 2021-05-21 10:39:22 -06:00
Ben Johnson
fb80bc10ae Refactor replica system 2021-05-21 07:44:36 -06:00