Litestream Design
=================

Litestream provides a file system layer to intercept writes to a SQLite database
to construct a persistent write-ahead log that can be replicated.


## File Layout

### Local

```
dir/
	db                               # SQLite database
	db-wal                           # SQLite WAL
	.db-lightstream.config           # per-db configuration
	.db-lightstream/
		log                          # recent event log
		stat                         # per-db Prometheus statistics
		snapshot                     # stores snapshot number (e.g. 0000000000000001)
		wal/                         # each WAL file contains pages in flush interval
			active                   # active WAL file exists until flush; renamed
			0000000000000001.wal.gz  # flushed, compressed WAL files
			0000000000000002.wal.gz
```

### Remote (S3)

```
bkt/
	db/                                                # database path
		0000000000000001/                              # snapshot directory
			snapshot                                   # full db snapshot
			0000000000000001.wal.gz                    # compressed WAL file
			0000000000000002.wal.gz
		0000000000000002/
			snapshot
			0000000000000001-0000000000000003.tar.gz
```


## Process

### File System Startup

File system startup:

1. Load litestream.config file.
2. Load all per-db "-lightstream.config" files.


### DB startup:

```
IF "db" NOT EXISTS {
	ensureWALRemovedIfDBNotExist()
	restore()
	setDBStatus("ok")
	return
} 

IF "-wal" EXISTS {
	syncToShadowWAL()
	IF err {
		setDBStatus("error")
	} ELSE {
		setDBStatus("ok")
	}
} ELSE {
	ensureShadowWALMatchesDB() // check last page written to DB
	IF err {
		setDBStatus("error")
	} ELSE {
		setDBStatus("ok")
	}
}
```


### DB Recovery

TODO


### WAL Write

1. Write to regular WAL
2. On fsync to regular WAL, copy WAL to shadow WAL.
2a. On copy error, mark errored & begin recovery