v0.5.17-pre.11

Cargo.lock

@@ -2260,7 +2260,7 @@ dependencies = [
 
 [[package]]
 name = "pict-rs"
-version = "0.5.17-pre.10"
+version = "0.5.17-pre.11"
 dependencies = [
  "actix-form-data",
  "actix-web",

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "pict-rs"
 description = "A simple image hosting service"
-version = "0.5.17-pre.10"
+version = "0.5.17-pre.11"
 authors = ["asonix <asonix@asonix.dog>"]
 license = "AGPL-3.0"
 readme = "README.md"

pict-rs.nix

@@ -11,7 +11,7 @@
 
 rustPlatform.buildRustPackage {
   pname = "pict-rs";
-  version = "0.5.17-pre.10";
+  version = "0.5.17-pre.11";
   src = ./.;
 
   cargoLock = {

releases/0.5.17-pre.11.md

@@ -0,0 +1,94 @@
+# pict-rs 0.5.17-pre.11
+
+pict-rs is a simple image hosting microservice, designed to handle storing and retrieving images,
+animations, and videos, as well as providing basic image processing functionality.
+
+## Overview
+
+pict-rs 0.5.17-pre.11 includes a major bugfix for repo migrations. If you are planning to migrate
+from sled to postgres please update to this release before doing so.
+
+
+### Fixes
+
+- [UTF-8 Errors in Repo Migrations](#utf-8-errors-in-repo-migrations)
+
+
+### Changes
+
+- [Add Concurrency to Repo Migration](#add-concurrency-to-repo-migration)
+- [Include Service Version in Tracing Context](#include-service-version-in-tracing-context)
+
+
+## Upgrade Notes
+
+There are no functional changes from 0.5.17-pre.10. Upgrading should be as simple as pulling a new
+version of pict-rs.
+
+For postgres users, a script will run on first launch to attempt to fix the
+[utf-8 errors in repo migrations](#utf-8-errors-in-repo-migrations).
+
+## Details
+
+### UTF-8 Errors in Repo Migrations
+
+As it turns out, sometimes UUIDs are made entirely of bytes that are
+valid utf-8. This causes problems if a repo migration takes place.
+
+Here is an example of the problem
+
+1. A UUID is generated which is valid utf8 for an Alias and stored in
+   sled.
+2. A migration is triggered, and this Alias is read out of sled as a
+   Name, rather than a UUID.
+3. This alias is inserted into postgres with a representation that isn't
+   formatted like a UUID.
+4. That alias is requested by a user, and doesn't match because the
+   representation is different.
+
+Another example of the problem is the following
+
+1. A Delete Token is generated which is valid utf8 and contains a null
+   byte. Rust says this is fine. This delete token is stored in sled.
+2. A migration is triggered, and this Delete Token is read out of sled
+   as a Name, rather than a UUID.
+3. This Delete Token is not inserted into postgres, because postgres
+   does not permit null bytes in utf8 strings.
+
+Of these scenarios, only the first can be fixed retroactively. This
+commit includes a script which runs on pict-rs' first launch to find and
+fix any aliases or tokens that have improper representations in the
+postgres database.
+
+Any aliases lost to null bytes in past migrations cannot be recovered
+without running the migration a second time, but it's likely that a
+given pict-rs install has diverged enough from it's previous database
+that a new migration would cause more problems than it would solve.
+
+In order to fix this going forward, pict-rs always represents aliases
+and delete tokens that are 16 bytes long as UUIDs in memory, which
+matches the expected format in the repos. This means any future
+migrations will not experience this problem.
+
+
+### Add Concurrency to Repo Migrations
+
+pict-rs can now migrate multiple hashes between repos concurrently. This doesn't use more threads,
+but it does allow pict-rs to get more done on the migration thread. Anecdotally it decreased a
+simple 3000 file migration from not-too-fast to pretty-fast.
+
+The default migration concurrency is `32` and that makes a noticable difference. You can change this
+with the `-c` flag on the commandline.
+
+```bash
+pict-rs \
+    migrate-repo -c 128 \
+    sled -p data/sled-repo-local \
+    postgres -u postgres://pictrs:pictrs@localhost:5432/pictrs
+```
+
+
+### Include Service Version in Tracing Context
+
+pict-rs now reports its version to opentelemetry collectors if those are enabled. This can help keep
+track of how timings and spans change after upgrades.