asonix/pict-rs
1
1
Fork 1
Code Issues 2 Pull requests Actions Projects Releases 134 Wiki Activity

Add initial migration information for 0.5, postgres
Some checks are pending
continuous-integration/drone/push Build is running
Details

Browse source
This commit is contained in:
asonix 2023-09-30 22:45:59 -05:00
parent 4c7067d4ca
commit d2745c4cd5
1 changed files with 159 additions and 43 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

202
README.md
View file

@ -15,9 +15,16 @@ _a simple image hosting service_
2. [Api](#api)
3. [Administration](#administration)
1. [Backups](#backups)
2. [0.3 to 0.4 Migration Guide](#0-3-to-0-4-migration-guide)
2. [0.4 to 0.5 Migration Guide](#0-4-to-0-5-migration-guide)
1. [Overview](#overview)
2. [Configuration Updates](#configuration-updates)
1. [Image Changes](#image-changes)
2. [Animation Changes](#animation-changes)
3. [Video Changes](#video-changes)
3. [Upgrading Directly to Postgres](#upgrading-directly-to-postgres)
3. [Filesystem to Object Storage Migration](#filesystem-to-object-storage-migration)
1. [Troubleshooting](#migration-troubleshooting)
4. [Sled to Postgres Migration](#sled-to-postgres-migration)
4. [Development](#development)
1. [Nix Development](#nix-development)
1. [With direnv and nix-direnv](#with-direnv-and-nix-direnv)
@ -254,7 +261,7 @@ More information is available in the [Ubuntu and Debian docs](./docs/ubuntu-and-
##### Compile from Source
pict-rs can be compiled from source using a recent version of the rust compiler. I do development
and produce releases on 1.70. pict-rs also requires the `protoc` protobuf compiler to be present at
and produce releases on 1.72. pict-rs also requires the `protoc` protobuf compiler to be present at
build-time in order to enable use of [`tokio-console`](https://github.com/tokio-rs/console).
Like the Binary Download option, `imagemagick`, `ffmpeg`, and `exiftool` must be installed for
@ -669,62 +676,122 @@ If you can't stop pict-rs, but would like to back up the database, there is an i
`/internal/export` documented in [Api](#api) that can be used to produce a copy of the current
database for easy backups.
### 0.3 to 0.4 Migration Guide
pict-rs will automatically migrate from the 0.3 db format to the 0.4 db format on the first launch
of 0.4. If you are running the provided docker container without any custom configuration, there are
no additional steps.
### 0.4 to 0.5 Migration Guide
#### Overview
pict-rs will automatically migrate from the 0.4 db format to the 0.5 db format on the first launch
of 0.5. This process might take a while, especially if you've been running pict-rs since before 0.3.
The reason for this is pict-rs now requires original files to have associated details records stored
in the database, and while generating these records happened by default for 0.3 and 0.4, images
uploaded before this was standard may not have ever had their details records generated.
If you have any custom configuration for file paths, or you are running outside of docker, then
there is some extra configuration that needs to be done.
_This upgrade must be performed while pict-rs is offline._
If your previous `PICTRS__PATH` variable or `path` config was set, it needs to be translated to the
new configuration format.
#### Configuration Updates
Previously, pict-rs only had two categories for files: images and videos. pict-rs 0.5 adds a third
category: animation. With the new explicit support for animated file types, some configuration
options have moved.
`PICTRS_PATH` has split into three separate config options:
- `PICTRS__OLD_DB__PATH`: This should be set to the same value that `PICTRS__PATH` was. It is used
during the migration from 0.3 to 0.4
- `PICTRS__REPO__PATH`: This is the location of the 0.4 database. It should be set to a subdirectory
of the previous `PICTRS__PATH` directory. I would recommend `/previous/path/sled-repo`
- `PICTRS__STORE__PATH`: This is the location of the files. It should be the `files` subdirectory of
the previous PICTRS__PATH directory.
##### Image Changes
| Old Environment Variable | New Environment Variable |
| ------------------------------ | ------------------------------------- |
| `PICTRS__MEDIA__FORMAT` | `PICTRS__MEDIA__IMAGE__FORMAT` |
| `PICTRS__MEDIA__MAX_WIDTH` | `PICTRS__MEDIA__IMAGE__MAX_WIDTH` |
| `PICTRS__MEDIA__MAX_HEIGHT` | `PICTRS__MEDIA__IMAGE__MAX_HEIGHT` |
| `PICTRS__MEDIA__MAX_AREA` | `PICTRS__MEDIA__IMAGE__MAX_AREA` |
| | `PICTRS__MEDIA__IMAGE__MAX_FILE_SIZE` |
if you configured via the configuration file, these would be
| Old TOML Value | New TOML Value |
| ----------------------- | ----------------------------- |
| `[media] format` | `[media.image] format` |
| `[media] max_width` | `[media.image] max_width` |
| `[media] max_height` | `[media.image] max_height` |
| `[media] max_area` | `[media.image] max_area` |
| | `[media.image] max_file_size` |
##### Animation Changes
| Old Environment Variable | New Environment Variable |
| ------------------------------------- | ------------------------------------------- |
| `PICTRS__MEDIA__GIF__MAX_WIDTH` | `PICTRS__MEDIA__ANIMATION__MAX_WIDTH` |
| `PICTRS__MEDIA__GIF__MAX_HEIGHT` | `PICTRS__MEDIA__ANIMATION__MAX_HEIGHT` |
| `PICTRS__MEDIA__GIF__MAX_AREA` | `PICTRS__MEDIA__ANIMATION__MAX_AREA` |
| `PICTRS__MEDIA__GIF__MAX_FILE_SIZE` | `PICTRS__MEDIA__ANIMATION__MAX_FILE_SIZE` |
| `PICTRS__MEDIA__GIF__MAX_FRAME_COUNT` | `PICTRS__MEDIA__ANIMATION__MAX_FRAME_COUNT` |
| | `PICTRS__MEDIA__ANIMATION__FORMAT` |
| | `PICTRS__MEDIA__ANIMATION__MAX_FILE_SIZE` |
| Old TOML Value | New TOML Value |
| ----------------------------- | ----------------------------------- |
| `[media.gif] max_width` | `[media.animation] max_width` |
| `[media.gif] max_height` | `[media.animation] max_height` |
| `[media.gif] max_area` | `[media.animation] max_area` |
| `[media.gif] max_file_size` | `[media.animation] max_file_size` |
| `[media.gif] max_frame_count` | `[media.animation] max_frame_count` |
| | `[media.animation] format` |
| | `[media.animation] max_file_size` |
##### Video Changes
| Old Environment Variable | New Environment Variable |
| ------------------------------------ | --------------------------------------- |
| `PICTRS__MEDIA__ENABLE_SILENT_VIDEO` | `PICTRS__MEDIA__VIDEO__ENABLE` |
| `PICTRS__MEDIA__ENABLE_FULL_VIDEO` | `PICTRS__MEDIA__VIDEO__ALLOW_AUDIO` |
| `PICTRS__MEDIA__VIDEO_CODEC` | `PICTRS__MEDIA__VIDEO__VIDEO_CODEC` |
| `PICTRS__MEDIA__AUDIO_CODEC` | `PICTRS__MEDIA__VIDEO__AUDIO_CODEC` |
| `PICTRS__MEDIA__MAX_FRAME_COUNT` | `PICTRS__MEDIA__VIDEO__MAX_FRAME_COUNT` |
| `PICTRS__MEDIA__ENABLE_FULL_VIDEO` | `PICTRS__MEDIA__VIDEO__ALLOW_AUDIO` |
| | `PICTRS__MEDIA__VIDEO__MAX_WIDTH` |
| | `PICTRS__MEDIA__VIDEO__MAX_HEIGHT` |
| | `PICTRS__MEDIA__VIDEO__MAX_AREA` |
| | `PICTRS__MEDIA__VIDEO__MAX_FILE_SIZE` |
| Old TOML Value | New TOML Value |
| ----------------------------- | ------------------------------- |
| `[media] enable_silent_video` | `[media.video] enable` |
| `[media] enable_full_video` | `[media.video] allow_audio` |
| `[media] video_codec` | `[media.video] video_codec` |
| `[media] audio_codec` | `[media.video] audio_codec` |
| `[media] max_frame_count` | `[media.video] max_frame_count` |
| `[media] enable_full_video` | `[media.video] allow_audio` |
| | `[media.video] max_width` |
| | `[media.video] max_height` |
| | `[media.video] max_area` |
| | `[media.video] max_file_size` |
Note that although each media type now includes its own `MAX_FILE_SIZE` configuration, the
`PICTRS__MEDIA__MAX_FILE_SIZE` value still exists as a global limit for any file type.
In addition to all the configuration options mentioned above, there are now individual quality
settings that can be configured for each image and animation type, as well as for video files.
Please see the [pict-rs.toml](./pict-rs.toml) file for more information.
#### Upgrading Directly to Postgres
pict-rs supports migrating directly to the postgres repo during the upgrade. In order to do this,
the postgres repo needs to be configured and the `old_repo` needs to be specified. The `old_repo`
section just contains the `path` of the `repo` section in your 0.4 config.
Example:
```toml
[old_db]
path = "/previous/path"
[old_repo]
path = '/mnt/sled-repo'
[repo]
path = "/previous/path/sled-repo"
[store]
path = "/previous/path/files"
type = 'postgres'
url = 'postgres://user:password@host:5432/db'
```
If your previous `RUST_LOG` variable was set, it has been split into two different configuration
options:
- `PICTRS__TRACING__LOGGING__TARGETS`: This dictates what logs should be printed in the console while
pict-rs is running.
- `PICTRS__TRACING__OPENTELEMETRY__TARGETS`: This dictates what spans and events should be exported
as opentelemetry data, if enabled.
You can also configure these options via the configuration file:
```toml
[tracing.logging]
targets = "debug"
[tracing.opentelemetry]
targets = "debug"
Or with environment varaibles:
```
PICTRS__OLD_REPO__PATH=/mnt/sled-repo
PICTRS__REPO__TYPE=postgres
PICTRS__REPO__URL=postgres://user:password@host:5432/db
```
If the migration doesn't work due to a configuration error, the new sled-repo directory can be
deleted and a new migration will be automatically triggered on the next launch.
Once these variables are set, 0.5 can be started and the migration will automatically occur.
### Filesystem to Object Storage migration
_Make sure you take a backup of the sled-repo directory before running this command!!! Migrating to
object storage updates the database and if you need to revert for any reason, you'll want a backup._
After migrating from 0.3 to 0.4, it is possible to migrate to object storage. This can be useful if
It is possible to migrate to object storage. This can be useful if
hosting in a cloud environment, since object storage is generally far cheaper than block storage.
There's a few required configuration options for object storage. I will try to explain:
@ -871,6 +938,7 @@ Error:
Cause: the region was set improperly. Additionaly a path-style endpoint was used without passing
`--use-path-style`
Error:
```
0: Error in store
@ -884,7 +952,6 @@ Error:
```
Cause: the access key was set improperly
If you have enabled object storage without first migrating your existing files to object storage,
these migrate commands may end up retrying file migrations indefinitely. In order to successfully
resolve this multi-store problem the `--skip-missing-files` flag has been added to the
@ -934,6 +1001,55 @@ Additionally, some providers might require you include the `region` in your endp
Check your object storage provider's documentation to be sure you're setting the right values.
### Sled to Postgres Migration
If you upgraded to 0.5 without migrating to postgres at the same time, you can migrate to postgres
after the fact with the built-in migration utility. Before running the migration, make sure you have
a postgres role and database ready for pict-rs. The first thing pict-rs will do upon connecting to
a new database is attempt to add the `pgcrypto` extension, so if the role you created for pict-rs
does not have that permission, it will fail.
The migration command is fairly simple. It just needs the path to your existing repo and the URL to
your new repo.
```bash
$ pict-rs \
migrate-repo \
sled -p /path/to-/sled-repo \
postgres -u postgres://user:password@host:5432/db
```
If you're running with docker-compose, this could look like the following:
```bash
$ sudo docker compose stop pictrs # stop the pict-rs container
$ sudo docker compose run pictrs sh # launch a shell in the pict-rs container
> pict-rs --version # verify pict-rs version is recent (should probably be 0.4.0 or later)
> pict-rs \
migrate-repo \
sled -p /mnt/sled-repo \
postgres -u postgres://user:password@host:5432/db
> exit
$ vi docker-compose.yml # edit the docker-compose yaml however you like to edit it, make sure all the variables described below are set
$ sudo docker compose up -d pictrs # start pict-rs again after the migration. Note that this is not 'docker compose start'. using the `up` subcommand explicitly reloads configurations
```
_This command must be run while pict-rs is offline._
This migration should be pretty quick, since there's no actual files getting moved around. After the
migration completes, make sure pict-rs is configured to use the postgres repo, then start it back
up.
Example:
```toml
[repo]
type = 'postgres'
url = 'postgres://user:password@host:5432/db'
```
or
```
PICTRS__REPO__TYPE=postgres
PICTRS__REPO__URL=postgres://user:password@host:5432/db
```
## Development
pict-rs has a few native dependencies that need to be installed in order for it to run properly.