-
pict-rs 0.5.10 Stable
released this
2024-03-24 00:29:39 +00:00 | 6 commits to main since this releasepict-rs 0.5.10
Overview
pict-rs 0.5.10 is a small release with changes to how pict-rs handles temporary files.
Changes
Upgrade Notes
There are no significant changes from 0.5.9. Upgrading should be as simple as pulling the new
version.Descriptions
Temporary File Cleanup
pict-rs now nests its temporary files inside a
pict-rs
toplevel temporary folder. This is useful
because pict-rs 0.5.10 introduces a new behavior: it will completely delete that folder and its
contents on launch. If you are running multiple copies of pict-rs on the same host and they share
your temporary folder, this might cause problems. In that scenario, this behavior can be disabled by
settingPICTRS__SERVER__CLEANUP_TEMPORARY_DIRECTORY=false
or passing
--no-cleanup-temporary-directory
on the commandline.This new behavior has been introduced in order to better clean up after crashes. If pict-rs is
killed while processing media, maybe due to an OOM, it will leave files behind in the temporary
directory. This can cause the temporary directory to grow, leading to memory or disk problems.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
32 MiB
-
pict-rs-linux-arm32v7
27 MiB
-
pict-rs-linux-arm64v8
28 MiB
-
pict-rs 0.5.9 Stable
released this
2024-03-11 18:51:49 +00:00 | 11 commits to main since this releasepict-rs 0.5.9
Overview
pict-rs 0.5.9 is a bugfix release for 0.5.8. All deployments on 0.5.8 should upgrade to 0.5.9
Fixes
Upgrade Notes
There are no significant changes from 0.5.8. Upgrading should be as simple as pulling the new
version.Descriptions
Fix Postgres Pooling
When pict-rs 0.5.8 was built without the
tokio_unstable
flag, it would use tokio'sspwan_local
utility from outside a LocalSet, leading to panics and timeouts. This release replaces the use of
spawn_local
withspawn
in that scenario.Fix io-uring Feature
As mentioned in the 0.5.8 release notes, io-uring is not considered to be a stable feature. However,
0.5.9 should make it usable again. Instead of manually launching a tokio_uring runtime, pict-rs once
again relies on actix-rt to configure a System for use with io-uring.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
32 MiB
-
pict-rs-linux-arm32v7
27 MiB
-
pict-rs-linux-arm64v8
28 MiB
-
pict-rs 0.5.8 Stable
released this
2024-03-11 04:10:06 +00:00 | 16 commits to main since this releasepict-rs 0.5.8
Overview
pict-rs 0.5.8 improves reliability of deletions by allowing background tasks to be retried.
Otherwise changes are fairly minor.Changes
Upgrade Notes
There is a small repo format migration between 0.5.7 and 0.5.8. For sled it's simply opening a new
tree, for postgre it involves adding a new column to the job_queue table. These changes will
automatically apply when launching pict-rs 0.5.8. Upgrading should be as simple as pulling a new
version of pict-rs.Configuration Notes
Check your configurations to make sure you haven't enabled the tokio-console integration unless
you're using it. In my local testing, I've found the console subscriber to use a significant amount
of CPU. While it is very useful for debugging, it shouldn't be used generally in production.The relevant configuration values are
PICTRS__TRACING__CONSOLE__ADDRESS
with environment variables
or[tracing.console] address = ""
in the toml.Packaging Notes
While I have never recommended packaging pict-rs with non-default crate features enabled, and the
binaries and containers I provide enable only the default features, there are two new crate features
in this release that I would advise against enabling in downstream packaging environments.The new features are
poll-timer-warnings
andrandom-errors
. These are each described below if
you want to learn about them, but as a general recommendation, do not enable non-default features
when packaging pict-rs (yes, i'm talking to yougrawlinson
from the AUR).The other optional feature,
io-uring
, is considered less stable. It's possible that folks will
find it works alright, and maybe Arch can enable it since they can assume recent kernels, but I
don't personally test much withio-uring
. It exists mostly as a historical curiosity. Please
consider carefully before enabling io-uring for pict-rs.Descriptions
Improved Task Reliability
pict-rs 0.5.8 adds the ability for tasks to be retried. pict-rs generally spawns background tasks to
handle things like Image deletion or other cleanup operations. Until now, if a background task
failed, the only indication would be a warning that appeared in the logs. These warnings are
generally descriptive and help track the error source, but end users aren't notified, and the repo
or store state can become inconsistant.With the newly added ability to retry tasks, operations should be completed more reliably. By
default, a failed task will be retried after a 2 minute wait, and if it continues to fail, it will
be retried up to five times. If a task fails after 5 retries, an additional warning will be output
to the log.In order to test this, I've added a new optional crate feature called
random-errors
, which will
inject errors into various pict-rs operations randomly. This feature should never be enabled in
production scenarios, and two warnings will be printed when launching pict-rs if it was compiled
with this feature enabled.Improved Latency
pict-rs 0.5.8 implements a couple new techniques to improve system latency.
- The postgres connection pooling library has been swapped from deadpool to bb8. Not only does this
(slightly) improve connection pool access times, but it also means pict-rs is no longer pinned
to an outdated version of deadpool. - Processes like ffmpeg, imagemagick, and exiftool are now spawned from background threads,
rather than from within the webserver threads. This is notable, since the act of spawning a
process ends up using a good amount of time, and prevents other requests from being handled
until the spawning has completed. - pict-rs now has the ability to monitor polling times for futures. By default, any task pict-rs
spawns itself will be monitored to report polling times, and a trait has been added to enable
easily tracking more polling times in the future. These polling times will appear in the
prometheus metrics, as well as in logs at DEBUG or TRACE visibility. There's an optional crate
feature calledpoll-timer-warnings
that will upgrade some of these logs to WARN visibility.
Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
32 MiB
-
pict-rs-linux-arm32v7
27 MiB
-
pict-rs-linux-arm64v8
28 MiB
- The postgres connection pooling library has been swapped from deadpool to bb8. Not only does this
-
pict-rs 0.5.7 Stable
released this
2024-03-03 17:35:32 +00:00 | 26 commits to main since this releasepict-rs 0.5.7
Overview
pict-rs 0.5.7 adds support for generating blurhashes from images and includes a couple unrelated
fixes and tweaks.Features
Changes
Fixes
Upgrade Notes
There is a small repo format migration between 0.5.6 and 0.5.7. For sled it's simply opening a new
tree, for postgres it involves adding a new column to the hashes table. These changes will
automatically apply when launching pict-rs 0.5.7. Upgrading should be as simple as pulling a new
version of pict-rs.Descriptions
Blurhash Endpoint
A new endpoint at
/image/blurhash
has been added for generating blurhashes from uploaded media. A
blurhash is a short string that encodes a few notable color values from an image that can be
reconstructed into a blurred approximation of the original image. Notably, blurhashes are used by
Mastodon to act as placeholders for sensitive media. For more information about blurhashes, see
blurha.sh.This endpoint is powered by my new blurhash encoding library,
blurhash-update.On success, the blurhash endpoint returns the following JSON.
{ "msg": "ok", "blurhash": "LGF5]+Yk^6#M@-5c,1J5@[or[Q6." }
pict-rs does not provide a blurhash decoding mechanism (it would defeat the purpose of blurhashes to
do so).File Path Changes
pict-rs has dropped its dependency on my
storage-path-generator
library in favor of using UUIDs to
create unique file paths for uploaded media. This means that newly uploaded media will be stored in
a different directory structure, and with different filenames. The purpose of this is to reduce
database use by removing the need to synchronize the current path state.pict-rs 0.5.7 also adds file extensions back to file paths, since they are now somewhat-publicly
visible (when using the public_endpoint configuration with object storage).Neither of these changes affect previously uploaded media.
Performance Improvements
pict-rs 0.5.7 now buffers media in memory less frequently, opting to stream bytes directly from
sources to sinks. This should improve general memory use, as well as decrease time pict-rs would
spend waiting to aggregate bytes.pict-rs also no longer requires bytes be present in contiguous buffers, avoiding large allocations
and reducing memcpys.More Consistent Errors
pict-rs 0.5 introduced machine-readable error codes that returned alongside error messages, but
there were a couple locations in pict-rs that failed to include them. pict-rs 0.5.7 resolves this,
ensuring all error paths properly return codes.APNG Detection
pict-rs 0.5.7 fixes the imagemagick delegate policy for ffmpeg, which allows for properly detecting
certain media, notably APNG files. pict-rs should once again be able to properly handle uploaded
APNGs.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
31 MiB
-
pict-rs-linux-arm32v7
26 MiB
-
pict-rs-linux-arm64v8
27 MiB
-
pict-rs 0.4.8 Stable
released this
2024-02-17 18:00:52 +00:00 | 498 commits to main since this releasepict-rs 0.4.8
Overview
This is a maintenance release. There's no meaningful changes outside of dependency upgrades and use
of the new CI system (Migrated from Drone to Forgejo Actions).Upgrade Notes
There's no significant changes from 0.4.7, so upgrading should be as simple as pulling a new version
of pict-rs.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
22 MiB
-
pict-rs-linux-arm32v7
19 MiB
-
pict-rs-linux-arm64v8
20 MiB
-
pict-rs 0.5.6 Stable
released this
2024-02-05 20:38:39 +00:00 | 156 commits to main since this releasepict-rs 0.5.6
Overview
pict-rs 0.5.6 adds more configuration for imagemagick security policies and updates the default
value formax_area
Features
Changes
Upgrade Notes
There's no significant changes from 0.5.5, so upgrading should be as simple as pulling a new version
of pict-rs.Descriptions
More Imagemagick Configuration
Three new configuration values have been added to the imagemagick security configuration for
pict-rs:memory
,map
, anddisk
. These options describe sizes for three tiers of storage that
imagemagick is allowed to use when processing media. The first ismemory
, this is a simple value
that represents how much RAM imagemagick is allowed to use to store image pixels. If this size is
exceeded, it will start using the next tier of storage for image pixels, which ismap
.map
represents space on disk that's mapped into RAM for quicker access. Since it's disk-backed, it can
be larger thanmemory
. Finally, ifmap
is exceeded, imagemagick will start using thedisk
for
storing pixels without mapping into memory. If thedisk
size is exceeded, media processing is
aborted.The configuration for these values can be set via the pict-rs.toml file, via environment variables,
or via the commandline.# pict-rs.toml # values are in MiB [media.magick] memory = 256 map = 512 disk = 1024
# environment variables # values are in MiB PICTRS__MEDIA__MAGICK__MEMORY=256 PICTRS__MEDIA__MAGICK__MAP=512 PICTRS__MEDIA__MAGICK__DISK=1024
# commandline # values are in MiB pict-rs run \ --media-magick-memory 256 \ --media-magick-map 512 \ --media-magick-disk 1024
Metrics Descriptions
pict-rs now has descriptions for each of the metrics it emits. They're all pretty short for now, but
can be expanded in the future if there's confusion. This is only useful when pict-rs is configured
to host a prometheus scrape endpoint.Imagemagick Area Defaults
The default value for
max_area
has been decreased from 40 million to 20 thousand. The reason for
this is it doesn't impose a hard limit on the area of uploaded images, it instead imposes a limit on
how much of an image can be held in memory at a time, with the rest of the image residing on disk.Imagemagick Frame Configuration
Imagemagick now inherits pict-rs' animation
max_frame_count
value to set it's maximum
list-length
, which should allow longer animations to be configured.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
32 MiB
-
pict-rs-linux-arm32v7
27 MiB
-
pict-rs-linux-arm64v8
27 MiB
-
pict-rs 0.5.5 Stable
released this
2024-02-04 04:10:43 +00:00 | 164 commits to main since this releasepict-rs 0.5.5
Overview
pict-rs 0.5.5 adds a bugfix for uploading images with trailing bytes and few new features for
advanced deployments.Features
Bugfixes
Upgrade Notes
There's no significant changes from 0.5.4, so upgrading should be as simple as pulling a new version
of pict-rs.Descriptions
Imagemagick Security Policy Configuration
pict-rs now supports configuring the imagemagick security policy via the pict-rs.toml file,
environment variables, or via the commandline. The security policy defines the boundaries that
imagemagick will operate with, and will allow it to abort processing media that would exceed those
boundaries.Currently, there are only a few items that can be configured.
# pict-rs.toml [media.magick] max_width = 10000 max_hight = 10000 max_area = 40000000
# environment variables PICTRS__MEDIA__MAGICK__MAX_WIDTH=10000 PICTRS__MEDIA__MAGICK__MAX_HEIGHT=10000 PICTRS__MEDIA__MAGICK__MAX_AREA=40000000
# commandline pict-rs run \ --media-magick-max-width 10000 \ --media-magick-max-height 10000 \ --media-magick-max-aread 40000000
It will also apply the configured
process_timeout
to the security policy.Serving with TLS
pict-rs can now be configured to serve itself over TLS if provided with a server key and a server
certificate. This is for more advanced deployments that have Certificate Authority infrastructure in
place. When serving over TLS, downstream services need to be configured to access pict-rs over TLS.# pict-rs.toml [server] certificate = "/path/to/server.crt" private_key = "/path/to/server.key"
# environment variables PICTRS__SERVER__CERTIFICATE=/path/to/server.crt PICTRS__SERVER__PRIVATE_KEY=/path/to/server.key
# commandline pict-rs run \ --certificate /path/to/server.crt \ --private-key /path/to/server.key
Broken Pipe Error
In previous 0.5 releases with the default configurations, it was possible for valid images to fail
to upload if they contained excess trailing bytes. This was caused by exiftool completing metadata
processing on the image bytes before pict-rs had written the entire buffer to exiftool's stdin. The
fix was to simply treat the case of stdin closing early as a success, rather than a failure. In the
event there was actually an error in exiftool, the command will fail and pict-rs will return a
proper status error instead.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
32 MiB
-
pict-rs-linux-arm32v7
27 MiB
-
pict-rs-linux-arm64v8
27 MiB
-
pict-rs 0.5.4 Stable
released this
2024-01-25 23:11:40 +00:00 | 190 commits to main since this releasepict-rs 0.5.4
Overview
Another quick release to improve postgres performance, and reduce growth of the job_queue during
cleanup spikes.Changes
Descriptions
Limit LISTEN Connections
When postgres is configured, pict-rs sometimes issues LISTEN queries in order to be notified of new
jobs in the job queue, due to connection pooling, it would be possible for every connection pict-rs
makes to postgres to be listening for notifications. This is needless and wastes resources. 0.5.4
now limits the number of connections issuing LISTEN queries to at most 4, or as few as the number of
cores in your machine, if less than 4.Limit Spawned Jobs
As mentioned in the 0.5.2 release document, pict-rs runs cleanup jobs that can spawn a lot of other
jobs. In order to keep the queue size from growing too large for larger deployments, pict-rs now
limits each general cleanup job to 2048 iterations, meaning it will complete after spawning just
2048 jobs.Since pict-rs processes jobs in-order, this means that the majority of these jobs will be completed
before the next general cleanup job runs, keeping the maximum number of queued jobs around 2048.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
31 MiB
-
pict-rs-linux-arm32v7
26 MiB
-
pict-rs-linux-arm64v8
26 MiB
-
pict-rs 0.5.3 Stable
released this
2024-01-25 14:57:40 +00:00 | 196 commits to main since this releasepict-rs 0.5.3
Overview
A quick release after 0.5.2 to add a default value for
use_tls
, and to fix launching pict-rs with
postgres using commandline flags.Bugfixes
Upgrade Notes
There's no significant changes from 0.5.2, so upgrading should be as simple as pulling a new version
of pict-rs.Descriptions
Semver Break in 0.5.2
When adding TLS support to postgres in 0.5.2, I had forgotten to set a default value for the new
use_tls
configuration option, leading to postgres deployments no longer launching after upgrade.
This has been fixed by setting it tofalse
by default, which aligns with the previous behavior of
not using TLS.Postgres Commandline Configuration Panic
When adding TLS support to postgres in 0.5.2, I had accidentally added two conflicting commandline
arguments that expected to use-u
. The first is for the postgres URL, and the second is for the
use_tls
setting. This caused panics when attempting to configure postgres with commandline flags.
0.5.3 fixes this by changinguse_tls
to use the-t
short flag, rather than-u
.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
31 MiB
-
pict-rs-linux-arm32v7
26 MiB
-
pict-rs-linux-arm64v8
26 MiB
-
pict-rs 0.5.2 Stable
released this
2024-01-24 23:41:42 +00:00 | 199 commits to main since this releasepict-rs 0.5.2
Overview
A quick release to avoid a bug leading to incredibly large cleanup queue sizes and resulting in high
CPU use. Upgrading to 0.5.2 is recommended for any deployment that recently upgraded to 0.5.0 or
0.5.1.Bugfixes
Changes
Upgrade Notes
There's no significant changes from 0.5.1, so upgrading should be as simple as pulling a new version
of pict-rs.Descriptions
At-Most Once Cleanup Jobs
pict-rs 0.5.2 now is smarter about queueing certain cleanup jobs. Previously, it might queue an
arbitrary number of the same cleanup job while under significant load, resulting in many redundant
jobs consuming CPU. The reason for this is pict-rs' strategy for handling variants and proxied media
that hadn't been accessed within the configured timeframe was to queue a job to check every 30
seconds. This is fine in normal conditions, but can be a problem after upgrading to 0.5 from 0.4.During the upgrade, pict-rs marks all variants as having been accessed at the time of the upgrade,
since this information was not tracked in 0.4. This results in many variants being queued for
cleanup exactly 1 week (or otherwise configured cache duration) after upgrading. When the cleanup
queue fills with these cleanup jobs, it takes longer to process them all than the 30 second window
between queues of the general cleanup jobs, resulting in redundant cleanup jobs that spawn the same
redundant removal jobs, increasing the queue size by the number of outdated variants each time.By adding an option to the queueing system to mark certain jobs as unique, pict-rs can now ensure
that only one copy of these general cleanup jobs can be queued or running at any given time,
ensuring that all individual variant cleanup jobs will be queued after a general cleanup job runs
before the next general cleanup job is queued.Support TLS in Postgres Connections
The postgres repo now supports some new options to enable connecting via TLS. If the postgres
server's certificate is not generally trusted, a CA certificate can be provided to pict-rs to allow
it to trust the postgres connection.If the postgres endpoint uses TLS and has a publicly trusted certificate, the configuration can be
updated to simply adduse_tls = true
[repo] type = 'postgres' url = 'postgres://pictrs:1234@localhost:5432/pictrs' use_tls = true
or with environment variables
PICTRS__REPO__TYPE=postgres PICTRS__REPO__URL=postgres://pictrs:1234@localhost:5432/pictrs PICTRS__REPO__USE_TLS=true
If the server uses a self-signed certificate, the CA's certificate can be added to pict-rs' trusted
CAs.[repo] type = 'postgres' url = 'postgres://pictrs:1234@localhost:5432/pictrs' use_tls = true certificate_file = '/path/to/ca/certificate.crt'
or with environment variables
PICTRS__REPO__TYPE=postgres PICTRS__REPO__URL=postgres://pictrs:1234@localhost:5432/pictrs PICTRS__REPO__USE_TLS=true PICTRS__REPO__CERTIFICATE_FILE=/path/to/ca/certificate.crt
Compiler Version Requirements
pict-rs 0.5.2 now takes advantage of rust's native support for AFIT (Async Fn In Trait) for the
store trait. This trait already was not object-safe, so using AFIT does not come with any downsides.
This does mean that the minimum supported rust version for building pict-rs is now 1.75, which
released in December 2023.Downloads
- Source Code (ZIP)
- Source Code (TAR.GZ)
-
pict-rs-linux-amd64
31 MiB
-
pict-rs-linux-arm32v7
26 MiB
-
pict-rs-linux-arm64v8
26 MiB