• v0.5.10 5f850f8c86

    asonix released this 2024-03-24 00:29:39 +00:00 | 6 commits to main since this release

    pict-rs 0.5.10

    Overview

    pict-rs 0.5.10 is a small release with changes to how pict-rs handles temporary files.

    Changes

    Upgrade Notes

    There are no significant changes from 0.5.9. Upgrading should be as simple as pulling the new
    version.

    Descriptions

    Temporary File Cleanup

    pict-rs now nests its temporary files inside a pict-rs toplevel temporary folder. This is useful
    because pict-rs 0.5.10 introduces a new behavior: it will completely delete that folder and its
    contents on launch. If you are running multiple copies of pict-rs on the same host and they share
    your temporary folder, this might cause problems. In that scenario, this behavior can be disabled by
    setting PICTRS__SERVER__CLEANUP_TEMPORARY_DIRECTORY=false or passing
    --no-cleanup-temporary-directory on the commandline.

    This new behavior has been introduced in order to better clean up after crashes. If pict-rs is
    killed while processing media, maybe due to an OOM, it will leave files behind in the temporary
    directory. This can cause the temporary directory to grow, leading to memory or disk problems.

    Downloads
  • v0.5.9 2ead3e00e2

    pict-rs 0.5.9 Stable

    asonix released this 2024-03-11 18:51:49 +00:00 | 11 commits to main since this release

    pict-rs 0.5.9

    Overview

    pict-rs 0.5.9 is a bugfix release for 0.5.8. All deployments on 0.5.8 should upgrade to 0.5.9

    Fixes

    Upgrade Notes

    There are no significant changes from 0.5.8. Upgrading should be as simple as pulling the new
    version.

    Descriptions

    Fix Postgres Pooling

    When pict-rs 0.5.8 was built without the tokio_unstable flag, it would use tokio's spwan_local
    utility from outside a LocalSet, leading to panics and timeouts. This release replaces the use of
    spawn_local with spawn in that scenario.

    Fix io-uring Feature

    As mentioned in the 0.5.8 release notes, io-uring is not considered to be a stable feature. However,
    0.5.9 should make it usable again. Instead of manually launching a tokio_uring runtime, pict-rs once
    again relies on actix-rt to configure a System for use with io-uring.

    Downloads
  • v0.5.8 7021c50156

    pict-rs 0.5.8 Stable

    asonix released this 2024-03-11 04:10:06 +00:00 | 16 commits to main since this release

    pict-rs 0.5.8

    Overview

    pict-rs 0.5.8 improves reliability of deletions by allowing background tasks to be retried.
    Otherwise changes are fairly minor.

    Changes

    Upgrade Notes

    There is a small repo format migration between 0.5.7 and 0.5.8. For sled it's simply opening a new
    tree, for postgre it involves adding a new column to the job_queue table. These changes will
    automatically apply when launching pict-rs 0.5.8. Upgrading should be as simple as pulling a new
    version of pict-rs.

    Configuration Notes

    Check your configurations to make sure you haven't enabled the tokio-console integration unless
    you're using it. In my local testing, I've found the console subscriber to use a significant amount
    of CPU. While it is very useful for debugging, it shouldn't be used generally in production.

    The relevant configuration values are PICTRS__TRACING__CONSOLE__ADDRESS with environment variables
    or [tracing.console] address = "" in the toml.

    Packaging Notes

    While I have never recommended packaging pict-rs with non-default crate features enabled, and the
    binaries and containers I provide enable only the default features, there are two new crate features
    in this release that I would advise against enabling in downstream packaging environments.

    The new features are poll-timer-warnings and random-errors. These are each described below if
    you want to learn about them, but as a general recommendation, do not enable non-default features
    when packaging pict-rs (yes, i'm talking to you grawlinson from the AUR).

    The other optional feature, io-uring, is considered less stable. It's possible that folks will
    find it works alright, and maybe Arch can enable it since they can assume recent kernels, but I
    don't personally test much with io-uring. It exists mostly as a historical curiosity. Please
    consider carefully before enabling io-uring for pict-rs.

    Descriptions

    Improved Task Reliability

    pict-rs 0.5.8 adds the ability for tasks to be retried. pict-rs generally spawns background tasks to
    handle things like Image deletion or other cleanup operations. Until now, if a background task
    failed, the only indication would be a warning that appeared in the logs. These warnings are
    generally descriptive and help track the error source, but end users aren't notified, and the repo
    or store state can become inconsistant.

    With the newly added ability to retry tasks, operations should be completed more reliably. By
    default, a failed task will be retried after a 2 minute wait, and if it continues to fail, it will
    be retried up to five times. If a task fails after 5 retries, an additional warning will be output
    to the log.

    In order to test this, I've added a new optional crate feature called random-errors, which will
    inject errors into various pict-rs operations randomly. This feature should never be enabled in
    production scenarios, and two warnings will be printed when launching pict-rs if it was compiled
    with this feature enabled.

    Improved Latency

    pict-rs 0.5.8 implements a couple new techniques to improve system latency.

    1. The postgres connection pooling library has been swapped from deadpool to bb8. Not only does this
      (slightly) improve connection pool access times, but it also means pict-rs is no longer pinned
      to an outdated version of deadpool.
    2. Processes like ffmpeg, imagemagick, and exiftool are now spawned from background threads,
      rather than from within the webserver threads. This is notable, since the act of spawning a
      process ends up using a good amount of time, and prevents other requests from being handled
      until the spawning has completed.
    3. pict-rs now has the ability to monitor polling times for futures. By default, any task pict-rs
      spawns itself will be monitored to report polling times, and a trait has been added to enable
      easily tracking more polling times in the future. These polling times will appear in the
      prometheus metrics, as well as in logs at DEBUG or TRACE visibility. There's an optional crate
      feature called poll-timer-warnings that will upgrade some of these logs to WARN visibility.
    Downloads
  • v0.5.7 40bb58d603

    pict-rs 0.5.7 Stable

    asonix released this 2024-03-03 17:35:32 +00:00 | 26 commits to main since this release

    pict-rs 0.5.7

    Overview

    pict-rs 0.5.7 adds support for generating blurhashes from images and includes a couple unrelated
    fixes and tweaks.

    Features

    Changes

    Fixes

    Upgrade Notes

    There is a small repo format migration between 0.5.6 and 0.5.7. For sled it's simply opening a new
    tree, for postgres it involves adding a new column to the hashes table. These changes will
    automatically apply when launching pict-rs 0.5.7. Upgrading should be as simple as pulling a new
    version of pict-rs.

    Descriptions

    Blurhash Endpoint

    A new endpoint at /image/blurhash has been added for generating blurhashes from uploaded media. A
    blurhash is a short string that encodes a few notable color values from an image that can be
    reconstructed into a blurred approximation of the original image. Notably, blurhashes are used by
    Mastodon to act as placeholders for sensitive media. For more information about blurhashes, see
    blurha.sh.

    This endpoint is powered by my new blurhash encoding library,
    blurhash-update.

    On success, the blurhash endpoint returns the following JSON.

    {
        "msg": "ok",
        "blurhash": "LGF5]+Yk^6#M@-5c,1J5@[or[Q6."
    }
    

    pict-rs does not provide a blurhash decoding mechanism (it would defeat the purpose of blurhashes to
    do so).

    File Path Changes

    pict-rs has dropped its dependency on my storage-path-generator library in favor of using UUIDs to
    create unique file paths for uploaded media. This means that newly uploaded media will be stored in
    a different directory structure, and with different filenames. The purpose of this is to reduce
    database use by removing the need to synchronize the current path state.

    pict-rs 0.5.7 also adds file extensions back to file paths, since they are now somewhat-publicly
    visible (when using the public_endpoint configuration with object storage).

    Neither of these changes affect previously uploaded media.

    Performance Improvements

    pict-rs 0.5.7 now buffers media in memory less frequently, opting to stream bytes directly from
    sources to sinks. This should improve general memory use, as well as decrease time pict-rs would
    spend waiting to aggregate bytes.

    pict-rs also no longer requires bytes be present in contiguous buffers, avoiding large allocations
    and reducing memcpys.

    More Consistent Errors

    pict-rs 0.5 introduced machine-readable error codes that returned alongside error messages, but
    there were a couple locations in pict-rs that failed to include them. pict-rs 0.5.7 resolves this,
    ensuring all error paths properly return codes.

    APNG Detection

    pict-rs 0.5.7 fixes the imagemagick delegate policy for ffmpeg, which allows for properly detecting
    certain media, notably APNG files. pict-rs should once again be able to properly handle uploaded
    APNGs.

    Downloads
  • v0.4.8 42d3427dae

    pict-rs 0.4.8 Stable

    asonix released this 2024-02-17 18:00:52 +00:00 | 498 commits to main since this release

    pict-rs 0.4.8

    Overview

    This is a maintenance release. There's no meaningful changes outside of dependency upgrades and use
    of the new CI system (Migrated from Drone to Forgejo Actions).

    Upgrade Notes

    There's no significant changes from 0.4.7, so upgrading should be as simple as pulling a new version
    of pict-rs.

    Downloads
  • v0.5.6 533db60fdd

    pict-rs 0.5.6 Stable

    asonix released this 2024-02-05 20:38:39 +00:00 | 156 commits to main since this release

    pict-rs 0.5.6

    Overview

    pict-rs 0.5.6 adds more configuration for imagemagick security policies and updates the default
    value for max_area

    Features

    Changes

    Upgrade Notes

    There's no significant changes from 0.5.5, so upgrading should be as simple as pulling a new version
    of pict-rs.

    Descriptions

    More Imagemagick Configuration

    Three new configuration values have been added to the imagemagick security configuration for
    pict-rs: memory, map, and disk. These options describe sizes for three tiers of storage that
    imagemagick is allowed to use when processing media. The first is memory, this is a simple value
    that represents how much RAM imagemagick is allowed to use to store image pixels. If this size is
    exceeded, it will start using the next tier of storage for image pixels, which is map. map
    represents space on disk that's mapped into RAM for quicker access. Since it's disk-backed, it can
    be larger than memory. Finally, if map is exceeded, imagemagick will start using the disk for
    storing pixels without mapping into memory. If the disk size is exceeded, media processing is
    aborted.

    The configuration for these values can be set via the pict-rs.toml file, via environment variables,
    or via the commandline.

    # pict-rs.toml
    # values are in MiB
    [media.magick]
    memory = 256
    map = 512
    disk = 1024
    
    # environment variables
    # values are in MiB
    PICTRS__MEDIA__MAGICK__MEMORY=256
    PICTRS__MEDIA__MAGICK__MAP=512
    PICTRS__MEDIA__MAGICK__DISK=1024
    
    # commandline
    # values are in MiB
    pict-rs run \
        --media-magick-memory 256 \
        --media-magick-map 512 \
        --media-magick-disk 1024
    

    Metrics Descriptions

    pict-rs now has descriptions for each of the metrics it emits. They're all pretty short for now, but
    can be expanded in the future if there's confusion. This is only useful when pict-rs is configured
    to host a prometheus scrape endpoint.

    Imagemagick Area Defaults

    The default value for max_area has been decreased from 40 million to 20 thousand. The reason for
    this is it doesn't impose a hard limit on the area of uploaded images, it instead imposes a limit on
    how much of an image can be held in memory at a time, with the rest of the image residing on disk.

    Imagemagick Frame Configuration

    Imagemagick now inherits pict-rs' animation max_frame_count value to set it's maximum
    list-length, which should allow longer animations to be configured.

    Downloads
  • v0.5.5 a6c4acd6f3

    pict-rs 0.5.5 Stable

    asonix released this 2024-02-04 04:10:43 +00:00 | 164 commits to main since this release

    pict-rs 0.5.5

    Overview

    pict-rs 0.5.5 adds a bugfix for uploading images with trailing bytes and few new features for
    advanced deployments.

    Features

    Bugfixes

    Upgrade Notes

    There's no significant changes from 0.5.4, so upgrading should be as simple as pulling a new version
    of pict-rs.

    Descriptions

    Imagemagick Security Policy Configuration

    pict-rs now supports configuring the imagemagick security policy via the pict-rs.toml file,
    environment variables, or via the commandline. The security policy defines the boundaries that
    imagemagick will operate with, and will allow it to abort processing media that would exceed those
    boundaries.

    Currently, there are only a few items that can be configured.

    # pict-rs.toml
    [media.magick]
    max_width = 10000
    max_hight = 10000
    max_area = 40000000
    
    # environment variables
    PICTRS__MEDIA__MAGICK__MAX_WIDTH=10000
    PICTRS__MEDIA__MAGICK__MAX_HEIGHT=10000
    PICTRS__MEDIA__MAGICK__MAX_AREA=40000000
    
    # commandline
    pict-rs run \
        --media-magick-max-width 10000 \
        --media-magick-max-height 10000 \
        --media-magick-max-aread 40000000
    

    It will also apply the configured process_timeout to the security policy.

    Serving with TLS

    pict-rs can now be configured to serve itself over TLS if provided with a server key and a server
    certificate. This is for more advanced deployments that have Certificate Authority infrastructure in
    place. When serving over TLS, downstream services need to be configured to access pict-rs over TLS.

    # pict-rs.toml
    [server]
    certificate = "/path/to/server.crt"
    private_key = "/path/to/server.key"
    
    # environment variables
    PICTRS__SERVER__CERTIFICATE=/path/to/server.crt
    PICTRS__SERVER__PRIVATE_KEY=/path/to/server.key
    
    # commandline
    pict-rs run \
        --certificate /path/to/server.crt \
        --private-key /path/to/server.key
    

    Broken Pipe Error

    In previous 0.5 releases with the default configurations, it was possible for valid images to fail
    to upload if they contained excess trailing bytes. This was caused by exiftool completing metadata
    processing on the image bytes before pict-rs had written the entire buffer to exiftool's stdin. The
    fix was to simply treat the case of stdin closing early as a success, rather than a failure. In the
    event there was actually an error in exiftool, the command will fail and pict-rs will return a
    proper status error instead.

    Downloads
  • v0.5.4 a4f1a1e843

    pict-rs 0.5.4 Stable

    asonix released this 2024-01-25 23:11:40 +00:00 | 190 commits to main since this release

    pict-rs 0.5.4

    Overview

    Another quick release to improve postgres performance, and reduce growth of the job_queue during
    cleanup spikes.

    Changes

    Descriptions

    Limit LISTEN Connections

    When postgres is configured, pict-rs sometimes issues LISTEN queries in order to be notified of new
    jobs in the job queue, due to connection pooling, it would be possible for every connection pict-rs
    makes to postgres to be listening for notifications. This is needless and wastes resources. 0.5.4
    now limits the number of connections issuing LISTEN queries to at most 4, or as few as the number of
    cores in your machine, if less than 4.

    Limit Spawned Jobs

    As mentioned in the 0.5.2 release document, pict-rs runs cleanup jobs that can spawn a lot of other
    jobs. In order to keep the queue size from growing too large for larger deployments, pict-rs now
    limits each general cleanup job to 2048 iterations, meaning it will complete after spawning just
    2048 jobs.

    Since pict-rs processes jobs in-order, this means that the majority of these jobs will be completed
    before the next general cleanup job runs, keeping the maximum number of queued jobs around 2048.

    Downloads
  • v0.5.3 e22fd7aa40

    pict-rs 0.5.3 Stable

    asonix released this 2024-01-25 14:57:40 +00:00 | 196 commits to main since this release

    pict-rs 0.5.3

    Overview

    A quick release after 0.5.2 to add a default value for use_tls, and to fix launching pict-rs with
    postgres using commandline flags.

    Bugfixes

    Upgrade Notes

    There's no significant changes from 0.5.2, so upgrading should be as simple as pulling a new version
    of pict-rs.

    Descriptions

    Semver Break in 0.5.2

    When adding TLS support to postgres in 0.5.2, I had forgotten to set a default value for the new
    use_tls configuration option, leading to postgres deployments no longer launching after upgrade.
    This has been fixed by setting it to false by default, which aligns with the previous behavior of
    not using TLS.

    Postgres Commandline Configuration Panic

    When adding TLS support to postgres in 0.5.2, I had accidentally added two conflicting commandline
    arguments that expected to use -u. The first is for the postgres URL, and the second is for the
    use_tls setting. This caused panics when attempting to configure postgres with commandline flags.
    0.5.3 fixes this by changing use_tls to use the -t short flag, rather than -u.

    Downloads
  • v0.5.2 2f588aa613

    pict-rs 0.5.2 Stable

    asonix released this 2024-01-24 23:41:42 +00:00 | 199 commits to main since this release

    pict-rs 0.5.2

    Overview

    A quick release to avoid a bug leading to incredibly large cleanup queue sizes and resulting in high
    CPU use. Upgrading to 0.5.2 is recommended for any deployment that recently upgraded to 0.5.0 or
    0.5.1.

    Bugfixes

    Changes

    Upgrade Notes

    There's no significant changes from 0.5.1, so upgrading should be as simple as pulling a new version
    of pict-rs.

    Descriptions

    At-Most Once Cleanup Jobs

    pict-rs 0.5.2 now is smarter about queueing certain cleanup jobs. Previously, it might queue an
    arbitrary number of the same cleanup job while under significant load, resulting in many redundant
    jobs consuming CPU. The reason for this is pict-rs' strategy for handling variants and proxied media
    that hadn't been accessed within the configured timeframe was to queue a job to check every 30
    seconds. This is fine in normal conditions, but can be a problem after upgrading to 0.5 from 0.4.

    During the upgrade, pict-rs marks all variants as having been accessed at the time of the upgrade,
    since this information was not tracked in 0.4. This results in many variants being queued for
    cleanup exactly 1 week (or otherwise configured cache duration) after upgrading. When the cleanup
    queue fills with these cleanup jobs, it takes longer to process them all than the 30 second window
    between queues of the general cleanup jobs, resulting in redundant cleanup jobs that spawn the same
    redundant removal jobs, increasing the queue size by the number of outdated variants each time.

    By adding an option to the queueing system to mark certain jobs as unique, pict-rs can now ensure
    that only one copy of these general cleanup jobs can be queued or running at any given time,
    ensuring that all individual variant cleanup jobs will be queued after a general cleanup job runs
    before the next general cleanup job is queued.

    Support TLS in Postgres Connections

    The postgres repo now supports some new options to enable connecting via TLS. If the postgres
    server's certificate is not generally trusted, a CA certificate can be provided to pict-rs to allow
    it to trust the postgres connection.

    If the postgres endpoint uses TLS and has a publicly trusted certificate, the configuration can be
    updated to simply add use_tls = true

    [repo]
    type = 'postgres'
    url = 'postgres://pictrs:1234@localhost:5432/pictrs'
    use_tls = true
    

    or with environment variables

    PICTRS__REPO__TYPE=postgres
    PICTRS__REPO__URL=postgres://pictrs:1234@localhost:5432/pictrs
    PICTRS__REPO__USE_TLS=true
    

    If the server uses a self-signed certificate, the CA's certificate can be added to pict-rs' trusted
    CAs.

    [repo]
    type = 'postgres'
    url = 'postgres://pictrs:1234@localhost:5432/pictrs'
    use_tls = true
    certificate_file = '/path/to/ca/certificate.crt'
    

    or with environment variables

    PICTRS__REPO__TYPE=postgres
    PICTRS__REPO__URL=postgres://pictrs:1234@localhost:5432/pictrs
    PICTRS__REPO__USE_TLS=true
    PICTRS__REPO__CERTIFICATE_FILE=/path/to/ca/certificate.crt
    

    Compiler Version Requirements

    pict-rs 0.5.2 now takes advantage of rust's native support for AFIT (Async Fn In Trait) for the
    store trait. This trait already was not object-safe, so using AFIT does not come with any downsides.
    This does mean that the minimum supported rust version for building pict-rs is now 1.75, which
    released in December 2023.

    Downloads