133 Releases 304 Tags

RSS Feed

• v0.5.13 6d40fbee47

  Compare

  pict-rs 0.5.13 Stable

  asonix released this 2024-04-15 21:52:29 +00:00 | 24 commits to main since this release

  pict-rs 0.5.13

  pict-rs is a simple image hosting microservice, designed to handle storing and retrieving images,
  animations, and videos, as well as providing basic image processing functionality.

  Overview

  pict-rs 0.5.13 is a maintenance release aiming to enable better logging in some scenarios.

  Features

  • Colorless Logging

  Changes

  • Remove Flume

  Upgrade Notes

  There are no significant changes from 0.5.12. Upgrading should be as simple as pulling a new version
  of pict-rs.

  Descriptions

  Colorless Logging

  When opting to use the json logger, the tracing subscriber automatically disables colored output.
  This didn't remove colors from errors, though, and pict-rs hasn't had a way to disable colors while
  using other log formats. pict-rs 0.5.13 introduces a new configuration value to remove colored
  output from all logs regardless of logging format.

  With pict-rs.toml

  [tracing.logging]
  no_ansi = true
  

  With environment variables

  PICTRS__TRACING__LOGGING__NO_ANSI=true
  

  With commandline flags

  pict-rs --no-log-ansi run
  

  Colors in logs can be useful, so I imagine this option won't be used much. There has been a request
  for this functionality though and it's little cost to maintain.

  Remove Flume

  Recently I've been debugging a memory usage issue in another project of mine. I wasn't able to fully
  track down the cause, but I did notice that removing the
  flume channel library seemed to make the leak go away. Since I
  also use flume in pict-rs, I'm opting to replace it with tokio's native channel implementation. This
  may or may not improve memory usage, but it does reduce the depenency count and therefore build time
  for pict-rs.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    20 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    2 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    3 downloads · 28 MiB
• v0.5.12 4bb3bad703

  Compare

  pict-rs 0.5.12 Stable

  asonix released this 2024-04-05 18:29:35 +00:00 | 27 commits to main since this release

  pict-rs 0.5.12

  pict-rs is a simple image hosting microservice, designed to handle storing and retrieving images,
  animations, and videos, as well as providing basic image processing functionality.

  Overview

  pict-rs 0.5.12 is a bugfix release to remove two issues that, when compounded, would cause pict-rs
  to fail to process media.

  Fixes

  • Panic Handling in Background Jobs
  • BytesStream Divide-by-Zero

  Upgrade Notes

  There are no significant differences from 0.5.11. Upgrading should be as simple as pulling a new
  version of pict-rs.

  Descriptions

  Panic Handling in Background Jobs

  pict-rs makes an effort to never use explicitly panicking code, but since there's no static way to
  guarantee that a given function wont panic, pict-rs needs to be able to deal with that. pict-rs
  0.5.12 now wraps invocations of jobs in spawned tasks, which can catch and report panics that happen
  in background jobs.

  Previously, a panic in a background job would bring down that thread's job processor, which resulted
  in future jobs never being processed. Now job processing should properly continue after panics
  occur.

  BytesStream Divide-by-Zero

  Part of my rework of BytesStream recently included adding debug logs around how many bytes chunks
  were in a given stream, and their average length. Unfortunately, if there were no bytes in the
  stream, this would cause the "average chunk length" calculation to divide by 0. In previous versions
  of pict-rs, this would generally result in a failed request for processed media, but in pict-rs
  0.5.11 this would end up killing the background jobs processor.

  This specific panic has been fixed by ensuring we divide by the number of chunks or 1, whichever is
  greater.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    6 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    4 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    4 downloads · 28 MiB
• v0.5.11 cef9a68307

  Compare

  v0.5.11 Stable

  asonix released this 2024-04-01 23:33:34 +00:00 | 33 commits to main since this release

  pict-rs 0.5.11

  pict-rs is a simple image hosting microservice, designed to handle storing and retrieving images,
  animations, and videos, as well as providing basic image processing functionality.

  Overview

  pict-rs 0.5.11 introduces new per-upload media validations, and new per-upload media processing.
  These features will enable applications to be more precise about their media requirements, such as
  allowing different media types and sizes for different endpoints, or pre-processing certain media to
  optimize for size.

  Features

  • Upload Validations
  • Upload Processing

  Changes

  • Backgrounded Variants

  Upgrade Notes

  For postgres-based installations, a small migration will be run when pict-rs 0.5.11 first launches
  to create a new notifications table. No manual intervention is required. Upgrading should be as
  simple as pulling a new version of pict-rs.

  Descriptions

  Upload Validations

  When ingesting media using POST /image, POST /image/backgrounded, POST /internal/import, or
  GET /image/download, validations can now be applied per-upload. These can be provided in the
  request query. The following query parameters are supported:

  • max_width: maximum width, in pixels, allowed for the uploaded media
  • max_height: maximum height, in pixels, allowed for the uploaded media
  • max_area: maximum area, in pixels, allowed for the uploaded media
  • max_frame_count: maximum number of frames permitted for animations and videos
  • max_file_size: maximum size, in megabytes, allowed
  • allow_image: whether to permit still images in the upload
  • allow_animation: whether to permit animations in the upload
  • allow_video: whether to permit video in the upload

  An example request could look like this: POST /image/backgrounded?max_area=3200&allow_video=false

  Validations are performed in addition to the validations specified in the pict-rs configuration, so
  if uploaded media violates any of the validations, it will fail to ingest.

  Upload Processing

  In a similar vein to the upload validations, preprocessing steps can now be applied on a per-upload
  basis. These are also provided as query parameters, and will be applied instead of the configured
  preprocess steps. The preprocess query parameters are provided and processed the same way as in the
  GET image/process.{ext} endpoint.

  An example request could be POST /image/backgrounded?blur=2.5&resize=300, which would blur the
  uploaded image and fit it inside a 300x300 box before saving it.

  Backgrounded Variants

  When serving images from the /process.{ext} endpoint, pict-rs will now queue the processing to
  happen via the job queue, rather than processing media inline. It will still wait up to 30 seconds
  for the processing to be complete, and return the processed image the same way it always has.

  If processing exceeds 30 seconds, pict-rs will return a timeout error, but the processing will
  continue in the background. The same variant can be requested again, and it will wait for the same
  background process to complete, rather than trying to process the variant a second time.

  pict-rs has historically had a method of reducing variant processing to prevent two requests for the
  same variant from doing the same work, but this was only effective in environments that only ran 1
  copy of pict-rs. In environments that had multiple replicas, each one could end up processing the
  same variant if it was requested more than once at a time. This has been solved by using postgres as
  a notification system to enable globally unique processing for a given variant.

  In sled-based configurations there shouldn't be a noticible difference, aside from the 30 second
  timeout on variant endpoints.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    6 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    6 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    6 downloads · 28 MiB
• v0.5.10 5f850f8c86

  Compare

  pict-rs 0.5.10 Stable

  asonix released this 2024-03-24 00:29:39 +00:00 | 52 commits to main since this release

  pict-rs 0.5.10

  Overview

  pict-rs 0.5.10 is a small release with changes to how pict-rs handles temporary files.

  Changes

  • Temporary File Cleanup

  Upgrade Notes

  There are no significant changes from 0.5.9. Upgrading should be as simple as pulling the new
  version.

  Descriptions

  Temporary File Cleanup

  pict-rs now nests its temporary files inside a pict-rs toplevel temporary folder. This is useful
  because pict-rs 0.5.10 introduces a new behavior: it will completely delete that folder and its
  contents on launch. If you are running multiple copies of pict-rs on the same host and they share
  your temporary folder, this might cause problems. In that scenario, this behavior can be disabled by
  setting PICTRS__SERVER__CLEANUP_TEMPORARY_DIRECTORY=false or passing
  --no-cleanup-temporary-directory on the commandline.

  This new behavior has been introduced in order to better clean up after crashes. If pict-rs is
  killed while processing media, maybe due to an OOM, it will leave files behind in the temporary
  directory. This can cause the temporary directory to grow, leading to memory or disk problems.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    8 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    5 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    5 downloads · 28 MiB
• v0.5.9 2ead3e00e2

  Compare

  pict-rs 0.5.9 Stable

  asonix released this 2024-03-11 18:51:49 +00:00 | 57 commits to main since this release

  pict-rs 0.5.9

  Overview

  pict-rs 0.5.9 is a bugfix release for 0.5.8. All deployments on 0.5.8 should upgrade to 0.5.9

  Fixes

  • Fix Postgres Pooling
  • Fix io-uring feature

  Upgrade Notes

  There are no significant changes from 0.5.8. Upgrading should be as simple as pulling the new
  version.

  Descriptions

  Fix Postgres Pooling

  When pict-rs 0.5.8 was built without the tokio_unstable flag, it would use tokio's spwan_local
  utility from outside a LocalSet, leading to panics and timeouts. This release replaces the use of
  spawn_local with spawn in that scenario.

  Fix io-uring Feature

  As mentioned in the 0.5.8 release notes, io-uring is not considered to be a stable feature. However,
  0.5.9 should make it usable again. Instead of manually launching a tokio_uring runtime, pict-rs once
  again relies on actix-rt to configure a System for use with io-uring.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    17 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    5 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    5 downloads · 28 MiB
• v0.5.8 7021c50156

  Compare

  pict-rs 0.5.8 Stable

  asonix released this 2024-03-11 04:10:06 +00:00 | 62 commits to main since this release

  pict-rs 0.5.8

  Overview

  pict-rs 0.5.8 improves reliability of deletions by allowing background tasks to be retried.
  Otherwise changes are fairly minor.

  Changes

  • Improved Task Reliability
  • Improved Latency

  Upgrade Notes

  There is a small repo format migration between 0.5.7 and 0.5.8. For sled it's simply opening a new
  tree, for postgre it involves adding a new column to the job_queue table. These changes will
  automatically apply when launching pict-rs 0.5.8. Upgrading should be as simple as pulling a new
  version of pict-rs.

  Configuration Notes

  Check your configurations to make sure you haven't enabled the tokio-console integration unless
  you're using it. In my local testing, I've found the console subscriber to use a significant amount
  of CPU. While it is very useful for debugging, it shouldn't be used generally in production.

  The relevant configuration values are PICTRS__TRACING__CONSOLE__ADDRESS with environment variables
  or [tracing.console] address = "" in the toml.

  Packaging Notes

  While I have never recommended packaging pict-rs with non-default crate features enabled, and the
  binaries and containers I provide enable only the default features, there are two new crate features
  in this release that I would advise against enabling in downstream packaging environments.

  The new features are poll-timer-warnings and random-errors. These are each described below if
  you want to learn about them, but as a general recommendation, do not enable non-default features
  when packaging pict-rs (yes, i'm talking to you grawlinson from the AUR).

  The other optional feature, io-uring, is considered less stable. It's possible that folks will
  find it works alright, and maybe Arch can enable it since they can assume recent kernels, but I
  don't personally test much with io-uring. It exists mostly as a historical curiosity. Please
  consider carefully before enabling io-uring for pict-rs.

  Descriptions

  Improved Task Reliability

  pict-rs 0.5.8 adds the ability for tasks to be retried. pict-rs generally spawns background tasks to
  handle things like Image deletion or other cleanup operations. Until now, if a background task
  failed, the only indication would be a warning that appeared in the logs. These warnings are
  generally descriptive and help track the error source, but end users aren't notified, and the repo
  or store state can become inconsistant.

  With the newly added ability to retry tasks, operations should be completed more reliably. By
  default, a failed task will be retried after a 2 minute wait, and if it continues to fail, it will
  be retried up to five times. If a task fails after 5 retries, an additional warning will be output
  to the log.

  In order to test this, I've added a new optional crate feature called random-errors, which will
  inject errors into various pict-rs operations randomly. This feature should never be enabled in
  production scenarios, and two warnings will be printed when launching pict-rs if it was compiled
  with this feature enabled.

  Improved Latency

  pict-rs 0.5.8 implements a couple new techniques to improve system latency.

  1. The postgres connection pooling library has been swapped from deadpool to bb8. Not only does this
     (slightly) improve connection pool access times, but it also means pict-rs is no longer pinned
     to an outdated version of deadpool.
  2. Processes like ffmpeg, imagemagick, and exiftool are now spawned from background threads,
     rather than from within the webserver threads. This is notable, since the act of spawning a
     process ends up using a good amount of time, and prevents other requests from being handled
     until the spawning has completed.
  3. pict-rs now has the ability to monitor polling times for futures. By default, any task pict-rs
     spawns itself will be monitored to report polling times, and a trait has been added to enable
     easily tracking more polling times in the future. These polling times will appear in the
     prometheus metrics, as well as in logs at DEBUG or TRACE visibility. There's an optional crate
     feature called poll-timer-warnings that will upgrade some of these logs to WARN visibility.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    4 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    5 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    5 downloads · 28 MiB
• v0.5.7 40bb58d603

  Compare

  pict-rs 0.5.7 Stable

  asonix released this 2024-03-03 17:35:32 +00:00 | 72 commits to main since this release

  pict-rs 0.5.7

  Overview

  pict-rs 0.5.7 adds support for generating blurhashes from images and includes a couple unrelated
  fixes and tweaks.

  Features

  • Blurhash Endpoint

  Changes

  • File Path Changes
  • Performance Improvements

  Fixes

  • More Consistent Errors
  • APNG Detection

  Upgrade Notes

  There is a small repo format migration between 0.5.6 and 0.5.7. For sled it's simply opening a new
  tree, for postgres it involves adding a new column to the hashes table. These changes will
  automatically apply when launching pict-rs 0.5.7. Upgrading should be as simple as pulling a new
  version of pict-rs.

  Descriptions

  Blurhash Endpoint

  A new endpoint at /image/blurhash has been added for generating blurhashes from uploaded media. A
  blurhash is a short string that encodes a few notable color values from an image that can be
  reconstructed into a blurred approximation of the original image. Notably, blurhashes are used by
  Mastodon to act as placeholders for sensitive media. For more information about blurhashes, see
  blurha.sh.

  This endpoint is powered by my new blurhash encoding library,
  blurhash-update.

  On success, the blurhash endpoint returns the following JSON.

  {
      "msg": "ok",
      "blurhash": "LGF5]+Yk^6#M@-5c,1J5@[or[Q6."
  }
  

  pict-rs does not provide a blurhash decoding mechanism (it would defeat the purpose of blurhashes to
  do so).

  File Path Changes

  pict-rs has dropped its dependency on my storage-path-generator library in favor of using UUIDs to
  create unique file paths for uploaded media. This means that newly uploaded media will be stored in
  a different directory structure, and with different filenames. The purpose of this is to reduce
  database use by removing the need to synchronize the current path state.

  pict-rs 0.5.7 also adds file extensions back to file paths, since they are now somewhat-publicly
  visible (when using the public_endpoint configuration with object storage).

  Neither of these changes affect previously uploaded media.

  Performance Improvements

  pict-rs 0.5.7 now buffers media in memory less frequently, opting to stream bytes directly from
  sources to sinks. This should improve general memory use, as well as decrease time pict-rs would
  spend waiting to aggregate bytes.

  pict-rs also no longer requires bytes be present in contiguous buffers, avoiding large allocations
  and reducing memcpys.

  More Consistent Errors

  pict-rs 0.5 introduced machine-readable error codes that returned alongside error messages, but
  there were a couple locations in pict-rs that failed to include them. pict-rs 0.5.7 resolves this,
  ensuring all error paths properly return codes.

  APNG Detection

  pict-rs 0.5.7 fixes the imagemagick delegate policy for ffmpeg, which allows for properly detecting
  certain media, notably APNG files. pict-rs should once again be able to properly handle uploaded
  APNGs.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    12 downloads · 31 MiB
  • pict-rs-linux-arm32v7
    6 downloads · 26 MiB
  • pict-rs-linux-arm64v8
    5 downloads · 27 MiB
• v0.4.8 42d3427dae

  Compare

  pict-rs 0.4.8 Stable

  asonix released this 2024-02-17 18:00:52 +00:00 | 544 commits to main since this release

  pict-rs 0.4.8

  Overview

  This is a maintenance release. There's no meaningful changes outside of dependency upgrades and use
  of the new CI system (Migrated from Drone to Forgejo Actions).

  Upgrade Notes

  There's no significant changes from 0.4.7, so upgrading should be as simple as pulling a new version
  of pict-rs.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    9 downloads · 22 MiB
  • pict-rs-linux-arm32v7
    6 downloads · 19 MiB
  • pict-rs-linux-arm64v8
    6 downloads · 20 MiB
• v0.5.6 533db60fdd

  Compare

  pict-rs 0.5.6 Stable

  asonix released this 2024-02-05 20:38:39 +00:00 | 202 commits to main since this release

  pict-rs 0.5.6

  Overview

  pict-rs 0.5.6 adds more configuration for imagemagick security policies and updates the default
  value for max_area

  Features

  • More Imagemagick Configuration
  • Metrics Descriptions

  Changes

  • Imagemagick Area Defaults
  • Imagemagick Frame Configuration

  Upgrade Notes

  There's no significant changes from 0.5.5, so upgrading should be as simple as pulling a new version
  of pict-rs.

  Descriptions

  More Imagemagick Configuration

  Three new configuration values have been added to the imagemagick security configuration for
  pict-rs: memory, map, and disk. These options describe sizes for three tiers of storage that
  imagemagick is allowed to use when processing media. The first is memory, this is a simple value
  that represents how much RAM imagemagick is allowed to use to store image pixels. If this size is
  exceeded, it will start using the next tier of storage for image pixels, which is map. map
  represents space on disk that's mapped into RAM for quicker access. Since it's disk-backed, it can
  be larger than memory. Finally, if map is exceeded, imagemagick will start using the disk for
  storing pixels without mapping into memory. If the disk size is exceeded, media processing is
  aborted.

  The configuration for these values can be set via the pict-rs.toml file, via environment variables,
  or via the commandline.

  # pict-rs.toml
  # values are in MiB
  [media.magick]
  memory = 256
  map = 512
  disk = 1024
  

  # environment variables
  # values are in MiB
  PICTRS__MEDIA__MAGICK__MEMORY=256
  PICTRS__MEDIA__MAGICK__MAP=512
  PICTRS__MEDIA__MAGICK__DISK=1024
  

  # commandline
  # values are in MiB
  pict-rs run \
      --media-magick-memory 256 \
      --media-magick-map 512 \
      --media-magick-disk 1024
  

  Metrics Descriptions

  pict-rs now has descriptions for each of the metrics it emits. They're all pretty short for now, but
  can be expanded in the future if there's confusion. This is only useful when pict-rs is configured
  to host a prometheus scrape endpoint.

  Imagemagick Area Defaults

  The default value for max_area has been decreased from 40 million to 20 thousand. The reason for
  this is it doesn't impose a hard limit on the area of uploaded images, it instead imposes a limit on
  how much of an image can be held in memory at a time, with the rest of the image residing on disk.

  Imagemagick Frame Configuration

  Imagemagick now inherits pict-rs' animation max_frame_count value to set it's maximum
  list-length, which should allow longer animations to be configured.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    9 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    7 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    7 downloads · 27 MiB
• v0.5.5 a6c4acd6f3

  Compare

  pict-rs 0.5.5 Stable

  asonix released this 2024-02-04 04:10:43 +00:00 | 210 commits to main since this release

  pict-rs 0.5.5

  Overview

  pict-rs 0.5.5 adds a bugfix for uploading images with trailing bytes and few new features for
  advanced deployments.

  Features

  • Imagemagick Security Policy Configuration
  • Serving with TLS

  Bugfixes

  • Broken Pipe Error

  Upgrade Notes

  There's no significant changes from 0.5.4, so upgrading should be as simple as pulling a new version
  of pict-rs.

  Descriptions

  Imagemagick Security Policy Configuration

  pict-rs now supports configuring the imagemagick security policy via the pict-rs.toml file,
  environment variables, or via the commandline. The security policy defines the boundaries that
  imagemagick will operate with, and will allow it to abort processing media that would exceed those
  boundaries.

  Currently, there are only a few items that can be configured.

  # pict-rs.toml
  [media.magick]
  max_width = 10000
  max_hight = 10000
  max_area = 40000000
  

  # environment variables
  PICTRS__MEDIA__MAGICK__MAX_WIDTH=10000
  PICTRS__MEDIA__MAGICK__MAX_HEIGHT=10000
  PICTRS__MEDIA__MAGICK__MAX_AREA=40000000
  

  # commandline
  pict-rs run \
      --media-magick-max-width 10000 \
      --media-magick-max-height 10000 \
      --media-magick-max-aread 40000000
  

  It will also apply the configured process_timeout to the security policy.

  Serving with TLS

  pict-rs can now be configured to serve itself over TLS if provided with a server key and a server
  certificate. This is for more advanced deployments that have Certificate Authority infrastructure in
  place. When serving over TLS, downstream services need to be configured to access pict-rs over TLS.

  # pict-rs.toml
  [server]
  certificate = "/path/to/server.crt"
  private_key = "/path/to/server.key"
  

  # environment variables
  PICTRS__SERVER__CERTIFICATE=/path/to/server.crt
  PICTRS__SERVER__PRIVATE_KEY=/path/to/server.key
  

  # commandline
  pict-rs run \
      --certificate /path/to/server.crt \
      --private-key /path/to/server.key
  

  Broken Pipe Error

  In previous 0.5 releases with the default configurations, it was possible for valid images to fail
  to upload if they contained excess trailing bytes. This was caused by exiftool completing metadata
  processing on the image bytes before pict-rs had written the entire buffer to exiftool's stdin. The
  fix was to simply treat the case of stdin closing early as a success, rather than a failure. In the
  event there was actually an error in exiftool, the command will fail and pict-rs will return a
  proper status error instead.

  Downloads

  • Source code (ZIP)
  • Source code (TAR.GZ)

  ---

  • pict-rs-linux-amd64
    9 downloads · 32 MiB
  • pict-rs-linux-arm32v7
    9 downloads · 27 MiB
  • pict-rs-linux-arm64v8
    9 downloads · 27 MiB