asonix/pict-rs
2
2
Fork 1
Code Issues 4 Pull requests Actions Projects Releases 141 Wiki Activity

141 Releases 312 Tags

RSS feed
  • v0.4.1 20daced620

    v0.4.1
    All checks were successful
    continuous-integration/drone/push Build is passing
    Details
    continuous-integration/drone/tag Build is passing
    Details
    Stable

    asonix released this 2023-07-17 01:58:09 +00:00 | 669 commits to main since this release

    pict-rs 0.4.1

    Overview

    Today I am releasing pict-rs 0.4.1. This includes a couple bugfixes and a few requested features.
    Most notably is the object-storage migration now uploads files concurrently. It should be much
    faster than before.

    Features

    Bugfix

    Upgrade Notes

    There's no significant changes from 0.4, so upgrading should be as simple as pulling a newer version
    of pict-rs.

    Descriptions

    Add Concurrency to Migration

    Adding concurrency means changing how pict-rs kept track of migration progress. Previously it was
    setting a single flag with the "newest" hash that it had migrated, so a resuming migration would
    iterate until it found that hash before continuing linearly. Now, pict-rs keeps track of each
    individual file. That way it does not matter the order in which files are migrated.

    The migration is hardcoded to try moving 32 files at a time, so ideally this would mean a 32x
    speedup over the previous linear version. Whether it actually ends up being so fast is yet to be
    seen.

    Improved Error Messages

    Previously errors encountered when failing to run a process were pretty opaque. pict-rs relies on
    running the ffmpeg, ffprobe, magick, and exiftool commands, and if it could not run them it
    would print something vague like "File not found" or "Permission Denied", without much context for
    what wasn't found or what was denied.

    Now, pict-rs properly informs the admin that "Required command magick not found" or "Cannot run
    command ffmpeg due to invalid permissions on binary".

    Additionally, the error messages returned in the API were not very useful either. "Error in ffmpeg"
    or "Error in imagemagick" are not super useful. Now pict-rs will return the "root cause" of errors,
    which is generally more helpful. There's still more work to be done here, so this topic will be
    mentioned in future releases.

    Configurable Object Storage Timeouts

    pict-rs now provides the ability to increase the HTTP timeout and the Signature Expiration for
    requests to object storage. This may not be super helpful, but if your object storage is
    particularly slow you can now increase the values.

    The relevant variables are PICTRS__SToRE__SIGNATURE_EXPIRATION and
    PICTRS__STORE__CLIENT_TIMEOUT.

    In toml, that would be

    [store]
signature_expiration = 15
client_timeout = 30

    Set Content-Type in Object Storage

    This is not immediately useful, but it means that the object storage will begin knowing about the
    content-type of uploaded files, so in the future when pict-rs supports serving directly from
    object-storage it will provide a meaningful content-type header to browsers.

    Alias Cleanup

    There was a bug introduced in 0.4 that made using the "delete" endpoint not actually remove the file
    from storage. It would un-link the alias so it couldn't be accessed, but full deletion never
    happened. This was a result of me reordering a couple lines in the alias cleanup. 0.4.1 fixes this
    issue by un-reordering them.

    ImageMagick Security Policy

    The security policy used in the pict-rs docker container has been updated to allow for more file
    types. This makes the new JSON-based imagemagick and ffmpeg output parsers work, and might make one
    or two rejected files now no longer be rejected. I don't know the full extent to which this was a
    real problem in 0.4, but regardless it's better now.

    Broken Pipe on Upload Error

    Previously if a file was uploaded that was too large, it would be rejected immediately with a file
    size error. This sometimes resulted in clients receiving a Broken Pipe error on the connection
    instead of properly receiving an HTTP Response. This has been fixed in the actix-form-data library
    (that I also maintain) by ensuring that we read the entire upload (dropping the bytes immediately)
    before replying.

    Downloads