595 commits

Author	SHA1	Message	Date
asonix	e0968dcce3	Remove serde-cbor, num_cpus	2023-09-24 15:22:57 -05:00
asonix	fd965bb1a5	Allow processing still images into animation formats	2023-09-24 13:20:33 -05:00
asonix	f3ef2d870b	Generate details during ingest	2023-09-24 12:58:16 -05:00
asonix	8f62b0c325	clippy	2023-09-24 12:42:23 -05:00
asonix	9ffdd0432f	Infer video thumbnail format from image format	2023-09-24 12:37:43 -05:00
asonix	6c921817e1	Enable thumbnailing animations, use dynamic dispatch for a number of async readers	2023-09-24 11:54:16 -05:00
asonix	cde4a72203	Add metrics to object storage body streams	2023-09-20 19:46:10 -05:00
asonix	49d3037358	Add metrics to object storage requests	2023-09-20 19:39:03 -05:00
asonix	7df6f3906e	Add timeout, metrics to external validation request	2023-09-20 19:29:19 -05:00
asonix	d0646dee25	Add config option for external validation timeout	2023-09-20 19:23:55 -05:00
asonix	4a5b9712aa	Combine map_ok and map_err underlying impl	2023-09-10 23:51:07 -04:00
asonix	2b10c48619	Update io-uring to use streem for file bytes	2023-09-10 23:45:24 -04:00
asonix	ea75ca24b5	Add track_caller to sync methods, update streem	2023-09-10 20:43:51 -04:00
asonix	b2674f06d0	More streme	2023-09-10 20:08:01 -04:00
asonix	1b97ac1c5a	Start replacing manual stream implementations with streem	2023-09-10 18:55:13 -04:00
asonix	858899b943	Add metrics & tracing to dropped uploads, turn pop into a single query	2023-09-07 19:20:41 -05:00
asonix	783df60f81	Fix on_conflict for postgres store migration path	2023-09-06 20:49:00 -05:00
asonix	a761daca34	clippy	2023-09-05 20:55:11 -05:00
asonix	e15a82c0c7	Add external validation check	2023-09-05 20:45:07 -05:00
asonix	bc49f8ca37	Migrate with timestamp	2023-09-04 22:25:11 -05:00
asonix	9c50bbe23a	match -> matches!	2023-09-04 22:06:28 -05:00
asonix	492b99922b	10 second worker delay after disconnect error	2023-09-04 22:04:15 -05:00
asonix	62ebc1c60a	Move timeout after metrics - timeouts count as incomplete	2023-09-04 22:02:59 -05:00
asonix	3bd0f78e75	Use with_timeout for all timeouts	2023-09-04 21:58:57 -05:00
asonix	a4b1ab7dfb	Instrument postgres db calls	2023-09-04 21:51:27 -05:00
asonix	37448722de	postgres: Rework job & upload notifications (more) ... postgres: Add metrics to job push & pop, upload wait sled: add upload wait metrics	2023-09-04 16:20:26 -05:00
asonix	00aa00c55c	postgres: Enable 'previous page' when less than full limit fit on previous page	2023-09-03 21:58:31 -05:00
asonix	a43de122f9	postgres: add already-claimed case, general: tracing paranoia	2023-09-03 21:30:47 -05:00
asonix	31caea438e	Fix slow connection pool access	2023-09-03 20:05:29 -05:00
asonix	8a892ba622	it might work	2023-09-03 18:21:46 -05:00
asonix	a2933dbebc	Implement all the todos	2023-09-03 17:11:34 -05:00
asonix	94cb2a9ef3	Pass clippy	2023-09-03 16:59:41 -05:00
asonix	443d327edf	Implement a couple more repo traits	2023-09-03 12:47:06 -05:00
asonix	eac4cd54a4	Initial work for pg notifications	2023-09-02 20:13:32 -05:00
asonix	e580e7701e	Finish implementing HashRepo	2023-09-02 19:13:00 -05:00
asonix	8921f57a21	Remove Identifier trait, Replace with Arc<str>	2023-09-02 18:38:02 -05:00
asonix	8eb4cda256	Implement first query	2023-09-02 13:52:15 -05:00
asonix	d475723087	Finish writing migrations & generate diesel schema	2023-09-02 13:35:30 -05:00
asonix	8c532c97e6	Initial postgres work	2023-09-02 11:52:55 -05:00
asonix	ba3a23ed43	Add error codes	2023-09-01 20:51:01 -05:00
asonix	fd74161c61	Rename some repo methods, generate UploadId in repo	2023-09-01 18:41:04 -05:00
asonix	cbb66f1b75	Add internal endpoint for deleting alias without a delete token	2023-09-01 17:25:13 -05:00
asonix	8fb90a6f69	Give a meaningful distinction for format in details	2023-08-31 16:26:45 -05:00
asonix	dce0827099	Increase default animation bounds, remove references to converting animations to videos	2023-08-30 21:55:53 -05:00
asonix	0b70ca49e4	clippy	2023-08-30 21:00:15 -05:00
asonix	721d9e1706	Don't try parsing an internal format for OldDetails	2023-08-30 20:55:38 -05:00
asonix	92ee60549f	Enable copying input av1 stream from mp4 to webm	2023-08-30 20:40:08 -05:00
asonix	b48a9233b2	Remove transcode from animation to video, make video transcoding 'optional' ... Video transcoding still happens, but in many cases the video stream is able to be copied verbatim rather than being decoded & encoded	2023-08-30 20:37:54 -05:00
asonix	08fd96c2f7	clippy nit	2023-08-29 13:29:40 -05:00
asonix	c0f6b1e988	Enable searching hashes by date	2023-08-29 13:27:18 -05:00
asonix	0146202236	Fully commit to HumanDate	2023-08-29 12:59:36 -05:00
asonix	64950bfe0e	Shorten pagination slugs	2023-08-29 12:53:29 -05:00
asonix	1271ff2cc7	Clippy	2023-08-28 18:49:30 -05:00
asonix	2b6ea97773	Enable setting timestamp on hash creation, improve paging api	2023-08-28 18:43:24 -05:00
asonix	24812afeba	Add paged hash access	2023-08-28 16:02:11 -05:00
asonix	e302ab1f3d	Migrate not-found alias	2023-08-28 13:40:19 -05:00
asonix	b28cc38c38	Simplify Streamer	2023-08-23 19:12:59 -05:00
asonix	95637fdfe5	Remove direct dep on futures-util	2023-08-23 19:10:10 -05:00
asonix	8f50a15b25	Reduce reliance on futures-util	2023-08-23 11:59:42 -05:00
asonix	0352ada9d8	Don't doulbe-open repo	2023-08-16 16:39:41 -05:00
asonix	88d4cc1a6a	Exit after repo migration	2023-08-16 16:34:42 -05:00
asonix	f95e61e402	Expose repo migrations	2023-08-16 16:32:19 -05:00
asonix	ba27a1a223	Build out repo-repo migration	2023-08-16 16:09:40 -05:00
asonix	1559d57f0a	Don't overwrite existing variants	2023-08-16 15:12:16 -05:00
asonix	22cfbe979d	Note how many failures have occurred for a given hash	2023-08-16 13:29:22 -05:00
asonix	ee88823d41	clippy	2023-08-16 12:43:12 -05:00
asonix	521cdd5b9d	Enforce format on new details	2023-08-16 12:36:18 -05:00
asonix	f3c6239979	Allow skipping details generation if already generated	2023-08-16 11:57:09 -05:00
asonix	4d909ba32a	Make migration concurrent	2023-08-16 11:47:36 -05:00
asonix	ac996ff832	Remove unused error	2023-08-15 21:25:04 -05:00
asonix	1de257bb07	Ensure access values are unique	2023-08-15 21:18:25 -05:00
asonix	bd3b668d81	Don't error on duplicate - happens on retried migration	2023-08-15 20:28:42 -05:00
asonix	9fe7410dd6	Add logging to migration	2023-08-15 19:33:19 -05:00
asonix	1ee77b19d8	Make Repo object safe. Create migration	2023-08-15 19:19:03 -05:00
asonix	09236d731d	Fix poll after completion, misused compare_and_swap	2023-08-14 21:17:57 -05:00
asonix	26ca3a7195	Remove associated Bytes type	2023-08-14 20:00:00 -05:00
asonix	da876fd553	Make it compile	2023-08-14 14:25:19 -05:00
asonix	3129f7844e	BROKEN: start work on hash discriminant	2023-08-13 22:06:42 -05:00
asonix	e4e93cddf9	Return JobId from push	2023-08-13 19:52:27 -05:00
asonix	5d4486fdf4	Finish queue implementation update for sled	2023-08-13 19:47:20 -05:00
asonix	35d0c065e9	Start work on 0.5 db format - initial queuerepo changes	2023-08-13 14:12:38 -05:00
asonix	0c4d632dda	Consider timeouts a client error	2023-08-05 16:18:06 -05:00
asonix	fee4ed1e3e	Add process timeout	2023-08-05 12:41:06 -05:00
asonix	5f12be0c6a	Clippy	2023-07-26 22:53:41 -05:00
asonix	2a59a6492a	Bring back hash_alias_key	2023-07-25 20:30:22 -05:00
asonix	bd3975f455	Simplify some repo functions, remove 0.3 migration	2023-07-25 20:08:18 -05:00
asonix	f221887599	Clippy	2023-07-23 16:04:17 -05:00
asonix	aac1bb7bc4	Finish media proxy implementation	2023-07-23 15:45:52 -05:00
asonix	4cca7d0f86	Prepare for proxying media	2023-07-23 12:10:36 -05:00
asonix	75bf425c6e	Update alias query, process query to accept proxy url	2023-07-23 11:45:10 -05:00
asonix	e59483c12c	Add ProxyRepo	2023-07-23 10:23:46 -05:00
asonix	d0c619bdf7	Clippy	2023-07-22 21:14:24 -05:00
asonix	77cdeab57e	Improve metrics, add job metrics	2023-07-22 21:11:28 -05:00
asonix	7dc8ede8eb	Gate cleanup behind read-only config	2023-07-22 20:26:28 -05:00
asonix	13d6e10886	Reduce log level on variants access	2023-07-22 20:20:58 -05:00
asonix	1f3298363f	Attempt to mark unmarked variants as accessed on launch	2023-07-22 19:55:50 -05:00
asonix	fe1f7c869f	Enable cleanup of variants ... Still TODO: doing a first pass to mark variants as accessed on launch	2023-07-22 19:41:50 -05:00
asonix	47e13ec04e	Mark variants as accessed on fetch	2023-07-22 18:50:04 -05:00
asonix	b9e6d67d15	Add retention configuration ... This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets	2023-07-22 18:37:25 -05:00
asonix	13387dec43	Fix specs	2023-07-22 17:58:03 -05:00
asonix	b786406ad0	Change IdentifierAccess to VariantAccess	2023-07-22 17:57:52 -05:00
asonix	ce0df080f4	Add prometheus metrics	2023-07-22 16:47:59 -05:00
asonix	d661732fb4	Tidy imports	2023-07-22 12:47:32 -05:00
asonix	a390b21df1	Make running pict-rs fluent	2023-07-22 12:39:39 -05:00
asonix	3ca994ee31	Replace global config with passed-around config ... This will enable spinning up pict-rs multiple times in the same process with different configurations	2023-07-22 12:31:01 -05:00
asonix	e1262a5fda	Extract preprocess steps as urlencoded in config deserialization	2023-07-22 11:38:09 -05:00
asonix	ac48003f45	Extract ProcessMap out of static	2023-07-22 11:16:19 -05:00
asonix	8d35c2449d	Simplify the cancel-safe processor	2023-07-22 10:46:16 -05:00
asonix	c1127b4117	Set content-length in PutObject, CompleteMultipart	2023-07-21 19:45:22 -05:00
asonix	2f0a3618d8	Replace awc with reqwest	2023-07-21 16:58:31 -05:00
asonix	fc14ac2038	Clippy	2023-07-18 21:59:27 -05:00
asonix	3d93300ba9	Add access repos for alias & identifier	2023-07-18 21:56:13 -05:00
asonix	127dd6cdad	Make max file count configurable	2023-07-18 20:32:17 -05:00
asonix	eeac900d7e	Add quality value to process method, move -coalesce earlier in pipelines	2023-07-18 16:18:01 -05:00
asonix	558605381d	Fix dangling unprocessed uploads ... Adds error boundary around backgrounded ingest	2023-07-17 22:32:08 -05:00
asonix	496bcc5dc1	Only include -b:v 0 for VP9 ... also simplify building arguments	2023-07-17 18:13:50 -05:00
asonix	3aecef3a9e	Remove println	2023-07-17 17:47:48 -05:00
asonix	b1bbc6b159	Attempt adding quality settings to pict-rs	2023-07-17 17:45:26 -05:00
asonix	c4c920191f	Add read-only mode	2023-07-17 14:24:49 -05:00
asonix	dd1d509bb1	Enable configuring download client timeout	2023-07-17 13:44:31 -05:00
asonix	5e8ab7856d	Extract Status errors into command-specific errors	2023-07-17 13:30:08 -05:00
asonix	2961aae6e3	Port migration changes from 0.4.1	2023-07-16 22:07:42 -05:00
asonix	a751d92436	Port error changes to 0.5	2023-07-16 21:51:14 -05:00
asonix	9871ad2ee0	Clippy	2023-07-14 23:16:57 -05:00
asonix	4d78ce7689	Fix store delegation for public_url	2023-07-14 15:23:07 -05:00
asonix	786f583d98	Add public_url to Store for optional redirects to public object store urls ... Add details generation to file migration to set the content-type for uploads	2023-07-14 14:53:37 -05:00
asonix	4e58388b41	Clean alias before checking hash option	2023-07-13 19:58:31 -05:00
asonix	afeac8294a	Fix Alias cleanup ordering ... Update dev.toml	2023-07-13 19:57:09 -05:00
asonix	ec6c9aac1e	Better discovery validation, proper invoking of imagemagick	2023-07-13 19:21:57 -05:00
asonix	9b1dd1f8c3	Copy process from 0.4.x branch	2023-07-13 19:21:28 -05:00
asonix	2c22f2ee3a	It compiles and runs, but doesn't work	2023-07-13 17:42:21 -05:00
asonix	27451971a6	Fix tests, add animated_webp test	2023-07-13 14:34:40 -05:00
asonix	33b83f97f2	It compiles ... Need to reinstate media limits	2023-07-13 13:48:59 -05:00
asonix	ad1837f9dd	VERY BROKEN: start replacing parts of pict-rs	2023-07-12 22:12:21 -05:00
asonix	58d9765594	WIP: Rework format detection and conversions ... - Add a bunch of types & methods that aren't used yet. The idea is creating the output from the known input parameters is pure, and works for any uploaded media - Introduce a new kind of media "animation" which is neither an image nor a video. It is likely that animations can be processed properly at the /image/process.{ext} endpoint with a little massaging TODO: - Integrate this into exiftool, ffmpeg, magick commands - Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a hint) - Infallible conversion from Details to OutputFile - this might be impossible given that we don't currently store more than a mime type and a frame count to give any sort of indication of real type - Try not to break API	2023-07-11 23:11:25 -05:00
asonix	691bca286c	Make object storage timeouts configurable	2023-07-11 13:01:58 -05:00
asonix	f5c39f9be5	Typo, error formatting	2023-07-10 17:15:43 -05:00
asonix	4be2eb67d1	Increase object storage signature expiry to 15 seconds	2023-07-10 16:28:51 -05:00
asonix	9004ecaadf	Better classify process related errors	2023-07-10 15:29:41 -05:00
asonix	e10cb1401d	Bump version	2023-07-09 15:16:04 -05:00
asonix	7dadbcf3ed	Replace UnsupportedFormat error with more helpful errors	2023-07-09 15:07:49 -05:00
asonix	1694f49436	Use json for ffmpeg, imagemagick details parsing	2023-07-09 14:50:58 -05:00
asonix	686be1d93b	Turn export_path into flag for sled	2023-07-08 19:56:26 -05:00
asonix	7768d4e58e	Add database exports ... This should help admins running pict-rs on traditional filesystems to keep valid backups of pict-rs' sled repo	2023-07-08 17:36:01 -05:00
asonix	4680f9c3aa	Instrument repo and store build methods	2023-07-08 10:42:03 -05:00
asonix	d51c348a3f	Run health checks on old and new stores before migrating	2023-07-07 14:05:13 -05:00
asonix	3ccb8ecd8c	Validate alias represents real file before setting not found	2023-07-07 13:33:27 -05:00
asonix	9e7376d411	Remove is_missing error, make Identifier and DeleteToken return Option on fetch	2023-07-07 13:17:26 -05:00
asonix	7274538b70	Try adding store healthcheck	2023-07-07 12:05:42 -05:00
asonix	c150903980	Expand store differentiation	2023-07-07 11:19:39 -05:00
asonix	75be876d3c	Differentiate missing files in old vs new store for migration	2023-07-07 11:03:10 -05:00
asonix	508dd4340b	Fix panic when migrating < 100 files, improve resumed migration messaging	2023-07-06 16:06:37 -05:00
asonix	3487cb0e30	Add ability to set 404 image ... Fix imagemagick blur arguments	2023-07-05 19:07:03 -05:00
asonix	e7e4876908	Explicitly handle cases of invalid aliases	2023-07-05 16:46:44 -05:00
asonix	23e67b9697	Enhance migration handling of missing identifiers ... Queue a cleanup task for hashes that don't have original file identifiers Be more specific about what field is missing when a field is missing	2023-07-05 09:52:19 -05:00
asonix	2274dfecb4	Fix io-uring - read_to_end was polling Stream mapped through StreamReader after completion	2023-07-05 00:35:07 -05:00
asonix	c8e84b8914	Greatly improve error information during migration	2023-07-02 21:07:15 -05:00
asonix	e6f17cbac6	Log on successful migration	2023-06-29 11:45:06 -05:00
asonix	01d3610f46	Add basic progress indication for store migration	2023-06-29 11:39:47 -05:00
asonix	0d1df5ebc2	Avoid failed identifier cleanup on failed duplicate upload	2023-06-23 11:43:15 -05:00
asonix	7ba2fc9b26	Fix duplicate uploads potentially purging existing uploads on failure	2023-06-23 11:39:43 -05:00
asonix	58f0c328d9	Fix typo in error message	2023-06-23 11:20:55 -05:00
asonix	a1d48cbb42	Share client between object store and request extractor ... Add config to limit connection pool size	2023-06-23 11:20:20 -05:00
asonix	071d788349	Include original error in object storage sendrequest error	2023-06-21 17:07:14 -05:00
asonix	a3ff072350	Remove unused cache duration parameter	2023-06-21 17:05:53 -05:00
asonix	6d2aef8cc0	Add jpegxl and avif support	2023-06-21 17:05:35 -05:00
asonix	554d852e68	Create unique errors for Repo and Store, separate from UploadError ... Add .is_not_found() for Store errors, wire up Object and File storage to properly set NotFound Allow skipping files that are not found during store migration	2023-06-20 15:59:08 -05:00
asonix	1214b51ad7	Make use of FilesystemDefaults when constructing migrate commands	2023-06-19 15:04:36 -05:00
asonix	a3a986638d	Improve gif handling. ... - Allow a fast-path exif cleaning if we'd re-encode a gif to a gif - Use single-frame palettes to better map transparency from frame to frame Unrelated: - decrease ffmpeg logging when creating thumbnails	2023-06-19 14:25:31 -05:00
asonix	22e526fd2a	Allow type underscore (probably in instrument)	2023-06-01 17:33:43 -05:00
asonix	cc314c37a6	Use libopus for ffmpeg	2023-03-09 21:29:56 -06:00
asonix	dc27200a35	Add transparency to gifs, rustfmt	2023-02-25 14:31:57 -06:00
asonix	97ba4e806b	Clippy	2023-02-25 13:01:55 -06:00
asonix	9a2f846cd1	Detect input transparency for gif/webm	2023-02-25 12:59:47 -06:00
asonix	281ac43dff	Remove cache functionality	2023-02-25 11:34:48 -06:00
asonix	123488f617	clippy	2023-02-12 23:18:27 -06:00
asonix	788b87eceb	Remove obsolute pre-0.3 migrate check	2023-02-12 23:13:47 -06:00
asonix	138499d47f	Use transpose in a couple places	2023-02-05 20:50:59 -06:00
asonix	cc75d2d882	Clippy	2023-02-04 17:54:56 -06:00
asonix	4cc810d372	Add control over gif frame count	2023-02-04 17:52:23 -06:00
asonix	88ca9793e8	s/vrames/frames:v	2023-02-04 17:47:03 -06:00
asonix	40f57be0c7	Allow uploading small gifs	2023-02-04 17:32:36 -06:00
asonix	0aa3f574a5	Clippy	2023-01-29 11:57:59 -06:00
asonix	f6d6d54b88	Update deps	2023-01-29 11:47:28 -06:00
asonix	f0f40db8c3	Add healthcheck for db	2023-01-29 11:36:09 -06:00
asonix	5e5dd27a05	Retry migrate_file, remove variant before relating new one	2023-01-04 18:58:05 -06:00
asonix	28f7a139a0	Retry in-process, and rate-limit self	2023-01-04 17:26:05 -06:00
asonix	c55b1115d1	Increase client timeout, attempt to keep track of migration progress for resuming	2023-01-04 15:58:32 -06:00
asonix	dabcc24f98	Parse environment variables - main app	2023-01-04 15:05:34 -06:00
asonix	a489411c6c	Parse environment variables	2023-01-04 14:42:04 -06:00
asonix	3c844d86c5	Allow configuring pict-rs through serializable types	2022-12-26 16:35:25 -06:00
asonix	2c1a37fb62	Extract pict-rs as library	2022-12-23 12:56:15 -06:00
asonix	74cd3392fa	Borrow parent spans when creating children	2022-12-07 22:43:12 -06:00
asonix	3325b222a2	clippy	2022-11-26 11:18:34 -06:00
asonix	7037ee5791	Support auto-orienting photos	2022-10-15 11:13:24 -05:00
asonix	669b3fb86f	Fix webp metadata stripping, more tracing cleanup	2022-10-01 22:47:52 -05:00
asonix	718f09c43a	Clean tracing, simplify validation, rename InputFormat -> VideoFormat	2022-10-01 21:17:18 -05:00
asonix	e7cf21f862	Allow concurrency in small object uploads	2022-10-01 13:06:19 -05:00
asonix	fe2a499110	Fix sending upload parts (set size), optimize small file uploads	2022-10-01 13:00:07 -05:00
asonix	0307dc5a3b	Introduce BytesStream to simplify aggregating, forwarding bytes	2022-10-01 12:06:33 -05:00