648 commits

Author	SHA1	Message	Date
asonix	db43392a3b	Simplify process future by removing explicit channel, don't lock mutex as often	2023-12-22 12:03:05 -06:00
asonix	c9155f7ce7	Include not_found key in repo migration, rework ProcessRead, add timeout to 0.4 migration	2023-12-17 23:15:27 -06:00
asonix	bd0b89b165	Clean spans a bit	2023-12-15 22:34:45 -06:00
asonix	2544cb478f	Add static bound to appease 1.74.1	2023-12-12 17:17:16 -06:00
asonix	f1c5a56353	Port prune_missing from 0.4.x	2023-12-12 16:54:41 -06:00
asonix	3ef4097ceb	Enable building without tokio_unstable	2023-12-12 16:23:02 -06:00
asonix	6daa745446	Add optional ability to log spans	2023-12-11 14:58:53 -06:00
asonix	cd80ab16c3	Downgrade retry warnings to debug	2023-12-11 13:08:10 -06:00
asonix	b452a577a7	Add object ID to not found error	2023-12-11 13:05:29 -06:00
asonix	7293628148	put timeout on collecting payloads	2023-12-10 19:26:16 -06:00
asonix	961de20a00	Add timeout and metrics to media processing	2023-12-10 19:11:36 -06:00
asonix	2523b6644b	Add AggregateTimeout, don't aggregate with permit	2023-12-04 15:39:34 -06:00
asonix	6b2ccbf975	Allow configuring pict-rs' temporary directory	2023-11-24 11:48:45 -06:00
asonix	a30044278c	Improve documentation for using pict-rs as a library	2023-11-24 11:38:14 -06:00
asonix	71f8f52b58	Tweak error reporting	2023-11-13 17:24:56 -06:00
asonix	1538389caf	Clippy	2023-11-11 14:28:05 -06:00
asonix	210af5d7d9	Add danger_dummy_mode	2023-11-11 14:22:12 -06:00
asonix	5fd59fc5b4	Change commandline flag to allow disabling video, since it is enabled by default	2023-11-11 13:22:12 -06:00
asonix	3a0546694f	Update opentelemetry dependencies	2023-11-10 15:19:54 -06:00
asonix	15189d293b	clippy	2023-11-09 18:26:57 -06:00
asonix	f61dac8187	Attempt to clean stray magick files ... This change sets a unique temp directory for each invocation of imagemagick and spawns a task to remove that directory after the command terminates	2023-11-09 18:20:59 -06:00
asonix	20d07ea29b	Remove direct dependency on quick-xml, update dependencies (minor & point)	2023-11-04 13:18:22 -05:00
asonix	c795c1edfa	Replace most of actix-rt with tokio, give names to tasks	2023-10-20 19:08:11 -05:00
asonix	0bf8b6a2a8	Update rusty-s3 to 0.5.0	2023-10-08 11:29:59 -05:00
asonix	71fbc4c55b	Just have different mains	2023-10-07 12:31:05 -05:00
asonix	760b9d8c59	Install tracing & metrics from within runtime	2023-10-07 12:29:01 -05:00
asonix	3f5bc1d311	Build same pict-rs config for either runtime option	2023-10-07 12:22:17 -05:00
asonix	f97f76c01e	Fix io-uring initialization	2023-10-07 12:12:40 -05:00
asonix	4a45aa889c	Enable running on bare tokio	2023-10-07 11:48:58 -05:00
asonix	7f5cbc4571	Attempt clean-drop of tmp_dir	2023-10-07 11:36:49 -05:00
asonix	7b5a3020fa	Don't rely on global state for tmp_dir	2023-10-07 11:32:36 -05:00
asonix	fff4afe105	Start working on per-server temporary directory	2023-10-06 19:42:24 -05:00
asonix	c202484e9d	clippy	2023-10-04 12:46:38 -05:00
asonix	914e21c043	Implement constant-time equality for delete tokens, inline alias cleanup	2023-10-04 12:11:29 -05:00
asonix	d28c1ac628	Upgrade Concurrency: Update docs, configs, limit maxiumum details operations	2023-10-03 17:04:40 -05:00
asonix	f223d477c7	Enable specifying concurrency for 0.4 to 0.5 migration	2023-10-03 16:27:19 -05:00
asonix	75551dc3a1	Increase deadpool wait timeout, helps with extreme concurrency	2023-10-03 16:15:49 -05:00
asonix	247902e600	Enable specifying migrate-store concurrency	2023-10-03 15:50:53 -05:00
asonix	8568ae3364	Clippy	2023-09-30 22:47:49 -05:00
asonix	4c7067d4ca	Allow running without old_repo specified	2023-09-30 22:46:15 -05:00
asonix	81c6e73b5f	Simplify old_repo, remove client_pool_size	2023-09-30 22:46:15 -05:00
asonix	47751f3875	Downgrade some WARN logs to INFO, update docs for public methods	2023-09-30 17:52:58 -05:00
asonix	01cbe34ac3	Remove unneeded annotation	2023-09-30 17:34:05 -05:00
asonix	4bb9d81f51	Move internal middleware into own file	2023-09-30 17:33:01 -05:00
asonix	e3462f6664	payload middleware: switch to Rc, always inject if Payload isn't None	2023-09-30 17:24:48 -05:00
asonix	3267fb8301	Fail requests for malformed deadlines	2023-09-30 17:20:32 -05:00
asonix	65ea518542	Default log level to info	2023-09-30 16:28:04 -05:00
asonix	66e1711723	Enable proper draining of dropped request payloads ... Doing this as the outermost middleware ensures all endpoints are covered. Update request deadline to turn negative deadlines into immediate failures	2023-09-30 16:26:43 -05:00
asonix	6f44b05c0a	Remove references to Arbiter	2023-09-29 11:54:43 -05:00
asonix	735fcd1b49	Remove dependency on actix-web-codegen	2023-09-24 15:52:31 -05:00
asonix	ecdbaa9c92	Use diesel-async re-export of Hook	2023-09-24 15:48:19 -05:00
asonix	745e990fa0	Remove direct dependency on actix-rt and actix-server	2023-09-24 15:45:10 -05:00
asonix	0926739d3c	Remove once_cell	2023-09-24 15:32:00 -05:00
asonix	e0968dcce3	Remove serde-cbor, num_cpus	2023-09-24 15:22:57 -05:00
asonix	fd965bb1a5	Allow processing still images into animation formats	2023-09-24 13:20:33 -05:00
asonix	f3ef2d870b	Generate details during ingest	2023-09-24 12:58:16 -05:00
asonix	8f62b0c325	clippy	2023-09-24 12:42:23 -05:00
asonix	9ffdd0432f	Infer video thumbnail format from image format	2023-09-24 12:37:43 -05:00
asonix	6c921817e1	Enable thumbnailing animations, use dynamic dispatch for a number of async readers	2023-09-24 11:54:16 -05:00
asonix	cde4a72203	Add metrics to object storage body streams	2023-09-20 19:46:10 -05:00
asonix	49d3037358	Add metrics to object storage requests	2023-09-20 19:39:03 -05:00
asonix	7df6f3906e	Add timeout, metrics to external validation request	2023-09-20 19:29:19 -05:00
asonix	d0646dee25	Add config option for external validation timeout	2023-09-20 19:23:55 -05:00
asonix	4a5b9712aa	Combine map_ok and map_err underlying impl	2023-09-10 23:51:07 -04:00
asonix	2b10c48619	Update io-uring to use streem for file bytes	2023-09-10 23:45:24 -04:00
asonix	ea75ca24b5	Add track_caller to sync methods, update streem	2023-09-10 20:43:51 -04:00
asonix	b2674f06d0	More streme	2023-09-10 20:08:01 -04:00
asonix	1b97ac1c5a	Start replacing manual stream implementations with streem	2023-09-10 18:55:13 -04:00
asonix	858899b943	Add metrics & tracing to dropped uploads, turn pop into a single query	2023-09-07 19:20:41 -05:00
asonix	783df60f81	Fix on_conflict for postgres store migration path	2023-09-06 20:49:00 -05:00
asonix	a761daca34	clippy	2023-09-05 20:55:11 -05:00
asonix	e15a82c0c7	Add external validation check	2023-09-05 20:45:07 -05:00
asonix	bc49f8ca37	Migrate with timestamp	2023-09-04 22:25:11 -05:00
asonix	9c50bbe23a	match -> matches!	2023-09-04 22:06:28 -05:00
asonix	492b99922b	10 second worker delay after disconnect error	2023-09-04 22:04:15 -05:00
asonix	62ebc1c60a	Move timeout after metrics - timeouts count as incomplete	2023-09-04 22:02:59 -05:00
asonix	3bd0f78e75	Use with_timeout for all timeouts	2023-09-04 21:58:57 -05:00
asonix	a4b1ab7dfb	Instrument postgres db calls	2023-09-04 21:51:27 -05:00
asonix	37448722de	postgres: Rework job & upload notifications (more) ... postgres: Add metrics to job push & pop, upload wait sled: add upload wait metrics	2023-09-04 16:20:26 -05:00
asonix	00aa00c55c	postgres: Enable 'previous page' when less than full limit fit on previous page	2023-09-03 21:58:31 -05:00
asonix	a43de122f9	postgres: add already-claimed case, general: tracing paranoia	2023-09-03 21:30:47 -05:00
asonix	31caea438e	Fix slow connection pool access	2023-09-03 20:05:29 -05:00
asonix	8a892ba622	it might work	2023-09-03 18:21:46 -05:00
asonix	a2933dbebc	Implement all the todos	2023-09-03 17:11:34 -05:00
asonix	94cb2a9ef3	Pass clippy	2023-09-03 16:59:41 -05:00
asonix	443d327edf	Implement a couple more repo traits	2023-09-03 12:47:06 -05:00
asonix	eac4cd54a4	Initial work for pg notifications	2023-09-02 20:13:32 -05:00
asonix	e580e7701e	Finish implementing HashRepo	2023-09-02 19:13:00 -05:00
asonix	8921f57a21	Remove Identifier trait, Replace with Arc<str>	2023-09-02 18:38:02 -05:00
asonix	8eb4cda256	Implement first query	2023-09-02 13:52:15 -05:00
asonix	d475723087	Finish writing migrations & generate diesel schema	2023-09-02 13:35:30 -05:00
asonix	8c532c97e6	Initial postgres work	2023-09-02 11:52:55 -05:00
asonix	ba3a23ed43	Add error codes	2023-09-01 20:51:01 -05:00
asonix	fd74161c61	Rename some repo methods, generate UploadId in repo	2023-09-01 18:41:04 -05:00
asonix	cbb66f1b75	Add internal endpoint for deleting alias without a delete token	2023-09-01 17:25:13 -05:00
asonix	8fb90a6f69	Give a meaningful distinction for format in details	2023-08-31 16:26:45 -05:00
asonix	dce0827099	Increase default animation bounds, remove references to converting animations to videos	2023-08-30 21:55:53 -05:00
asonix	0b70ca49e4	clippy	2023-08-30 21:00:15 -05:00
asonix	721d9e1706	Don't try parsing an internal format for OldDetails	2023-08-30 20:55:38 -05:00
asonix	92ee60549f	Enable copying input av1 stream from mp4 to webm	2023-08-30 20:40:08 -05:00
asonix	b48a9233b2	Remove transcode from animation to video, make video transcoding 'optional' ... Video transcoding still happens, but in many cases the video stream is able to be copied verbatim rather than being decoded & encoded	2023-08-30 20:37:54 -05:00
asonix	08fd96c2f7	clippy nit	2023-08-29 13:29:40 -05:00
asonix	c0f6b1e988	Enable searching hashes by date	2023-08-29 13:27:18 -05:00
asonix	0146202236	Fully commit to HumanDate	2023-08-29 12:59:36 -05:00
asonix	64950bfe0e	Shorten pagination slugs	2023-08-29 12:53:29 -05:00
asonix	1271ff2cc7	Clippy	2023-08-28 18:49:30 -05:00
asonix	2b6ea97773	Enable setting timestamp on hash creation, improve paging api	2023-08-28 18:43:24 -05:00
asonix	24812afeba	Add paged hash access	2023-08-28 16:02:11 -05:00
asonix	e302ab1f3d	Migrate not-found alias	2023-08-28 13:40:19 -05:00
asonix	b28cc38c38	Simplify Streamer	2023-08-23 19:12:59 -05:00
asonix	95637fdfe5	Remove direct dep on futures-util	2023-08-23 19:10:10 -05:00
asonix	8f50a15b25	Reduce reliance on futures-util	2023-08-23 11:59:42 -05:00
asonix	0352ada9d8	Don't doulbe-open repo	2023-08-16 16:39:41 -05:00
asonix	88d4cc1a6a	Exit after repo migration	2023-08-16 16:34:42 -05:00
asonix	f95e61e402	Expose repo migrations	2023-08-16 16:32:19 -05:00
asonix	ba27a1a223	Build out repo-repo migration	2023-08-16 16:09:40 -05:00
asonix	1559d57f0a	Don't overwrite existing variants	2023-08-16 15:12:16 -05:00
asonix	22cfbe979d	Note how many failures have occurred for a given hash	2023-08-16 13:29:22 -05:00
asonix	ee88823d41	clippy	2023-08-16 12:43:12 -05:00
asonix	521cdd5b9d	Enforce format on new details	2023-08-16 12:36:18 -05:00
asonix	f3c6239979	Allow skipping details generation if already generated	2023-08-16 11:57:09 -05:00
asonix	4d909ba32a	Make migration concurrent	2023-08-16 11:47:36 -05:00
asonix	ac996ff832	Remove unused error	2023-08-15 21:25:04 -05:00
asonix	1de257bb07	Ensure access values are unique	2023-08-15 21:18:25 -05:00
asonix	bd3b668d81	Don't error on duplicate - happens on retried migration	2023-08-15 20:28:42 -05:00
asonix	9fe7410dd6	Add logging to migration	2023-08-15 19:33:19 -05:00
asonix	1ee77b19d8	Make Repo object safe. Create migration	2023-08-15 19:19:03 -05:00
asonix	09236d731d	Fix poll after completion, misused compare_and_swap	2023-08-14 21:17:57 -05:00
asonix	26ca3a7195	Remove associated Bytes type	2023-08-14 20:00:00 -05:00
asonix	da876fd553	Make it compile	2023-08-14 14:25:19 -05:00
asonix	3129f7844e	BROKEN: start work on hash discriminant	2023-08-13 22:06:42 -05:00
asonix	e4e93cddf9	Return JobId from push	2023-08-13 19:52:27 -05:00
asonix	5d4486fdf4	Finish queue implementation update for sled	2023-08-13 19:47:20 -05:00
asonix	35d0c065e9	Start work on 0.5 db format - initial queuerepo changes	2023-08-13 14:12:38 -05:00
asonix	0c4d632dda	Consider timeouts a client error	2023-08-05 16:18:06 -05:00
asonix	fee4ed1e3e	Add process timeout	2023-08-05 12:41:06 -05:00
asonix	5f12be0c6a	Clippy	2023-07-26 22:53:41 -05:00
asonix	2a59a6492a	Bring back hash_alias_key	2023-07-25 20:30:22 -05:00
asonix	bd3975f455	Simplify some repo functions, remove 0.3 migration	2023-07-25 20:08:18 -05:00
asonix	f221887599	Clippy	2023-07-23 16:04:17 -05:00
asonix	aac1bb7bc4	Finish media proxy implementation	2023-07-23 15:45:52 -05:00
asonix	4cca7d0f86	Prepare for proxying media	2023-07-23 12:10:36 -05:00
asonix	75bf425c6e	Update alias query, process query to accept proxy url	2023-07-23 11:45:10 -05:00
asonix	e59483c12c	Add ProxyRepo	2023-07-23 10:23:46 -05:00
asonix	d0c619bdf7	Clippy	2023-07-22 21:14:24 -05:00
asonix	77cdeab57e	Improve metrics, add job metrics	2023-07-22 21:11:28 -05:00
asonix	7dc8ede8eb	Gate cleanup behind read-only config	2023-07-22 20:26:28 -05:00
asonix	13d6e10886	Reduce log level on variants access	2023-07-22 20:20:58 -05:00
asonix	1f3298363f	Attempt to mark unmarked variants as accessed on launch	2023-07-22 19:55:50 -05:00
asonix	fe1f7c869f	Enable cleanup of variants ... Still TODO: doing a first pass to mark variants as accessed on launch	2023-07-22 19:41:50 -05:00
asonix	47e13ec04e	Mark variants as accessed on fetch	2023-07-22 18:50:04 -05:00
asonix	b9e6d67d15	Add retention configuration ... This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets	2023-07-22 18:37:25 -05:00
asonix	13387dec43	Fix specs	2023-07-22 17:58:03 -05:00
asonix	b786406ad0	Change IdentifierAccess to VariantAccess	2023-07-22 17:57:52 -05:00
asonix	ce0df080f4	Add prometheus metrics	2023-07-22 16:47:59 -05:00
asonix	d661732fb4	Tidy imports	2023-07-22 12:47:32 -05:00
asonix	a390b21df1	Make running pict-rs fluent	2023-07-22 12:39:39 -05:00
asonix	3ca994ee31	Replace global config with passed-around config ... This will enable spinning up pict-rs multiple times in the same process with different configurations	2023-07-22 12:31:01 -05:00
asonix	e1262a5fda	Extract preprocess steps as urlencoded in config deserialization	2023-07-22 11:38:09 -05:00
asonix	ac48003f45	Extract ProcessMap out of static	2023-07-22 11:16:19 -05:00
asonix	8d35c2449d	Simplify the cancel-safe processor	2023-07-22 10:46:16 -05:00
asonix	c1127b4117	Set content-length in PutObject, CompleteMultipart	2023-07-21 19:45:22 -05:00
asonix	2f0a3618d8	Replace awc with reqwest	2023-07-21 16:58:31 -05:00
asonix	fc14ac2038	Clippy	2023-07-18 21:59:27 -05:00
asonix	3d93300ba9	Add access repos for alias & identifier	2023-07-18 21:56:13 -05:00
asonix	127dd6cdad	Make max file count configurable	2023-07-18 20:32:17 -05:00
asonix	eeac900d7e	Add quality value to process method, move -coalesce earlier in pipelines	2023-07-18 16:18:01 -05:00
asonix	558605381d	Fix dangling unprocessed uploads ... Adds error boundary around backgrounded ingest	2023-07-17 22:32:08 -05:00
asonix	496bcc5dc1	Only include -b:v 0 for VP9 ... also simplify building arguments	2023-07-17 18:13:50 -05:00
asonix	3aecef3a9e	Remove println	2023-07-17 17:47:48 -05:00
asonix	b1bbc6b159	Attempt adding quality settings to pict-rs	2023-07-17 17:45:26 -05:00
asonix	c4c920191f	Add read-only mode	2023-07-17 14:24:49 -05:00
asonix	dd1d509bb1	Enable configuring download client timeout	2023-07-17 13:44:31 -05:00
asonix	5e8ab7856d	Extract Status errors into command-specific errors	2023-07-17 13:30:08 -05:00
asonix	2961aae6e3	Port migration changes from 0.4.1	2023-07-16 22:07:42 -05:00
asonix	a751d92436	Port error changes to 0.5	2023-07-16 21:51:14 -05:00
asonix	9871ad2ee0	Clippy	2023-07-14 23:16:57 -05:00
asonix	4d78ce7689	Fix store delegation for public_url	2023-07-14 15:23:07 -05:00
asonix	786f583d98	Add public_url to Store for optional redirects to public object store urls ... Add details generation to file migration to set the content-type for uploads	2023-07-14 14:53:37 -05:00
asonix	4e58388b41	Clean alias before checking hash option	2023-07-13 19:58:31 -05:00
asonix	afeac8294a	Fix Alias cleanup ordering ... Update dev.toml	2023-07-13 19:57:09 -05:00
asonix	ec6c9aac1e	Better discovery validation, proper invoking of imagemagick	2023-07-13 19:21:57 -05:00
asonix	9b1dd1f8c3	Copy process from 0.4.x branch	2023-07-13 19:21:28 -05:00
asonix	2c22f2ee3a	It compiles and runs, but doesn't work	2023-07-13 17:42:21 -05:00
asonix	27451971a6	Fix tests, add animated_webp test	2023-07-13 14:34:40 -05:00
asonix	33b83f97f2	It compiles ... Need to reinstate media limits	2023-07-13 13:48:59 -05:00
asonix	ad1837f9dd	VERY BROKEN: start replacing parts of pict-rs	2023-07-12 22:12:21 -05:00
asonix	58d9765594	WIP: Rework format detection and conversions ... - Add a bunch of types & methods that aren't used yet. The idea is creating the output from the known input parameters is pure, and works for any uploaded media - Introduce a new kind of media "animation" which is neither an image nor a video. It is likely that animations can be processed properly at the /image/process.{ext} endpoint with a little massaging TODO: - Integrate this into exiftool, ffmpeg, magick commands - Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a hint) - Infallible conversion from Details to OutputFile - this might be impossible given that we don't currently store more than a mime type and a frame count to give any sort of indication of real type - Try not to break API	2023-07-11 23:11:25 -05:00
asonix	691bca286c	Make object storage timeouts configurable	2023-07-11 13:01:58 -05:00
asonix	f5c39f9be5	Typo, error formatting	2023-07-10 17:15:43 -05:00
asonix	4be2eb67d1	Increase object storage signature expiry to 15 seconds	2023-07-10 16:28:51 -05:00
asonix	9004ecaadf	Better classify process related errors	2023-07-10 15:29:41 -05:00
asonix	e10cb1401d	Bump version	2023-07-09 15:16:04 -05:00
asonix	7dadbcf3ed	Replace UnsupportedFormat error with more helpful errors	2023-07-09 15:07:49 -05:00
asonix	1694f49436	Use json for ffmpeg, imagemagick details parsing	2023-07-09 14:50:58 -05:00
asonix	686be1d93b	Turn export_path into flag for sled	2023-07-08 19:56:26 -05:00
asonix	7768d4e58e	Add database exports ... This should help admins running pict-rs on traditional filesystems to keep valid backups of pict-rs' sled repo	2023-07-08 17:36:01 -05:00
asonix	4680f9c3aa	Instrument repo and store build methods	2023-07-08 10:42:03 -05:00
asonix	d51c348a3f	Run health checks on old and new stores before migrating	2023-07-07 14:05:13 -05:00
asonix	3ccb8ecd8c	Validate alias represents real file before setting not found	2023-07-07 13:33:27 -05:00