648 commits

Author	SHA1	Message	Date
asonix	cf7c753e65	Pass a State value around rather than a bunch of arguments	2024-02-03 13:31:54 -06:00
asonix	c3e322f2c9	typo	2024-02-02 14:00:54 -06:00
asonix	3a6faef5ef	Add logs for pict-rs binds	2024-02-01 21:55:31 -06:00
asonix	5805eb0aed	Enable configuring imagemagick security policy from pictrs	2024-01-31 21:32:01 -06:00
asonix	50fd3e6182	Enable serving over TLS	2024-01-31 17:47:42 -06:00
asonix	08472eabeb	Log console address, prometheus address with tracing	2024-01-30 15:21:15 -06:00
asonix	e7357cbdaf	Count spawned tasks in cleanup, debug log queue length	2024-01-30 14:57:48 -06:00
asonix	34cadb86b8	Don't propogate pop span into blocking closure	2024-01-30 14:30:24 -06:00
asonix	ee00fafee4	Yield more often in queue, simplify traces, log job_id in higher span	2024-01-30 14:18:07 -06:00
asonix	dc12f6f79d	replace remaining web::block with spawn_blocking	2024-01-30 14:16:05 -06:00
asonix	7282ee9312	Give background jobs more opportunity to yield to runtime	2024-01-25 16:59:46 -06:00
asonix	819b83bab7	Use a far smaller pool to listen for db notifications	2024-01-25 16:50:36 -06:00
asonix	c65b49339f	Take at most 2048 variants and proxies during each cleanup job	2024-01-25 16:26:29 -06:00
asonix	574b12b23f	Fix postgres commandline, add default for use_tls	2024-01-25 08:55:51 -06:00
asonix	2f588aa613	Ignore tokio-uring test in docs	2024-01-24 17:41:42 -06:00
asonix	fe1132aec1	Enable at-most-once queueing for some cleanup jobs	2024-01-24 17:14:31 -06:00
asonix	fee2294901	clippy	2024-01-15 18:21:36 -05:00
asonix	19147e2035	postgres: allow connecting to TLS-enabled databases	2024-01-15 18:11:08 -05:00
asonix	f6087d65be	Use async fn in trait for Store	2024-01-15 15:51:20 -05:00
asonix	9ed90efed4	Add release document, use .extend rather than .join	2024-01-11 16:48:18 -06:00
asonix	708c25f20a	Join identifier onto public_endpoint, rather than overwriting the existing path	2024-01-11 16:06:44 -06:00
asonix	9ceb26bb5c	Clippy	2024-01-07 10:09:06 -06:00
asonix	65e165af49	Set default audio codecs if none are specified	2024-01-07 10:03:40 -06:00
asonix	ca34a15cd3	Debug-log command arguments	2024-01-07 10:01:02 -06:00
asonix	004a54c4ae	Improve cooperation from background jobs	2024-01-06 17:47:44 -06:00
asonix	bcf73eb4e4	Improve cooperation from from_iterator to prevent task starvation	2024-01-05 19:50:10 -06:00
asonix	4145637a33	Add a few more metrics, increase repo disconnect sleep	2024-01-05 19:35:52 -06:00
asonix	688c797082	Remove LIMIT 10, single_value is already in use	2024-01-04 11:13:29 -06:00
asonix	6514487ea9	Index on queue_time for job queue, limit to 10 jobs in ID query	2024-01-04 11:01:25 -06:00
asonix	0fd6c459d0	Clippy	2024-01-04 10:42:34 -06:00
asonix	7d4f44a540	Set worker-id to None when marking a job as runnable after expired heartbeats	2024-01-04 10:29:37 -06:00
asonix	7e215c691b	Support RUST_LOG	2023-12-28 12:40:59 -06:00
asonix	bfc2410552	Add trace-level logs to every loop (paranoid)	2023-12-28 11:58:38 -06:00
asonix	dc90e00324	Update metrics to 0.22 (plus mechanical changes)	2023-12-26 18:06:38 -06:00
asonix	bd3cdc925e	Remove special-case for animated avif	2023-12-23 21:44:11 -06:00
asonix	6d259a0eea	Don't kill already-exited processes, don't log tmp_dir when fetching details	2023-12-23 13:29:30 -06:00
asonix	c2e53766c2	Don't -ping when extracting file details (workaround for imagemagick json bug)	2023-12-23 12:48:32 -06:00
asonix	df8fc00ad5	Try to clean up properly even if command errors	2023-12-23 12:11:15 -06:00
asonix	89a4e3995f	Add warn logs when performing blocking removes due to dropped files	2023-12-23 12:10:25 -06:00
asonix	6fa79b9188	Do more cleanup inline	2023-12-23 11:58:20 -06:00
asonix	e8380c31c1	Fix remaining warnings	2023-12-22 21:03:51 -06:00
asonix	c2b1b3db5d	Apply cargo fix	2023-12-22 21:01:19 -06:00
asonix	c544f167fb	Remove unused code	2023-12-22 21:00:37 -06:00
asonix	b938b09901	Apply cargo fix	2023-12-22 20:54:02 -06:00
asonix	5624671cbf	Rework ProcessRead to require a closure for using the internal AsyncRead	2023-12-22 20:52:58 -06:00
asonix	df88660b9e	Lower priority of some object storage spans	2023-12-22 13:58:05 -06:00
asonix	4369283534	More tracing cleaning	2023-12-22 13:30:17 -06:00
asonix	aa8fe7f3e8	Clean unused imports	2023-12-22 13:20:36 -06:00
asonix	0dbe722524	Don't poll handle if closed, prevents panic if child wait finishes before all bytes are read	2023-12-22 13:20:21 -06:00
asonix	b94ba5fcfc	Inline process background future, clean tracing a bit	2023-12-22 13:12:19 -06:00
asonix	db43392a3b	Simplify process future by removing explicit channel, don't lock mutex as often	2023-12-22 12:03:05 -06:00
asonix	c9155f7ce7	Include not_found key in repo migration, rework ProcessRead, add timeout to 0.4 migration	2023-12-17 23:15:27 -06:00
asonix	bd0b89b165	Clean spans a bit	2023-12-15 22:34:45 -06:00
asonix	2544cb478f	Add static bound to appease 1.74.1	2023-12-12 17:17:16 -06:00
asonix	f1c5a56353	Port prune_missing from 0.4.x	2023-12-12 16:54:41 -06:00
asonix	3ef4097ceb	Enable building without tokio_unstable	2023-12-12 16:23:02 -06:00
asonix	6daa745446	Add optional ability to log spans	2023-12-11 14:58:53 -06:00
asonix	cd80ab16c3	Downgrade retry warnings to debug	2023-12-11 13:08:10 -06:00
asonix	b452a577a7	Add object ID to not found error	2023-12-11 13:05:29 -06:00
asonix	7293628148	put timeout on collecting payloads	2023-12-10 19:26:16 -06:00
asonix	961de20a00	Add timeout and metrics to media processing	2023-12-10 19:11:36 -06:00
asonix	2523b6644b	Add AggregateTimeout, don't aggregate with permit	2023-12-04 15:39:34 -06:00
asonix	6b2ccbf975	Allow configuring pict-rs' temporary directory	2023-11-24 11:48:45 -06:00
asonix	a30044278c	Improve documentation for using pict-rs as a library	2023-11-24 11:38:14 -06:00
asonix	71f8f52b58	Tweak error reporting	2023-11-13 17:24:56 -06:00
asonix	1538389caf	Clippy	2023-11-11 14:28:05 -06:00
asonix	210af5d7d9	Add danger_dummy_mode	2023-11-11 14:22:12 -06:00
asonix	5fd59fc5b4	Change commandline flag to allow disabling video, since it is enabled by default	2023-11-11 13:22:12 -06:00
asonix	3a0546694f	Update opentelemetry dependencies	2023-11-10 15:19:54 -06:00
asonix	15189d293b	clippy	2023-11-09 18:26:57 -06:00
asonix	f61dac8187	Attempt to clean stray magick files ... This change sets a unique temp directory for each invocation of imagemagick and spawns a task to remove that directory after the command terminates	2023-11-09 18:20:59 -06:00
asonix	20d07ea29b	Remove direct dependency on quick-xml, update dependencies (minor & point)	2023-11-04 13:18:22 -05:00
asonix	c795c1edfa	Replace most of actix-rt with tokio, give names to tasks	2023-10-20 19:08:11 -05:00
asonix	0bf8b6a2a8	Update rusty-s3 to 0.5.0	2023-10-08 11:29:59 -05:00
asonix	71fbc4c55b	Just have different mains	2023-10-07 12:31:05 -05:00
asonix	760b9d8c59	Install tracing & metrics from within runtime	2023-10-07 12:29:01 -05:00
asonix	3f5bc1d311	Build same pict-rs config for either runtime option	2023-10-07 12:22:17 -05:00
asonix	f97f76c01e	Fix io-uring initialization	2023-10-07 12:12:40 -05:00
asonix	4a45aa889c	Enable running on bare tokio	2023-10-07 11:48:58 -05:00
asonix	7f5cbc4571	Attempt clean-drop of tmp_dir	2023-10-07 11:36:49 -05:00
asonix	7b5a3020fa	Don't rely on global state for tmp_dir	2023-10-07 11:32:36 -05:00
asonix	fff4afe105	Start working on per-server temporary directory	2023-10-06 19:42:24 -05:00
asonix	c202484e9d	clippy	2023-10-04 12:46:38 -05:00
asonix	914e21c043	Implement constant-time equality for delete tokens, inline alias cleanup	2023-10-04 12:11:29 -05:00
asonix	d28c1ac628	Upgrade Concurrency: Update docs, configs, limit maxiumum details operations	2023-10-03 17:04:40 -05:00
asonix	f223d477c7	Enable specifying concurrency for 0.4 to 0.5 migration	2023-10-03 16:27:19 -05:00
asonix	75551dc3a1	Increase deadpool wait timeout, helps with extreme concurrency	2023-10-03 16:15:49 -05:00
asonix	247902e600	Enable specifying migrate-store concurrency	2023-10-03 15:50:53 -05:00
asonix	8568ae3364	Clippy	2023-09-30 22:47:49 -05:00
asonix	4c7067d4ca	Allow running without old_repo specified	2023-09-30 22:46:15 -05:00
asonix	81c6e73b5f	Simplify old_repo, remove client_pool_size	2023-09-30 22:46:15 -05:00
asonix	47751f3875	Downgrade some WARN logs to INFO, update docs for public methods	2023-09-30 17:52:58 -05:00
asonix	01cbe34ac3	Remove unneeded annotation	2023-09-30 17:34:05 -05:00
asonix	4bb9d81f51	Move internal middleware into own file	2023-09-30 17:33:01 -05:00
asonix	e3462f6664	payload middleware: switch to Rc, always inject if Payload isn't None	2023-09-30 17:24:48 -05:00
asonix	3267fb8301	Fail requests for malformed deadlines	2023-09-30 17:20:32 -05:00
asonix	65ea518542	Default log level to info	2023-09-30 16:28:04 -05:00
asonix	66e1711723	Enable proper draining of dropped request payloads ... Doing this as the outermost middleware ensures all endpoints are covered. Update request deadline to turn negative deadlines into immediate failures	2023-09-30 16:26:43 -05:00
asonix	6f44b05c0a	Remove references to Arbiter	2023-09-29 11:54:43 -05:00
asonix	735fcd1b49	Remove dependency on actix-web-codegen	2023-09-24 15:52:31 -05:00
asonix	ecdbaa9c92	Use diesel-async re-export of Hook	2023-09-24 15:48:19 -05:00
asonix	745e990fa0	Remove direct dependency on actix-rt and actix-server	2023-09-24 15:45:10 -05:00
asonix	0926739d3c	Remove once_cell	2023-09-24 15:32:00 -05:00
asonix	e0968dcce3	Remove serde-cbor, num_cpus	2023-09-24 15:22:57 -05:00
asonix	fd965bb1a5	Allow processing still images into animation formats	2023-09-24 13:20:33 -05:00
asonix	f3ef2d870b	Generate details during ingest	2023-09-24 12:58:16 -05:00
asonix	8f62b0c325	clippy	2023-09-24 12:42:23 -05:00
asonix	9ffdd0432f	Infer video thumbnail format from image format	2023-09-24 12:37:43 -05:00
asonix	6c921817e1	Enable thumbnailing animations, use dynamic dispatch for a number of async readers	2023-09-24 11:54:16 -05:00
asonix	cde4a72203	Add metrics to object storage body streams	2023-09-20 19:46:10 -05:00
asonix	49d3037358	Add metrics to object storage requests	2023-09-20 19:39:03 -05:00
asonix	7df6f3906e	Add timeout, metrics to external validation request	2023-09-20 19:29:19 -05:00
asonix	d0646dee25	Add config option for external validation timeout	2023-09-20 19:23:55 -05:00
asonix	4a5b9712aa	Combine map_ok and map_err underlying impl	2023-09-10 23:51:07 -04:00
asonix	2b10c48619	Update io-uring to use streem for file bytes	2023-09-10 23:45:24 -04:00
asonix	ea75ca24b5	Add track_caller to sync methods, update streem	2023-09-10 20:43:51 -04:00
asonix	b2674f06d0	More streme	2023-09-10 20:08:01 -04:00
asonix	1b97ac1c5a	Start replacing manual stream implementations with streem	2023-09-10 18:55:13 -04:00
asonix	858899b943	Add metrics & tracing to dropped uploads, turn pop into a single query	2023-09-07 19:20:41 -05:00
asonix	783df60f81	Fix on_conflict for postgres store migration path	2023-09-06 20:49:00 -05:00
asonix	a761daca34	clippy	2023-09-05 20:55:11 -05:00
asonix	e15a82c0c7	Add external validation check	2023-09-05 20:45:07 -05:00
asonix	bc49f8ca37	Migrate with timestamp	2023-09-04 22:25:11 -05:00
asonix	9c50bbe23a	match -> matches!	2023-09-04 22:06:28 -05:00
asonix	492b99922b	10 second worker delay after disconnect error	2023-09-04 22:04:15 -05:00
asonix	62ebc1c60a	Move timeout after metrics - timeouts count as incomplete	2023-09-04 22:02:59 -05:00
asonix	3bd0f78e75	Use with_timeout for all timeouts	2023-09-04 21:58:57 -05:00
asonix	a4b1ab7dfb	Instrument postgres db calls	2023-09-04 21:51:27 -05:00
asonix	37448722de	postgres: Rework job & upload notifications (more) ... postgres: Add metrics to job push & pop, upload wait sled: add upload wait metrics	2023-09-04 16:20:26 -05:00
asonix	00aa00c55c	postgres: Enable 'previous page' when less than full limit fit on previous page	2023-09-03 21:58:31 -05:00
asonix	a43de122f9	postgres: add already-claimed case, general: tracing paranoia	2023-09-03 21:30:47 -05:00
asonix	31caea438e	Fix slow connection pool access	2023-09-03 20:05:29 -05:00
asonix	8a892ba622	it might work	2023-09-03 18:21:46 -05:00
asonix	a2933dbebc	Implement all the todos	2023-09-03 17:11:34 -05:00
asonix	94cb2a9ef3	Pass clippy	2023-09-03 16:59:41 -05:00
asonix	443d327edf	Implement a couple more repo traits	2023-09-03 12:47:06 -05:00
asonix	eac4cd54a4	Initial work for pg notifications	2023-09-02 20:13:32 -05:00
asonix	e580e7701e	Finish implementing HashRepo	2023-09-02 19:13:00 -05:00
asonix	8921f57a21	Remove Identifier trait, Replace with Arc<str>	2023-09-02 18:38:02 -05:00
asonix	8eb4cda256	Implement first query	2023-09-02 13:52:15 -05:00
asonix	d475723087	Finish writing migrations & generate diesel schema	2023-09-02 13:35:30 -05:00
asonix	8c532c97e6	Initial postgres work	2023-09-02 11:52:55 -05:00
asonix	ba3a23ed43	Add error codes	2023-09-01 20:51:01 -05:00
asonix	fd74161c61	Rename some repo methods, generate UploadId in repo	2023-09-01 18:41:04 -05:00
asonix	cbb66f1b75	Add internal endpoint for deleting alias without a delete token	2023-09-01 17:25:13 -05:00
asonix	8fb90a6f69	Give a meaningful distinction for format in details	2023-08-31 16:26:45 -05:00
asonix	dce0827099	Increase default animation bounds, remove references to converting animations to videos	2023-08-30 21:55:53 -05:00
asonix	0b70ca49e4	clippy	2023-08-30 21:00:15 -05:00
asonix	721d9e1706	Don't try parsing an internal format for OldDetails	2023-08-30 20:55:38 -05:00
asonix	92ee60549f	Enable copying input av1 stream from mp4 to webm	2023-08-30 20:40:08 -05:00