304 commits

Author	SHA1	Message	Date
asonix	2c1a37fb62	Extract pict-rs as library	2022-12-23 12:56:15 -06:00
asonix	74cd3392fa	Borrow parent spans when creating children	2022-12-07 22:43:12 -06:00
asonix	3325b222a2	clippy	2022-11-26 11:18:34 -06:00
asonix	7037ee5791	Support auto-orienting photos	2022-10-15 11:13:24 -05:00
asonix	669b3fb86f	Fix webp metadata stripping, more tracing cleanup	2022-10-01 22:47:52 -05:00
asonix	718f09c43a	Clean tracing, simplify validation, rename InputFormat -> VideoFormat	2022-10-01 21:17:18 -05:00
asonix	e7cf21f862	Allow concurrency in small object uploads	2022-10-01 13:06:19 -05:00
asonix	fe2a499110	Fix sending upload parts (set size), optimize small file uploads	2022-10-01 13:00:07 -05:00
asonix	0307dc5a3b	Introduce BytesStream to simplify aggregating, forwarding bytes	2022-10-01 12:06:33 -05:00
asonix	c767804430	Default to vp9 for uploaded video	2022-10-01 10:17:32 -05:00
asonix	85c5ce955d	Clippy	2022-09-30 20:02:46 -05:00
asonix	d86d9a3228	Put proper file extension on videos	2022-09-30 20:00:14 -05:00
asonix	8eb2293808	Enable selecting video and audio codecs for uploaded media	2022-09-30 19:38:11 -05:00
asonix	f4542efcc1	Share ffprobe details code	2022-09-30 17:43:40 -05:00
asonix	3ceaaa391c	Update clap	2022-09-28 18:23:41 -05:00
asonix	656b0225af	Clippy	2022-09-27 23:23:06 -05:00
asonix	29cab025c6	Improve 0.3 migration code, repo traces	2022-09-27 23:19:52 -05:00
asonix	85f20bbe0e	Add new resize options, update README with current APIs	2022-09-26 18:14:25 -05:00
asonix	9db057fdc5	Add admin endpoint to fetch identifier from an alias	2022-09-25 20:59:57 -05:00
asonix	17dab63662	Clippy	2022-09-25 20:44:24 -05:00
asonix	80c83eb491	Add ffprobe for details inspection - vastly improve video detection speed	2022-09-25 20:39:09 -05:00
asonix	5449bb82f1	Better identify video uploads with ffprobe	2022-09-25 19:34:51 -05:00
asonix	21e3e63ac3	Instrument upload store methods	2022-09-25 18:46:50 -05:00
asonix	51d8621015	Defensively format errors in traces to avoid deadlocks	2022-09-25 18:46:26 -05:00
asonix	890478e794	Support audio in uploaded videos, allow webm uploads	2022-09-25 18:16:37 -05:00
asonix	c57a48db8a	Start introducing more video controls	2022-09-25 17:36:07 -05:00
asonix	c9a74a73ca	Reduce redundancy in process module	2022-09-25 17:35:52 -05:00
asonix	50d118a2a7	Implement image pre-processing on upload	2022-09-25 15:17:33 -05:00
asonix	04bc586a26	Fix hasher test	2022-09-25 09:09:05 -05:00
asonix	dc95417846	Improve concurrency for upload streams part 2	2022-09-25 08:27:11 -05:00
asonix	bf3c47e457	Improve concurrency for upload streams	2022-09-24 22:07:06 -05:00
asonix	a6adde874e	Implement s3 multipart uploads	2022-09-24 20:33:59 -05:00
asonix	25209e29c0	Make it compile	2022-09-24 17:18:53 -05:00
asonix	ff1771e016	More migration work for rusty-s3	2022-09-24 14:18:49 -05:00
asonix	ab7fd9aaf7	Start work on using rusty-s3 instead of rust-s3	2022-09-24 13:39:27 -05:00
asonix	c5b107eb4c	clippy	2022-09-11 10:04:37 -05:00
asonix	7f49af58fe	Update actix-form-data, impl Store & Repo for Data<T>	2022-09-10 19:29:22 -05:00
asonix	e0bc95f785	Remove unneeded reborrow	2022-08-15 15:00:12 -05:00
Aode (lion)	547934071a	Don't migrate from the old database if it doesn't exist	2022-08-14 11:49:20 -05:00
Aode (lion)	e0543b2c1c	Add HEAD endpoints for process, serve	2022-07-17 11:15:43 -05:00
asonix	cb9a594f24	Fix content type on already processed images	2022-06-21 11:27:22 -05:00
asonix	dc73bdbad4	Make already-claimed a 400	2022-06-05 23:24:58 -05:00
Aode (lion)	2db76ad8c5	Update dependencies	2022-04-29 15:54:54 -05:00
Aode (lion)	8ee26ccf4f	change some import paths	2022-04-27 20:13:49 -05:00
Aode (Lion)	4ac20546ce	Improve migration from 0.3 formats	2022-04-24 17:10:15 -05:00
Aode (Lion)	b3cbda1337	Add internal endpoint for cleaning variants	2022-04-11 16:56:39 -05:00
Aode (Lion)	e50ccfb48c	Update deps	2022-04-11 15:17:54 -05:00
Aode (lion)	05533f7e3a	Use rfc3339 dates for details responses	2022-04-08 13:36:06 -05:00
Aode (lion)	7436f15267	Dry download client logic	2022-04-08 13:16:22 -05:00
Aode (lion)	5f19b63166	Extract details processing	2022-04-08 13:03:00 -05:00
Aode (lion)	31c5a36c77	Better instrument drops, jobs. Properly disarm backgrounded downloads	2022-04-08 12:51:33 -05:00
Aode (lion)	55d5e43cd2	Consolidate endpoints for downloading media, cached or backgrounded	2022-04-08 12:05:14 -05:00
Aode (lion)	07c61cbe87	Add download backgrounded	2022-04-07 19:07:30 -05:00
Aode (lion)	8781bc8f28	Add follows-from relation for background tasks	2022-04-07 13:28:28 -05:00
Aode (lion)	e493e90dd4	Defensive tracing	2022-04-07 12:56:40 -05:00
Aode (lion)	c80d207a87	Improve tracing	2022-04-06 21:40:49 -05:00
Aode (Lion)	05e2cf5e08	Cache on Alias rather than Hash ... Make sled repo blocking tasks share parent span	2022-04-06 12:14:12 -05:00
Aode (Lion)	865f4ee09c	Caching works: ... - Insert new bucket for current access time before removing previous access time record. - Add instrument to more methods for more precise tracing	2022-04-05 21:47:35 -05:00
Aode (Lion)	6cdae7b318	Add 'cache' endpoint for ingesting ephemeral media ... By default, cached media should only stick around for 7 days, however The timeout is reset every time media is accessed, so only obscure cached media will be flushed from the cache. This '7 days' number is configurable through the commandline run options as --media-cache-duration and in the pict-rs.toml file as [media] cache_duration	2022-04-05 20:29:30 -05:00
Aode (lion)	b0670c2f66	Move migrate to old	2022-04-03 15:07:31 -05:00
Aode (lion)	409b7d4c54	Remove unused conversions	2022-04-03 12:51:21 -05:00
Aode (lion)	29f0774331	Add ability to claim uploads	2022-04-02 21:15:39 -05:00
Aode (lion)	8734dfbdc7	Add backgrounding routes ... - Accept backgrounded uploads - Allow backgrounded processing Still TODO: - Endpoint for waiting on/claiming an upload	2022-04-02 20:56:29 -05:00
Aode (Lion)	c4d014597e	Better job recovery	2022-04-02 18:53:03 -05:00
Aode (Lion)	77a400c7ca	Implement UploadRepo	2022-04-02 17:41:00 -05:00
Aode (Lion)	132e395e5c	Multiple items: ... - Reduce duplicate work in generate job - Use hash instead of identifier for unique processing - Move motion ident generation behind concurrent processor lock	2022-04-02 17:40:08 -05:00
Aode (Lion)	09f53b9ce6	Move away from UploadManager to direct repo & store actions	2022-04-02 16:44:03 -05:00
Aode (lion)	6ed592c432	In Progress: process jobs ... - Is this a good idea? it definitely will make interacting with pict-rs harder. Maybe it's best not to do this	2022-04-01 16:51:12 -05:00
Aode (lion)	c0d8e0e8e3	Support multiple queues for job processor	2022-04-01 11:51:46 -05:00
Aode (lion)	09281d9ae8	Move stream timeout to all response bodies	2022-03-29 16:48:26 -05:00
Aode (lion)	63d66050c8	Enable buffering iterator items for iterstream	2022-03-29 16:18:00 -05:00
Aode (lion)	5adb3fde89	Move some Stream implementations into 'stream' module	2022-03-29 15:59:17 -05:00
Aode (lion)	0e490ff54a	Remove custom BytesFreezer stream	2022-03-29 14:28:08 -05:00
Aode (lion)	25f4480809	Remove custom stream error mapping types	2022-03-29 14:17:20 -05:00
Aode (lion)	8226a3571d	Enable multi-threaded job processing	2022-03-29 13:18:47 -05:00
Aode (lion)	602d1ea935	Begin implementing queue, put cleanups in it	2022-03-29 12:51:16 -05:00
Aode (lion)	e25a4781a8	Remove unneeded variable	2022-03-29 11:09:23 -05:00
Aode (lion)	99d042f36e	Allow disabling gif/mp4	2022-03-29 11:04:56 -05:00
Aode (Lion)	eb5e39c634	Use color-eyre	2022-03-28 20:47:46 -05:00
Aode (lion)	1291bf8beb	Divorce reqwest from main application's spans	2022-03-28 18:42:22 -05:00
Aode (lion)	1662f153ce	Numerous changes: ... - Improve error printing (display chain in addition to spantrace) - Fix migration (read main identifier from identifier tree, not filename tree) - Ensure uniqueness for processed images in ConcurrentProcessor (use source identifier in addition to thumbnail path, include extension in thumbnail path) - Update default log levels (make pict-rs quieter) - Add timeout for serving images from object storage (5 seconds)	2022-03-28 15:34:36 -05:00
Aode (Lion)	3792a8923a	Finish redoing config probably	2022-03-27 23:27:07 -05:00
Aode (lion)	ca28f68ef5	Start redoing config again	2022-03-27 19:10:06 -05:00
Aode (lion)	37e6b21b55	Fix Range, consolidate errors, test object storage	2022-03-26 20:45:12 -05:00
Aode (lion)	15b52ba6ec	Get it compiling again	2022-03-26 16:49:23 -05:00
Aode (lion)	323016f994	Build out migration path	2022-03-25 18:47:50 -05:00
Aode (lion)	750ce4782e	Rework configuration	2022-03-24 22:06:29 -05:00
Aode (lion)	d6567fbbbd	Implement sled repo	2022-03-24 17:09:15 -05:00
Aode (lion)	a0c99d05eb	Create initial trait representation for database	2022-03-24 13:16:41 -05:00
Aode (Lion)	2ad536db17	Promote console from compile flag to runtime flag	2022-03-21 22:41:51 -05:00
Aode (Lion)	f13f870a92	Only use console config when console feature is enabled	2022-03-21 22:05:32 -05:00
Aode (Lion)	62615be371	Allow configuring console buffer capacity	2022-03-21 21:56:19 -05:00
Aode (Lion)	d30415bb49	Don't debug PathStore	2022-03-21 21:43:38 -05:00
Aode (Lion)	9a8dc0e44f	Fix sled cache capacity help text	2022-03-21 21:18:51 -05:00
Aode (Lion)	da159c69cb	Try setting a lower default cache capacity	2022-03-21 21:15:43 -05:00
Aode (lion)	21b6e1dc08	Use actix-web's Range header	2022-03-01 11:23:15 -06:00
Aode (Lion)	73137ede21	Release candidate 1	2022-02-26 12:22:30 -06:00
Aode (lion)	0e948a2bf5	Add flag for console	2022-02-01 10:10:52 -06:00
Aode (Lion)	445c99dbf1	Re-use existing BytesMut when streaming file	2022-01-13 18:11:46 -06:00
Aode (lion)	f79d38df8d	Update rust-runner image, improve tokio-console support	2021-12-19 12:59:51 -06:00