445 commits

Author	SHA1	Message	Date
asonix	0146202236	Fully commit to HumanDate	2023-08-29 12:59:36 -05:00
asonix	64950bfe0e	Shorten pagination slugs	2023-08-29 12:53:29 -05:00
asonix	1271ff2cc7	Clippy	2023-08-28 18:49:30 -05:00
asonix	2b6ea97773	Enable setting timestamp on hash creation, improve paging api	2023-08-28 18:43:24 -05:00
asonix	24812afeba	Add paged hash access	2023-08-28 16:02:11 -05:00
asonix	e302ab1f3d	Migrate not-found alias	2023-08-28 13:40:19 -05:00
asonix	b28cc38c38	Simplify Streamer	2023-08-23 19:12:59 -05:00
asonix	95637fdfe5	Remove direct dep on futures-util	2023-08-23 19:10:10 -05:00
asonix	8f50a15b25	Reduce reliance on futures-util	2023-08-23 11:59:42 -05:00
asonix	0352ada9d8	Don't doulbe-open repo	2023-08-16 16:39:41 -05:00
asonix	88d4cc1a6a	Exit after repo migration	2023-08-16 16:34:42 -05:00
asonix	f95e61e402	Expose repo migrations	2023-08-16 16:32:19 -05:00
asonix	ba27a1a223	Build out repo-repo migration	2023-08-16 16:09:40 -05:00
asonix	1559d57f0a	Don't overwrite existing variants	2023-08-16 15:12:16 -05:00
asonix	22cfbe979d	Note how many failures have occurred for a given hash	2023-08-16 13:29:22 -05:00
asonix	ee88823d41	clippy	2023-08-16 12:43:12 -05:00
asonix	521cdd5b9d	Enforce format on new details	2023-08-16 12:36:18 -05:00
asonix	f3c6239979	Allow skipping details generation if already generated	2023-08-16 11:57:09 -05:00
asonix	4d909ba32a	Make migration concurrent	2023-08-16 11:47:36 -05:00
asonix	ac996ff832	Remove unused error	2023-08-15 21:25:04 -05:00
asonix	1de257bb07	Ensure access values are unique	2023-08-15 21:18:25 -05:00
asonix	bd3b668d81	Don't error on duplicate - happens on retried migration	2023-08-15 20:28:42 -05:00
asonix	9fe7410dd6	Add logging to migration	2023-08-15 19:33:19 -05:00
asonix	1ee77b19d8	Make Repo object safe. Create migration	2023-08-15 19:19:03 -05:00
asonix	09236d731d	Fix poll after completion, misused compare_and_swap	2023-08-14 21:17:57 -05:00
asonix	26ca3a7195	Remove associated Bytes type	2023-08-14 20:00:00 -05:00
asonix	da876fd553	Make it compile	2023-08-14 14:25:19 -05:00
asonix	3129f7844e	BROKEN: start work on hash discriminant	2023-08-13 22:06:42 -05:00
asonix	e4e93cddf9	Return JobId from push	2023-08-13 19:52:27 -05:00
asonix	5d4486fdf4	Finish queue implementation update for sled	2023-08-13 19:47:20 -05:00
asonix	35d0c065e9	Start work on 0.5 db format - initial queuerepo changes	2023-08-13 14:12:38 -05:00
asonix	0c4d632dda	Consider timeouts a client error	2023-08-05 16:18:06 -05:00
asonix	fee4ed1e3e	Add process timeout	2023-08-05 12:41:06 -05:00
asonix	5f12be0c6a	Clippy	2023-07-26 22:53:41 -05:00
asonix	2a59a6492a	Bring back hash_alias_key	2023-07-25 20:30:22 -05:00
asonix	bd3975f455	Simplify some repo functions, remove 0.3 migration	2023-07-25 20:08:18 -05:00
asonix	f221887599	Clippy	2023-07-23 16:04:17 -05:00
asonix	aac1bb7bc4	Finish media proxy implementation	2023-07-23 15:45:52 -05:00
asonix	4cca7d0f86	Prepare for proxying media	2023-07-23 12:10:36 -05:00
asonix	75bf425c6e	Update alias query, process query to accept proxy url	2023-07-23 11:45:10 -05:00
asonix	e59483c12c	Add ProxyRepo	2023-07-23 10:23:46 -05:00
asonix	d0c619bdf7	Clippy	2023-07-22 21:14:24 -05:00
asonix	77cdeab57e	Improve metrics, add job metrics	2023-07-22 21:11:28 -05:00
asonix	7dc8ede8eb	Gate cleanup behind read-only config	2023-07-22 20:26:28 -05:00
asonix	13d6e10886	Reduce log level on variants access	2023-07-22 20:20:58 -05:00
asonix	1f3298363f	Attempt to mark unmarked variants as accessed on launch	2023-07-22 19:55:50 -05:00
asonix	fe1f7c869f	Enable cleanup of variants ... Still TODO: doing a first pass to mark variants as accessed on launch	2023-07-22 19:41:50 -05:00
asonix	47e13ec04e	Mark variants as accessed on fetch	2023-07-22 18:50:04 -05:00
asonix	b9e6d67d15	Add retention configuration ... This doesn't do anything yet, but is part of the Media Proxy and Variant Cleanup tickets	2023-07-22 18:37:25 -05:00
asonix	13387dec43	Fix specs	2023-07-22 17:58:03 -05:00
asonix	b786406ad0	Change IdentifierAccess to VariantAccess	2023-07-22 17:57:52 -05:00
asonix	ce0df080f4	Add prometheus metrics	2023-07-22 16:47:59 -05:00
asonix	d661732fb4	Tidy imports	2023-07-22 12:47:32 -05:00
asonix	a390b21df1	Make running pict-rs fluent	2023-07-22 12:39:39 -05:00
asonix	3ca994ee31	Replace global config with passed-around config ... This will enable spinning up pict-rs multiple times in the same process with different configurations	2023-07-22 12:31:01 -05:00
asonix	e1262a5fda	Extract preprocess steps as urlencoded in config deserialization	2023-07-22 11:38:09 -05:00
asonix	ac48003f45	Extract ProcessMap out of static	2023-07-22 11:16:19 -05:00
asonix	8d35c2449d	Simplify the cancel-safe processor	2023-07-22 10:46:16 -05:00
asonix	c1127b4117	Set content-length in PutObject, CompleteMultipart	2023-07-21 19:45:22 -05:00
asonix	2f0a3618d8	Replace awc with reqwest	2023-07-21 16:58:31 -05:00
asonix	fc14ac2038	Clippy	2023-07-18 21:59:27 -05:00
asonix	3d93300ba9	Add access repos for alias & identifier	2023-07-18 21:56:13 -05:00
asonix	127dd6cdad	Make max file count configurable	2023-07-18 20:32:17 -05:00
asonix	eeac900d7e	Add quality value to process method, move -coalesce earlier in pipelines	2023-07-18 16:18:01 -05:00
asonix	558605381d	Fix dangling unprocessed uploads ... Adds error boundary around backgrounded ingest	2023-07-17 22:32:08 -05:00
asonix	496bcc5dc1	Only include -b:v 0 for VP9 ... also simplify building arguments	2023-07-17 18:13:50 -05:00
asonix	3aecef3a9e	Remove println	2023-07-17 17:47:48 -05:00
asonix	b1bbc6b159	Attempt adding quality settings to pict-rs	2023-07-17 17:45:26 -05:00
asonix	c4c920191f	Add read-only mode	2023-07-17 14:24:49 -05:00
asonix	dd1d509bb1	Enable configuring download client timeout	2023-07-17 13:44:31 -05:00
asonix	5e8ab7856d	Extract Status errors into command-specific errors	2023-07-17 13:30:08 -05:00
asonix	2961aae6e3	Port migration changes from 0.4.1	2023-07-16 22:07:42 -05:00
asonix	a751d92436	Port error changes to 0.5	2023-07-16 21:51:14 -05:00
asonix	9871ad2ee0	Clippy	2023-07-14 23:16:57 -05:00
asonix	4d78ce7689	Fix store delegation for public_url	2023-07-14 15:23:07 -05:00
asonix	786f583d98	Add public_url to Store for optional redirects to public object store urls ... Add details generation to file migration to set the content-type for uploads	2023-07-14 14:53:37 -05:00
asonix	4e58388b41	Clean alias before checking hash option	2023-07-13 19:58:31 -05:00
asonix	afeac8294a	Fix Alias cleanup ordering ... Update dev.toml	2023-07-13 19:57:09 -05:00
asonix	ec6c9aac1e	Better discovery validation, proper invoking of imagemagick	2023-07-13 19:21:57 -05:00
asonix	9b1dd1f8c3	Copy process from 0.4.x branch	2023-07-13 19:21:28 -05:00
asonix	2c22f2ee3a	It compiles and runs, but doesn't work	2023-07-13 17:42:21 -05:00
asonix	27451971a6	Fix tests, add animated_webp test	2023-07-13 14:34:40 -05:00
asonix	33b83f97f2	It compiles ... Need to reinstate media limits	2023-07-13 13:48:59 -05:00
asonix	ad1837f9dd	VERY BROKEN: start replacing parts of pict-rs	2023-07-12 22:12:21 -05:00
asonix	58d9765594	WIP: Rework format detection and conversions ... - Add a bunch of types & methods that aren't used yet. The idea is creating the output from the known input parameters is pure, and works for any uploaded media - Introduce a new kind of media "animation" which is neither an image nor a video. It is likely that animations can be processed properly at the /image/process.{ext} endpoint with a little massaging TODO: - Integrate this into exiftool, ffmpeg, magick commands - Detect apng with ffmpeg (imagemagick treats it as a still png unless it's given a hint) - Infallible conversion from Details to OutputFile - this might be impossible given that we don't currently store more than a mime type and a frame count to give any sort of indication of real type - Try not to break API	2023-07-11 23:11:25 -05:00
asonix	691bca286c	Make object storage timeouts configurable	2023-07-11 13:01:58 -05:00
asonix	f5c39f9be5	Typo, error formatting	2023-07-10 17:15:43 -05:00
asonix	4be2eb67d1	Increase object storage signature expiry to 15 seconds	2023-07-10 16:28:51 -05:00
asonix	9004ecaadf	Better classify process related errors	2023-07-10 15:29:41 -05:00
asonix	e10cb1401d	Bump version	2023-07-09 15:16:04 -05:00
asonix	7dadbcf3ed	Replace UnsupportedFormat error with more helpful errors	2023-07-09 15:07:49 -05:00
asonix	1694f49436	Use json for ffmpeg, imagemagick details parsing	2023-07-09 14:50:58 -05:00
asonix	686be1d93b	Turn export_path into flag for sled	2023-07-08 19:56:26 -05:00
asonix	7768d4e58e	Add database exports ... This should help admins running pict-rs on traditional filesystems to keep valid backups of pict-rs' sled repo	2023-07-08 17:36:01 -05:00
asonix	4680f9c3aa	Instrument repo and store build methods	2023-07-08 10:42:03 -05:00
asonix	d51c348a3f	Run health checks on old and new stores before migrating	2023-07-07 14:05:13 -05:00
asonix	3ccb8ecd8c	Validate alias represents real file before setting not found	2023-07-07 13:33:27 -05:00
asonix	9e7376d411	Remove is_missing error, make Identifier and DeleteToken return Option on fetch	2023-07-07 13:17:26 -05:00
asonix	7274538b70	Try adding store healthcheck	2023-07-07 12:05:42 -05:00
asonix	c150903980	Expand store differentiation	2023-07-07 11:19:39 -05:00