asonix/background-jobs
1
0
Fork 0
Code Issues Pull requests Projects Releases Wiki Activity

Prepare release

Browse source
This commit is contained in:
asonix 2018-12-16 13:44:25 -06:00
parent a22d10242a
commit a7b494ca39
No known key found for this signature in database
GPG key ID: 6986797E36BFA1D4
11 changed files with 99 additions and 29 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

2
Cargo.toml
View file

@ -1,7 +1,7 @@
[package]
name = "background-jobs"
description = "Background Jobs implemented with tokio and futures"
version = "0.3.1"
version = "0.4.0"
license = "GPL-3.0"
authors = ["asonix <asonix@asonix.dog>"]
repository = "https://git.asonix.dog/asonix/background-jobs"

47
README.md
View file

@ -9,7 +9,7 @@ might not be the best experience.
#### Add Background Jobs to your project
```toml
[dependencies]
background-jobs = "0.3"
background-jobs = "0.4"
failure = "0.1"
futures = "0.1"
tokio = "0.1"
@ -129,9 +129,9 @@ use server_jobs_example::queue_set;
fn main() -> Result<(), Error> {
// Run our job server
tokio::run(ServerConfig::init(
1,
"127.0.0.1",
5555,
1,
queue_set(),
"example-db",
));
@ -211,7 +211,7 @@ fn main() -> Result<(), Error> {
// Queue each job
for job in jobs {
spawner.queue_sync::<MyProcessor>(job)?
spawner.queue_sync::<MyProcessor, _>(job)?
}
}
```
@ -226,9 +226,9 @@ but it's untested and hard to say. You can override this behavior by specifying
Cargo.toml
```toml
[Dependencies.background-jobs]
version = "0.1"
version = "0.4"
default-features = false
features = ["background-jobs-server", "background-jobs-server/futures-zmq"]
features = ["no_unix"]
```
[`futures-zmq`](https://crates.io/crates/futures-zmq) Is designed to be a drop-in replacement for
@ -236,7 +236,42 @@ tokio-zmq that works on non-unix and non-tokio platforms. The reason why it isn'
that it's slower than tokio-zmq, and in all likelihood, the production environment for projects
depending on this one will be linux.
#### Not using a ZeroMQ+LMDB based client/server model
#### Actix
Another implementation of a jobs processor is also provided by this library under a feature flag.
```toml
[dependencies.background-jobs]
version = "0.4"
default-features = false
features = ["actix"]
```
This provides an in-process implementation of a jobs server and worker setup. Here's some example usage.
```rust
use background_jobs::{Processor, ServerConfig, WorkerConfig};
let sys = actix::System::new("my-actix-thing");
let queue_handle = ServerConfig::new(1, db_path.into()).start::<MyState>();
let state = MyState {
queue_handle: queue_handle.clone(),
};
let mut worker_config = WorkerConfig::new(state);
WorkerConfig::register(&mut worker_config, FetchProcessor);
WorkerConfig::register(&mut worker_config, InstanceProcessor);
WorkerConfig::register(&mut worker_config, OpenProcessor);
WorkerConfig::set_processor_count(
&mut worker_config,
<InstanceProcessor as Processor<MyState>>::QUEUE,
16,
);
WorkerConfig::start(worker_config, queue_handle.clone());
let _ = sys.run();
```
#### Bringing your own server/worker implementation
If you want to create your own jobs processor based on this idea, you can depend on the
`background-jobs-core` crate, which provides the LMDB storage, Processor and Job traits, as well as some
other useful types for implementing a jobs processor.

2
examples/server-jobs-example/Cargo.toml
View file

@ -15,5 +15,5 @@ serde_derive = "1.0"
tokio = "0.1"
[dependencies.background-jobs]
version = "0.3"
version = "0.4"
path = "../.."

2
jobs-core/src/job_info.rs
View file

@ -88,7 +88,7 @@ impl NewJobInfo {
///
/// Although exposed publically, this type should only really be handled by the library itself, and
/// is impossible to create outside of a
/// [Processor](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Processor.html)'s
/// [Processor](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Processor.html)'s
/// new_job method.
pub struct JobInfo {
/// ID of the job

6
jobs-core/src/processor.rs
View file

@ -34,11 +34,11 @@ use crate::{Backoff, Job, JobError, MaxRetries, NewJobInfo};
/// - The job's default queue
/// - The job's default maximum number of retries
/// - The job's [backoff
/// strategy](https://docs.rs/background-jobs/0.3.0/background_jobs/enum.Backoff.html)
/// strategy](https://docs.rs/background-jobs/0.4.0/background_jobs/enum.Backoff.html)
///
/// Processors also provide the default mechanism for running a job, and the only mechanism for
/// creating a
/// [JobInfo](https://docs.rs/background-jobs-core/0.3.0/background_jobs_core/struct.JobInfo.html),
/// [JobInfo](https://docs.rs/background-jobs-core/0.4.0/background_jobs_core/struct.JobInfo.html),
/// which is the type required for queuing jobs to be executed.
///
/// ### Example
@ -170,7 +170,7 @@ where
/// Patterns like this could be useful if you want to use the same job type for multiple
/// scenarios. Defining the `process` method for multiple `Processor`s with different
/// before/after logic for the same
/// [`Job`](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Job.html) type is
/// [`Job`](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Job.html) type is
/// supported.
fn process(
&self,

8
jobs-core/src/processor_map.rs
View file

@ -28,16 +28,16 @@ use crate::{JobError, JobInfo, Processor};
/// A generic function that processes a job
///
/// Instead of storing
/// [`Processor`](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Processor.html) type
/// [`Processor`](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Processor.html) type
/// directly, the
/// [`ProcessorMap`](https://docs.rs/background-jobs-core/0.3.0/background_jobs_core/struct.ProcessorMap.html)
/// [`ProcessorMap`](https://docs.rs/background-jobs-core/0.4.0/background_jobs_core/struct.ProcessorMap.html)
/// struct stores these `ProcessFn` types that don't expose differences in Job types.
pub type ProcessFn =
Box<dyn Fn(Value) -> Box<dyn Future<Item = (), Error = JobError> + Send> + Send + Sync>;
/// A type for storing the relationships between processor names and the processor itself
///
/// [`Processor`s](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Processor.html) must
/// [`Processor`s](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Processor.html) must
/// be registered with the `ProcessorMap` in the initialization phase of an application before
/// workers are spawned in order to handle queued jobs.
pub struct ProcessorMap<S>
@ -65,7 +65,7 @@ where
}
/// Register a
/// [`Processor`](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Processor.html) with
/// [`Processor`](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Processor.html) with
/// this `ProcessorMap`.
///
/// `ProcessorMap`s are useless if no processors are registerd before workers are spawned, so

2
jobs-server/src/lib.rs
View file

@ -46,7 +46,7 @@ where
///
/// This trait should be implemented sparingly, but is provided so that synchronous tasks may be
/// executed. If you have the ability to implement the
/// [`Job`](https://docs.rs/background-jobs/0.3.0/background_jobs/trait.Job.html) trait directly,
/// [`Job`](https://docs.rs/background-jobs/0.4.0/background_jobs/trait.Job.html) trait directly,
/// you should.
///
/// ### Example

4
jobs-server/src/server/mod.rs
View file

@ -123,7 +123,7 @@ struct MissingQueue(String);
///
/// `ServerConfig` is used to spin up the infrastructure to manage queueing and storing jobs, but
/// it does not provide functionality to execute jobs. For that, you must create a
/// [`Worker`](https://docs.rs/background-jobs-server/0.3.0/background_jobs_server/struct.WorkerConfig.html)
/// [`Worker`](https://docs.rs/background-jobs-server/0.4.0/background_jobs_server/struct.WorkerConfig.html)
/// that will connect to the running server.
///
/// This type doesn't have any associated data, but is used as a proxy for starting the
@ -139,9 +139,9 @@ struct MissingQueue(String);
/// queue_set.insert("default".to_owned());
///
/// let start_server = ServerConfig::init(
/// 1,
/// "127.0.0.1",
/// 5555,
/// 1,
/// queue_set,
/// "example-db",
/// );

2
jobs-server/src/spawner.rs
View file

@ -38,7 +38,7 @@ use zmq::{Context, Message};
///
/// tokio::spawn(
/// spawner
/// .queue::<MyProcessor>(job)
/// .queue::<MyProcessor, _>(job)
/// .map_err(|_| ()),
/// );
/// ```

4
jobs-server/src/worker/mod.rs
View file

@ -34,7 +34,7 @@ use self::{config::Worker, portmap::PortMap};
///
/// A worker handles the processing of jobs, but not the queueing or storing of jobs. It connects
/// to a server (crated with
/// [`ServerConfig`](https://docs.rs/background-jobs-server/0.3.0/background_jobs_server/struct.ServerConfig.html))
/// [`ServerConfig`](https://docs.rs/background-jobs-server/0.4.0/background_jobs_server/struct.ServerConfig.html))
/// and receives work from there.
///
/// ```rust
@ -122,7 +122,7 @@ where
/// Register a processor with this worker
///
/// For more information, see
/// [`Processor`](https://docs.rs/background-jobs/0.3.0/background_jobs/enum.Processor.html).
/// [`Processor`](https://docs.rs/background-jobs/0.4.0/background_jobs/enum.Processor.html).
pub fn register_processor<P>(&mut self, processor: P)
where
P: Processor<S> + Send + Sync + 'static,

49
src/lib.rs
View file

@ -28,7 +28,7 @@
//! #### Add Background Jobs to your project
//! ```toml
//! [dependencies]
//! background-jobs = "0.3"
//! background-jobs = "0.4"
//! failure = "0.1"
//! futures = "0.1"
//! tokio = "0.1"
@ -204,7 +204,7 @@
//! // Queue each job
//! tokio::run(lazy(move || {
//! for job in jobs {
//! tokio::spawn(spawner.queue::<MyProcessor>(job).map_err(|_| ()));
//! tokio::spawn(spawner.queue::<MyProcessor, _>(job).map_err(|_| ()));
//! }
//!
//! Ok(())
@ -223,9 +223,9 @@
//! hard to say. You can override this behavior by specifying the following in your Cargo.toml
//! ```toml
//! [Dependencies.background-jobs]
//! version = "0.1"
//! version = "0.4"
//! default-features = false
//! features = ["background-jobs-server", "background-jobs-server/futures-zmq"]
//! features = ["no_unix"]
//! ```
//!
//! [`futures-zmq`](https://crates.io/crates/futures-zmq) Is designed to be a drop-in replacement
@ -233,10 +233,45 @@
//! by default is that it's slower than tokio-zmq, and in all likelihood, the production
//! environment for projects depending on this one will be linux.
//!
//! #### Not using a ZeroMQ+LMDB based client/server model
//! #### Actix
//! Another implementation of a jobs processor is also provided by this library under a feature flag.
//! ```toml
//! [dependencies.background-jobs]
//! version = "0.4"
//! default-features = false
//! features = ["actix"]
//! ```
//!
//! This provides an in-process implementation of a jobs server and worker setup. Here's some example usage.
//! ```rust,ignore
//! use background_jobs::{Processor, ServerConfig, WorkerConfig};
//!
//! let sys = actix::System::new("my-actix-thing");
//!
//! let queue_handle = ServerConfig::new(1, db_path.into()).start::<MyState>();
//!
//! let state = MyState {
//! queue_handle: queue_handle.clone(),
//! };
//!
//! let mut worker_config = WorkerConfig::new(state);
//! WorkerConfig::register(&mut worker_config, FetchProcessor);
//! WorkerConfig::register(&mut worker_config, InstanceProcessor);
//! WorkerConfig::register(&mut worker_config, OpenProcessor);
//! WorkerConfig::set_processor_count(
//! &mut worker_config,
//! <InstanceProcessor as Processor<MyState>>::QUEUE,
//! 16,
//! );
//! WorkerConfig::start(worker_config, queue_handle.clone());
//!
//! let _ = sys.run();
//! ```
//!
//! #### Bringing your own server/worker implementation
//! If you want to create your own jobs processor based on this idea, you can depend on the
//! `background-jobs-core` crate, which provides the LMDB storage, Processor and Job traits, as
//! well as some other useful types for implementing a jobs processor.
//! `background-jobs-core` crate, which provides the LMDB storage, Processor and Job traits, as well as some
//! other useful types for implementing a jobs processor.
pub use background_jobs_core::{Backoff, Job, MaxRetries, Processor};