Package detail

orkid

mugli193MIT0.11.0

Reliable and modern Redis-Streams based task queue for Node.js

task-queues, job-queues, task-schedulers, job-schedulers

readme

orkid

Dependencies Dev Dependencies Required Node License

Reliable and modern Redis-Streams based task queue for Node.js.

Screenshot

screenshot

Features

[x] Orkid lets Redis do the heavy lifting with Redis-Streams.
[x] Adjustable concurrency per consumer instance for scaling task processing. See example code. See example code.
[x] Job timeouts and retries. All configurable per consumer. See example code.
[x] Task Deduplication. If a task is already waiting in the queue, it can be configured to avoid queueing the same task again. (Useful for operations like posting database record updates to elasticsearch for re-indexing. Deduplication is a common pattern here to avoid unnecessary updates). See example code.
[x] Add tasks in bulk in one call. The producer will handle chunking and optimize round-trips to redis using pipelining.
[x] Monitoring and management UI for better visibility.
[ ] Rate-limiting workers. (work in progress)

Requirements

Node.js >= 10
Redis >= 5

👏 Important: Redis-Streams feature is not available before Redis version 5.

Install

npm install orkid --save

Examples

Basic example of producing and consuming tasks:

Producing tasks:

const { Producer } = require('orkid');

// `basic` is the queue name here
//  We'll use the same name in the consumer to process this task
const producer = new Producer('basic');

async function addTasks() {
  for (let i = 0; i < 10; i++) {
    console.log('Adding task ', i);
    await producer.addTask(i);
  }
}

addTasks()
  .then(() => {
    console.log('Done');
    process.exit(); // To disconnect from redis
  })
  .catch(e => console.error(e));

Consuming tasks:

const { Consumer } = require('orkid');

// Worker Function
async function workerFn(data, metadata) {
  let result;
  /*
    Do operation on `data` here
    and store the result in `result` variable

    Anything you return from this function will
    be saved in redis and can be viewed in the Orkid UI.

    Returning nothing is fine too.

    Throwing error will mark the job as failed,
    which can be retried too.
  */

  console.log('Task done!');
  return result;
}

// Consume task from the `basic` queue
const consumer = new Consumer('basic', workerFn);

// Start processing tasks!
// Important: Until you call this method, orkid consumer will do nothing.
consumer.start();

👏 More examples are available in the ./examples directory, including how to do task de-duplication, retry on failure, timeout etc. 👏

API Documentation

API Documentation is available here.

Monitoring and Management UI/Admin Panel

screenshot

You need to run orkid-ui separately for the dashboard. Detail instructions on how to run orkid-ui locally or in production using docker/docker-compose can be found here: https://github.com/mugli/orkid-ui#running-locally

Task/job life-cycle

[TODO: Add a flowchart here]

FAQ

<summary>Is this production ready?</summary> This project is under active development right now. API WILL introduce breaking changes until we reach version 1.0. After that semantic versioning will be followed.

<summary>Why a new job queue for Node.js?</summary> - All the redis-based solutions were created before Redis-Streams (https://redis.io/topics/streams-intro) became available. They all require a lot of work on the queue-side to ensure durability and atomicity of jobs handling operations. Redis-Streams was specifically designed to made this kind of tasks easier, thus allows simpler core in the queue and more reliable and maintainable operations.
- None of existing usable job queues in Node.js offers a monitoring option that we liked.
- None of the existing usable task queues support task de-duplication.

<summary>How do I set priority in the tasks?</summary> Redis-Streams isn't a right primitive to make a priority queue efficiently on top of it. Orkid doesn't support priority queues now and probably never will.
However, as a workaround, you can create a separate queue, keep its workload minimal and use it for high priority jobs with Orkid.

<summary>What is the order of job/task delivery?</summary> Jobs are processed in the order they are produced. However, if retry option is turned on and is applicable, failed jobs gets enqueued to the queue at once, along with other newly produced jobs.

Maintainer(s)

Mehdi Hasan Khan (@MehdiHK)

License

MIT

Related Projects

orkid-ui: Dashboard to monitor and manage Orkid task queue.
orkid-api: GraphQL API to monitor and manage Orkid task queue (used internally by orkid-ui).