Skip to content

Saloni1707/RedisDistro

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

28 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

RedisDistro

A fast, reliable, and extensible distributed queue system built atop Redis, written in TypeScript.
Perfect for background job processing, such as email delivery, with built-in retries, scheduling, dead-letter queues, and metrics.


Features

  • Redis-powered queue: Reliable and fast task queue leveraging Redis lists and sorted sets
  • Retry and dead-letter support: Automatic retries for failed tasks, with configurable dead-letter queue
  • Task scheduling: Schedule tasks for future execution
  • Worker-based processing: Scalable worker system to process tasks in parallel
  • REST API producer: Simple HTTP API for enqueuing and monitoring tasks
  • Metrics: Built-in queue and worker statistics
  • TypeScript-first: Modern and type-safe

Architecture Overview

  • Producer API: Accepts task enqueue requests via HTTP.
  • Redis: Stores main queue, scheduled tasks, dead-letter queue, and metrics.
  • Worker: Continuously processes and manages tasks, including retries and failures.

Workflow Diagram


How It Works

  1. Enqueue:
    Producer API validates and pushes tasks to Redis (immediately or scheduled).
  2. Worker Loop:
    • Periodically moves due scheduled tasks to the main queue.
    • Pops tasks from the main queue and processes them (e.g., sends email).
    • On failure, retries up to max attempts, then moves to dead-letter queue.
    • Updates metrics for processed/failed tasks.
  3. Monitoring:
    Both API and worker expose metrics for queue health and throughput.

Create a .env file with your configuration:

REDIS_URL=
QUEUE_KEY=
DEAD_KEY=

Quick Start

Install & Setup

git clone https://github.com/Saloni1707/RedisDistro.git
cd RedisDistro
npm install

Start the Producer API

npm run producer
# Or: node src/producer/server.js

Start a Worker Process

npm run worker
# Or: node src/worker/worker.js

Usage

Enqueue a Task

Send a POST request to enqueue an email job:

POST /enqueue/email
Content-Type: application/json

{
  "to": "recipient@example.com",
  "subject": "Your Subject",
  "body": "Your message body",
  "runAt": "tomorrow 8am" 

}

Response:

{ "message": "Task enqueued", "taskId": "..." }

License

MIT © Saloni1707

About

A fast, reliable redis-based background job processor for Node.js written in Typescript

Topics

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

 
 
 

Contributors