Handle Asynchronous Tasks with Node.js and BullMQ

Introduction

Handling asynchronous tasks efficiently is a key aspect of modern web development, and with Node.js and BullMQ, developers can streamline this process. Node.js, known for its non-blocking I/O operations, combined with BullMQ’s advanced job and queue management capabilities, creates a powerful solution for handling background tasks. This article delves into how Node.js, along with BullMQ, can improve the performance and scalability of your applications by managing heavy workloads, retrying failed jobs, and processing tasks asynchronously with minimal overhead. Let’s explore how this combination is revolutionizing asynchronous task handling in web development.

Prerequisites

To follow this tutorial, you’ll need to have the following:

• A Node.js development environment set up on your system. If you’re using Ubuntu 22.04, just check out our detailed guide on how to install Node.js on Ubuntu 22.04 to get everything ready. For other systems, you can follow our How to Install Node.js and Create a Local Development Environment guide, which has steps for different operating systems.
• Redis installed on your machine. If you’re on Ubuntu 22.04, follow Steps 1 through 3 in our tutorial on installing and securing Redis on Ubuntu 22.04. For other systems, don’t worry—we’ve got you covered in our guide on how to install and secure Redis, which walks you through the steps for different platforms.
• You should be comfortable with promises and async/await functions to follow along with this tutorial. If you’re still wrapping your head around these, check out our tutorial on understanding the Event Loop, Callbacks, Promises, and Async/Await in JavaScript. This will help you get a solid understanding of these core JavaScript concepts.
• A basic knowledge of using Express.js for building web apps is also needed. If you’re new to Express.js, no worries—take a look at our tutorial on getting started with Node.js and Express. It will guide you through creating and running a simple web app using Express.
• Familiarity with Embedded JavaScript (EJS) is required for this tutorial. If you haven’t worked with EJS templating before, we recommend checking out our tutorial on how to use EJS to template your Node.js app. It covers the basics of using EJS to render dynamic views in your Node.js app.
• A basic understanding of how to process images using the Sharp library is important too. Sharp is a super-efficient image processing library for Node.js, and you’ll need to be comfortable using it to follow along. If you’re not yet familiar with Sharp, take a look at our tutorial on processing images in Node.js with Sharp. It’ll help you get up to speed with resizing, cropping, and optimizing images in your Node.js applications.

Once you’ve got these prerequisites set up, you’ll be all set to dive into this tutorial and start implementing the concepts we’re going to cover.

Read more about setting up Node.js and Redis for web applications Installing Node.js and Redis for Web Applications.

Step 1 — Setting Up the Project Directory

So, here’s the deal: we’re going to create a directory and get everything ready for your app. In this tutorial, you’ll be building something that lets users upload an image, which will then get processed with the sharp package. Image processing can be a bit slow and resource-heavy, which is why we’re going to use bullmq to move that task to the background. That way, it doesn’t slow down the rest of the app. This method isn’t just for images; it can be used for any heavy-duty tasks that you’d rather not have hanging up the main process.

Alright, let’s start by creating a directory called image_processor and then jumping into it. To do that, just run:

$ mkdir image_processor && cd image_processor

Once you’re inside the directory, the next step is to initialize the project. This will set everything up like a basic package, and it’ll create a package.json file for you. To do this, simply run:

$ npm init -y

The -y flag means npm will automatically accept all the default options. Once you do that, your terminal should show something like this:

Wrote to /home/sammy/image_processor/package.json: {
    “name”: “image_processor”, 
    “version”: “1.0.0”, 
    “description”: “”,
    “main”: “index.js”, 
    “scripts”: {
        “test”: “echo "Error: no test specified" & exit 1”
        }
    “keywords”: [],
    “author”: “”,
    “license”: “ISC”
}

That’s npm confirming it’s created the package.json file for you. Here’s what you should keep an eye on:

• name: This is the name of your app (in this case, image_processor).
• version: The current version of your app (starting at 1.0.0).
• main: This is the entry point to your app (usually index.js).

If you want to dive deeper into the other properties in the file, you can check out the npm package.json docs.

Next up, we’re going to install all the dependencies you’ll need. These are the packages that’ll help you handle image uploads, process images, and manage background tasks. Here’s the list of packages to install:

• express: This is the framework we’ll use to build our web app. It makes things like routing and handling requests super easy.
• express-fileupload: This is a handy middleware that makes handling file uploads a breeze.
• sharp: This is the magic sauce for resizing, manipulating, and optimizing images.
• ejs: A templating engine that helps us generate HTML dynamically on the server side with Node.js.
• bullmq: This is a distributed task queue that we’ll use to send tasks (like image processing) to the background so the app stays responsive.
• bull-board: This gives us a nice UI dashboard on top of bullmq to monitor the status of all our jobs and tasks.

To install everything, just run:

$ npm install express express-fileupload sharp ejs bullmq @bull-board/express

Now, you’ll need an image to play around with for the tutorial. You can use the following curl command to grab one:

$ curl -O https://deved-images.nyc3.cdn.caasifyspaces.com/

At this point, you’ve got everything installed and the image you need. You’re all set to start building your Node.js app that integrates with bullmq to handle background tasks.

Read more about creating and setting up project directories for web applications Setting Up Project Directories for Web Applications.

Step 2 — Implementing a Time-Intensive Task Without bullmq

In this step, you’ll create an app using Express where users can upload images. Once an image is uploaded, the app will kick off a time-consuming task that uses the sharp module to resize the image into multiple sizes. After the app finishes processing, the resized images will be shown to the user. This will give you a solid understanding of how long-running tasks can affect the request/response cycle in a web app.

Setting Up the Project

To get started, open up your terminal and create a new file called index.js using nano or your favorite text editor:

$ nano index.js

In your index.js file, add the following code to import the necessary dependencies:

const path = require(“path”);
const fs = require(“fs”);
const express = require(“express”);
const bodyParser = require(“body-parser”);
const sharp = require(“sharp”);
const fileUpload = require(“express-fileupload”);

Here’s what each module does:

• The path module helps you handle file paths in Node.js.
• The fs module lets you interact with the file system, like reading and writing files.
• The express module is the web framework that powers the app and its routes.
• The body-parser module helps parse incoming request bodies, like JSON or URL-encoded data.
• The sharp module makes image processing easy, including resizing and converting images.
• The express-fileupload module makes it simple to upload files from an HTML form to the server.

Configuring Middleware

Now, let’s set up middleware for your app. This middleware will handle incoming requests and manage file uploads. Add the following code in your index.js file:

const app = express();
app.set(“view engine”, “ejs”);
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(fileUpload());
app.use(express.static(“public”));

Here’s what each part does:

• const app = express(); – This creates an Express application instance.
• app.set("view engine", "ejs"); – This configures Express to use the EJS templating engine for dynamic HTML rendering.
• app.use(bodyParser.json()); – This middleware parses incoming requests with JSON payloads.
• app.use(bodyParser.urlencoded({ extended: true })); – This middleware parses URL-encoded data, which is what forms usually send.
• app.use(fileUpload()); – This middleware handles file uploads.
• app.use(express.static("public")); – This serves static files like images or CSS from the “public” folder.

Setting Up Routes for Image Upload

Next, you’ll set up a route to show an HTML form for uploading images. Add the following code in your index.js :

app.get(“/”, function (req, res) {
    res.render(“form”);
});

This renders the form.ejs file when users visit the home page. The form will be used to upload the image. Now, create the views folder and go into it via the terminal:

$ mkdir views

$ cd views

Create the form.ejs file:

$ nano form.ejs

In your form.ejs file, add the following HTML code:

<!DOCTYPE html>
<html lang="en">
  <%- include('./head'); %>
  <body></p>
<div class="home-wrapper">
<h1>Image Processor</h1>
<p>Resize an image to multiple sizes and convert it to a <a href="https://en.wikipedia.org/wiki/WebP">webp</a> format.</p>
<form action="/upload" method="POST" enctype="multipart/form-data">
        <input type="file" name="image" placeholder="Select image from your computer" />
        <button type="submit">Upload Image</button>
      </form>
</p></div>
<p>  </body>
</html>

This form allows users to choose an image file from their computer and upload it to the server. The form uses multipart/form-data encoding to handle file uploads.

Setting Up the Image Upload Handling

Now, you’ll need to handle the file upload. Modify the index.js file and add the following code to handle the POST request to the /upload route:

app.post(“/upload”, async function (req, res) {
    const { image } = req.files;
    if (!image) return res.sendStatus(400);  // Send a 400 error if no image is uploaded
    const imageName = path.parse(image.name).name;
    const processImage = (size) => sharp(image.data)
        .resize(size, size)
        .webp({ lossless: true })
        .toFile(`./public/images/${imageName}-${size}.webp`);
    const sizes = [90, 96, 120, 144, 160, 180, 240, 288, 360, 480, 720, 1440];
    Promise.all(sizes.map(processImage));
    let counter = 0;
    for (let i = 0; i < 10_000_000_000; i++) {
        counter++;
    }
    res.redirect("/result");
});

Here’s how it works:

• req.files contains the uploaded file. You extract the image from this object.
• If no image is uploaded, the code sends a 400 status code back to the user.
• imageName is the name of the file without its extension.
• processImage() processes the image with sharp, resizing it, converting it to WebP format, and saving it in the public/images/ directory.
• The sizes array defines all the sizes you want to resize the image into.
• Promise.all() makes sure all images are processed asynchronously.
• A CPU-intensive loop is added just to simulate a delay, allowing you to see how long the image processing takes.

Displaying the Processed Images

Now, you need a route to show the resized images after processing. Add this code in your index.js :

app.get(“/result”, (req, res) => {
    const imgDirPath = path.join(__dirname, “./public/images”);
    let imgFiles = fs.readdirSync(imgDirPath).map((image) => {
        return `images/${image}`;
    });
    res.render(“result”, { imgFiles });
});

This code defines the /result route. It reads the image files from the public/images folder, creates an array of their paths, and sends that array to the result.ejs template.

Creating the Result Page

Next, create the result.ejs file to display the processed images. Open the views folder:

$ cd views

$ nano result.ejs

In your result.ejs file, add the following HTML code to show the resized images:

<!DOCTYPE html>
<html lang="en">
  <%- include('./head'); %>
  <body></p>
<div class="gallery-wrapper">
      <% if (imgFiles.length > 0) { %></p>
<p>The following are the processed images:</p>
<ul>
          <% imgFiles.forEach((imgFile) => { %></p>
<li><img decoding="async" src="<%= imgFile %>” /></li>
<p>          <% }) %>
        </ul>
<p>      <% } else { %></p>
<p>The image is being processed. Refresh after a few seconds to view the resized images.</p>
<p>      <% } %>
    </div>
<p>  </body>
</html>

This code checks if there are any images in the imgFiles array. If there are, it displays each image in a list. If not, it asks the user to refresh the page in a few seconds to see the resized images.

Styling the Application

To style your app, create the necessary directories and a main.css file. In the terminal, create the public and css directories:

$ mkdir -p public/css

$ cd public/css

$ nano main.css

In main.css , add the following styles:

body {
    background: #f8f8f8;
}
h1 {
    text-align: center;
}
p {
    margin-bottom: 20px;
}
a:link, a:visited {
    color: #00bcd4;
}
button[type=”submit”] {
    background: none;
    border: 1px solid orange;
    padding: 10px 30px;
    border-radius: 30px;
    transition: all 1s;
}
button[type=”submit”]:hover {
    background: orange;
}
input[type=”file”]::file-selector-button {
    border: 2px solid #2196f3;
    padding: 10px 20px;
    border-radius: 0.2em;
    background-color: #2196f3;
}
ul {
    list-style: none;
    padding: 0;
    display: flex;
    flex-wrap: wrap;
    gap: 20px;
}
.home-wrapper {
    max-width: 500px;
    margin: 0 auto;
    padding-top: 100px;
}
.gallery-wrapper {
    max-width: 1200px;
    margin: 0 auto;
}

These styles make the page look organized and visually appealing. After adding the CSS, save the file and close it.

Starting the Server

Now that everything is set up, start the Express server. Open the terminal and run:

$ node index.js

The terminal will show that the server is running:

Server running on port 3000

Visit http://localhost:3000/ in your browser. You’ll see the image upload form. Once an image is uploaded, the app will process the image, and you’ll be redirected to the /result route to see the resized images.

Stopping the Server

To stop the server, press CTRL+C in the terminal. Remember, Node.js doesn’t automatically reload the server when files change, so you’ll need to stop and restart it whenever you update the code.

That’s all for handling time-intensive tasks and observing how they impact your app’s request/response cycle.

Read more about handling time-intensive tasks in web applications Handling Time-Intensive Tasks in Web Applications.

Step 3 — Executing Time-Intensive Tasks Asynchronously with bullmq

In this step, you will offload a time-intensive task to the background using bullmq. This adjustment will free the request/response cycle and allow your app to respond to users immediately while the image is being processed. To do that, you need to create a succinct description of the job and add it to a queue with bullmq. A queue is a data structure that works similarly to how a queue works in real life. When people line up to enter a space, the first person on the line will be the first person to enter the space. Anyone who comes later will line up at the end of the line and will enter the space after everyone who precedes them in line until the last person enters the space. With the queue data structure’s First-In, First-Out (FIFO) process, the first item added to the queue is the first item to be removed (dequeue). With bullmq, a producer will add a job in a queue, and a consumer (or worker) will remove a job from the queue and execute it. The queue in bullmq is stored in Redis.

When you describe a job and add it to the queue, an entry for the job is created in a Redis queue. A job description can be a string or an object with properties that contain minimal data or references to the data that will allow bullmq to execute the job later. Once you define the functionality to add jobs to the queue, you move the time-intensive code into a separate function. Later, bullmq will call this function with the data you stored in the queue when the job is dequeued. Once the task has finished, bullmq will mark it completed, pull another job from the queue, and execute it.

Open index.js in your editor:

nano index.js

In your index.js file, add the highlighted lines to create a queue in Redis with bullmq:

const fileUpload = require(“express-fileupload”);
const { Queue } = require(“bullmq”);</p>
<p>const redisOptions = {
  host: “localhost”,
  port: 6379
};</p>
<p>const imageJobQueue = new Queue(“imageJobQueue”, {
  connection: redisOptions,
});</p>
<p>async function addJob(job) {
  await imageJobQueue.add(job.type, job);
}

You start by extracting the Queue class from bullmq, which is used to create a queue in Redis. You then set the redisOptions variable to an object with properties that the Queue class instance will use to establish a connection with Redis. You set the host property value to localhost because Redis is running on your local machine.

You also set the port property value to 6379, the default port that Redis uses to listen for connections. If you have set up port forwarding to a remote server running Redis and the app together, you do not need to update the host property, but you will need to use the port forwarding connection every time you log in to your server to run the app.

Next, you set the imageJobQueue variable to an instance of the Queue class, taking the queue’s name as its first argument and an object as a second argument. The object has a connection property with the value set to an object in the redisOptions variable. After instantiating the Queue class, a queue called imageJobQueue will be created in Redis. Finally, you define the addJob() function that you will use to add a job in the imageJobQueue. The function takes a parameter of job containing the information about the job (you will call the addJob() function with the data you want to save in a queue). In the function, you invoke the add() method of the imageJobQueue, taking the name of the job as the first argument and the job data as the second argument.

Add the highlighted code to call the addJob() function to add a job in the queue:

app.post(“/upload”, async function (req, res) {
  const { image } = req.files;</p>
<p>  if (!image) return res.sendStatus(400);
  const imageName = path.parse(image.name).name;</p>
<p>  await addJob({
    type: “processUploadedImages”,
    image: {
      data: image.data.toString(“base64”),
      name: image.name
    },
  });</p>
<p>  res.redirect(“/result”);
});

Here, you call the addJob() function with an object that describes the job. The object has the type attribute with a value of the name of the job. The second property, image, is set to an object containing the image data the user has uploaded. Because the image data in image.data is in a buffer (binary form), you invoke JavaScript’s toString() method to convert it to a string that can be stored in Redis, which will set the data property as a result. The image property is set to the name of the uploaded image (including the image extension). You have now defined the information needed for bullmq to execute this job later. Depending on your job, you may add more job information or less.

Save and close your file.

Next, create and open the utils.js file that will contain the image processing code:

nano utils.js

In your utils.js file, add the following code to define the function for processing an image:

const path = require(“path”);
const sharp = require(“sharp”);</p>
<p>function processUploadedImages(job) {}</p>
<p>module.exports = { processUploadedImages };

You import the modules necessary to process images and compute paths in the first two lines. Then you define the processUploadedImages() function, which will contain the time-intensive image processing task. This function takes a job parameter that will be populated when the worker fetches the job data from the queue and then invokes the processUploadedImages() function with the queue data. You also export the processUploadedImages() function so that you can reference it in other files.

Save and close your file.

Return to the index.js file:

nano index.js

Copy the highlighted lines from the index.js file, then delete them from this file. You will need the copied code momentarily, so save it to a clipboard. If you are using nano, you can highlight these lines and right-click with your mouse to copy the lines:

app.post(“/upload”, async function (req, res) {
  const { image } = req.files;</p>
<p>  if (!image) return res.sendStatus(400);
  const imageName = path.parse(image.name).name;
  const processImage = (size) => sharp(image.data)
    .resize(size, size)
    .webp({ lossless: true })
    .toFile(`./public/images/${imageName}-${size}.webp`);</p>
<p>  sizes = [90, 96, 120, 144, 160, 180, 240, 288, 360, 480, 720, 1440];
  Promise.all(sizes.map(processImage))</p>
<p>  let counter = 0;
  for (let i = 0; i < 10_000_000_000; i++) {
    counter++;
  };

  res.redirect("/result");
});

The post method for the upload route will now match the following:

app.post(“/upload”, async function (req, res) {
  const { image } = req.files;</p>
<p>  if (!image) return res.sendStatus(400);</p>
<p>  await addJob({
    type: “processUploadedImages”,
    image: {
      data: image.data.toString(“base64”),
      name: image.name
    },
  });</p>
<p>  res.redirect(“/result”);
});

Save and close your file, then open the utils.js file:

nano utils.js

In your utils.js file, paste the lines you just copied for the /upload route callback into the processUploadedImages function:

function processUploadedImages(job) {
  const imageName = path.parse(image.name).name;
  const processImage = (size) => sharp(image.data)
    .resize(size, size)
    .webp({ lossless: true })
    .toFile(`./public/images/${imageName}-${size}.webp`);</p>
<p>  sizes = [90, 96, 120, 144, 160, 180, 240, 288, 360, 480, 720, 1440];</p>
<p>  Promise.all(sizes.map(processImage))
  let counter = 0;
  for (let i = 0; i < 10_000_000_000; i++) {
    counter++;
  };
}

Now that you have moved the code for processing an image, you need to update it to use the image data from the job parameter of the processUploadedImages() function you defined earlier. To do that, add and update the highlighted lines below:

function processUploadedImages(job) {
  const imageFileData = Buffer.from(job.image.data, “base64”);
  const imageName = path.parse(job.image.name).name;</p>
<p>  const processImage = (size) => sharp(imageFileData)
    .resize(size, size)
    .webp({ lossless: true })
    .toFile(`./public/images/${imageName}-${size}.webp`);</p>
<p>  sizes = [90, 96, 120, 144, 160, 180, 240, 288, 360, 480, 720, 1440];</p>
<p>  Promise.all(sizes.map(processImage))
  let counter = 0;
  for (let i = 0; i < 10_000_000_000; i++) {
    counter++;
  };
}

You convert the stringified version of the image data back to binary with the Buffer.from() method. Then you update path.parse() with a reference to the image name saved in the queue. After that, you update the sharp() method to take the image binary data stored in the imageFileData variable. The complete utils.js file will now match the following:

const path = require(“path”);
const sharp = require(“sharp”);</p>
<p>function processUploadedImages(job) {
  const imageFileData = Buffer.from(job.image.data, “base64”);
  const imageName = path.parse(job.image.name).name;</p>
<p>  const processImage = (size) => sharp(imageFileData)
    .resize(size, size)
    .webp({ lossless: true })
    .toFile(`./public/images/${imageName}-${size}.webp`);</p>
<p>  sizes = [90, 96, 120, 144, 160, 180, 240, 288, 360, 480, 720, 1440];</p>
<p>  Promise.all(sizes.map(processImage))
  let counter = 0;
  for (let i = 0; i < 10_000_000_000; i++) {
    counter++;
  };
}

module.exports = { processUploadedImages };

Save and close your file, then return to the index.js file:

nano index.js

The sharp variable is no longer needed as a dependency since the image is now processed in the utils.js file. Delete the highlighted line from the file:

const bodyParser = require(“body-parser”);
const sharp = require(“sharp”);
const fileUpload = require(“express-fileupload”);
const { Queue } = require(“bullmq”);

Save and close your file.

You have now defined the functionality to create a queue in Redis and add a job. You also defined the processUploadedImages() function to process uploaded images. The remaining task is to create a consumer (or worker) that will pull a job from the queue and call the processUploadedImages() function with the job data.

Create a worker.js file in your editor:

nano worker.js

In your worker.js file, add the following code:

const { Worker } = require(“bullmq”);
const { processUploadedImages } = require(“./utils”);</p>
<p>const workerHandler = (job) => {
  console.log(“Starting job:”, job.name);
  processUploadedImages(job.data);
  console.log(“Finished job:”, job.name);
  return;
};

In the first line, you import the Worker class from bullmq; when instantiated, this will start a worker that dequeues jobs from the queue in Redis and executes them. Next, you reference the processUploadedImages() function from the utils.js file so that the worker can call the function with the data in the queue. You define a workerHandler() function that takes a job parameter containing the job data in the queue. In the function, you log that the job has started, then invoke processUploadedImages() with the job data. After that, you log a success message and return null.

To allow the worker to connect to Redis, dequeue a job from the queue, and call the workerHandler() with the job data, add the following lines to the file:

const workerOptions = {
  connection: {
    host: “localhost”,
    port: 6379,
  },
};</p>
<p>const worker = new Worker(“imageJobQueue”, workerHandler, workerOptions);
console.log(“Worker !”);

Here, you set the workerOptions variable to an object containing Redis’s connection settings. You set the worker variable to an instance of the Worker class that takes the following parameters: imageJobQueue: the name of the job queue, workerHandler: the function that will run after a job has been dequeued from the Redis queue, and workerOptions: the Redis config settings that the worker uses to establish a connection with Redis.

Finally, you log a success message. After adding the lines, save and close your file.

You have now defined the bullmq worker functionality to dequeue jobs from the queue and execute them.

In your terminal, remove the images in the public/images directory so that you can start fresh for testing your app:

rm public/images/*

Next, run the index.js file:

node index.js

The app will start:

Server running on port 3000

You’ll now start the worker. Open a second terminal session and navigate to the project directory:

cd image_processor/

Start the worker with the following command:

node worker.js

The worker will start:

Worker !

Visit http://localhost:3000/ in your browser. Press the “Choose File” button and select the underwater.png from your computer, then press the “Upload Image” button. You may receive an instant response that tells you to refresh the page after a few seconds. Alternatively, you might receive an instant response with some processed images on the page while others are still being processed. You can refresh the page a few times to load all the resized images.

Return to the terminal where your worker is running. That terminal will have a message that matches the following:

Worker ! Starting job: processUploadedImages Finished job: processUploadedImages

The output confirms that bullmq ran the job successfully.

Your app can still offload time-intensive tasks even if the worker is not running. To demonstrate this, stop the worker in the second terminal with CTRL+C. In your initial terminal session, stop the Express server and remove the images in public/images:

rm public/images/*

After that, start the server again:

node index.js

In your browser, visit http://localhost:3000/ and upload the underwater.png image again. When you are redirected to the /result path, the images will not show on the page because the worker is not running:

Return to the terminal where you ran the worker and start the worker again:

node worker.js

The output will match the following, which lets you know that the job has started:

Worker ! Starting job: processUploadedImages

After the job has been completed and the output includes a line that reads “Finished job: processUploadedImages,” refresh the browser. The images will now load.

Stop the server and the worker.

You now can offload a time-intensive task to the background and execute it asynchronously using bullmq. In the next step, you will set up a dashboard to monitor the status of the queue.

Read more about executing tasks asynchronously in Node.js applications Executing Tasks Asynchronously in Node.js Applications.

Step 4 — Adding a Dashboard to Monitor bullmq Queues

In this step, you’ll use the bull-board package to keep an eye on your jobs in the Redis queue with a visual dashboard. The bull-board package gives you a user-friendly interface (UI) that automatically sets up a dashboard. This dashboard will organize and show detailed info about the bullmq jobs stored in the Redis queue. You can then check out the jobs in various states—whether they’re completed, still waiting, or have failed—right from your browser, without needing to mess with the Redis CLI in the terminal.

Start by opening your index.js file in your text editor so you can modify your app:

$ nano index.js

Now, add this code to import the bull-board and related packages. These imports are necessary to make the dashboard work in your app:

const { Queue } = require(“bullmq”);
const { createBullBoard } = require(“@bull-board/api”);
const { BullMQAdapter } = require(“@bull-board/api/bullMQAdapter”);
const { ExpressAdapter } = require(“@bull-board/express”);

In the code above, you’re importing the createBullBoard() function from the bull-board package. You’re also bringing in the BullMQAdapter, which lets the bull-board access and manage your bullmq queues, and the ExpressAdapter, which connects the dashboard to an Express.js server to show the UI.

The next step is to link up bull-board with bullmq. You’ll do this by setting up the dashboard with the right queues and server adapter for your Express app.

async function addJob(job) {
  await imageJobQueue.add(job.type, job);
}

const serverAdapter = new ExpressAdapter();
const bullBoard = createBullBoard({
  queues: [new BullMQAdapter(imageJobQueue)],
  serverAdapter: serverAdapter,
});
serverAdapter.setBasePath(“/admin”);

Here, the serverAdapter is set up as an instance of the ExpressAdapter , which you need to integrate with Express. Then, the createBullBoard() function is called, with an object that has two properties: queues and serverAdapter . The queues property is an array that includes the bullmq queues you’ve set up—like the imageJobQueue here. The serverAdapter property holds the ExpressAdapter instance that will handle the routes for the dashboard.

Once the dashboard is set up, the next step is to define the /admin route where the dashboard will be available. You can do that by adding this middleware:

app.use(express.static(“public”));
app.use(“/admin”, serverAdapter.getRouter());

This code tells your Express server to serve static files from the public folder and sets up the /admin route to show the dashboard. Now, any traffic to http://localhost:3000/admin will open the dashboard interface.

Here’s what the complete index.js file will look like once you’ve added everything you need:

const path = require(“path”);
const fs = require(“fs”);
const express = require(“express”);
const bodyParser = require(“body-parser”);
const fileUpload = require(“express-fileupload”);
const { Queue } = require(“bullmq”);
const { createBullBoard } = require(“@bull-board/api”);
const { BullMQAdapter } = require(“@bull-board/api/bullMQAdapter”);
const { ExpressAdapter } = require(“@bull-board/express”);</p>
<p>const redisOptions = { host: “localhost”, port: 6379 };</p>
<p>const imageJobQueue = new Queue(“imageJobQueue”, { connection: redisOptions });</p>
<p>async function addJob(job) {
  await imageJobQueue.add(job.type, job);
}</p>
<p>const serverAdapter = new ExpressAdapter();
const bullBoard = createBullBoard({
  queues: [new BullMQAdapter(imageJobQueue)],
  serverAdapter: serverAdapter,
});
serverAdapter.setBasePath(“/admin”);</p>
<p>const app = express();
app.set(“view engine”, “ejs”);
app.use(bodyParser.json());
app.use(bodyParser.urlencoded({ extended: true }));
app.use(fileUpload());</p>
<p>app.use(express.static(“public”));
app.use(“/admin”, serverAdapter.getRouter());</p>
<p>app.get(“/”, function (req, res) {
  res.render(“form”);
});</p>
<p>app.get(“/result”, (req, res) => {
  const imgDirPath = path.join(__dirname, “./public/images”);
  let imgFiles = fs.readdirSync(imgDirPath).map((image) => {
    return `images/${image}`;
  });
  res.render(“result”, { imgFiles });
});</p>
<p>app.post(“/upload”, async function (req, res) {
  const { image } = req.files;
  if (!image) return res.sendStatus(400);
  await addJob({
    type: “processUploadedImages”,
    image: {
      data: Buffer.from(image.data).toString(“base64”),
      name: image.name,
    },
  });
  res.redirect(“/result”);
});</p>
<p>app.listen(3000, function () {
  console.log(“Server running on port 3000”);
});

After saving and closing the file, run your app by executing:

$ node index.js

Once the server is up and running, open your browser and go to http://localhost:3000/admin to see the dashboard. Now you can keep track of your jobs and interact with the UI to check out jobs that are done, have failed, or are paused.

In the dashboard, you’ll find detailed info about each job, such as its type, data, and status. You can also check out different tabs, like the Completed tab for jobs that finished successfully, the Failed tab for jobs that ran into issues, and the Paused tab for jobs that are currently on hold.

With this setup, you’ll be able to easily monitor and manage your Redis queue jobs with the bull-board dashboard.

Read more about setting up dashboards to monitor background jobs in web applications Setting Up Dashboards to Monitor Background Jobs in Web Applications.

Conclusion

Conclusion

In this article, we explored how to handle asynchronous tasks efficiently using Node.js and BullMQ. By integrating BullMQ with Node.js, you can manage background jobs and queues, ensuring that your applications scale and perform effectively. We discussed setting up queues, handling job retries, and optimizing performance for long-running tasks. With BullMQ, you gain a robust solution for managing async tasks with advanced features like rate-limiting and delayed jobs.

Node.js and BullMQ together provide a seamless way to handle asynchronous operations in large-scale applications. Whether you’re working on microservices or complex backend systems, leveraging these tools can significantly improve your workflow and application performance.

As you continue to build with Node.js, stay tuned for new developments in job queue management, as both Node.js and BullMQ are evolving with new features that further enhance scalability and ease of use.

Snippet for Search Results: Learn how to manage asynchronous tasks effectively with Node.js and BullMQ, ensuring optimal performance for your applications.

This approach will help ensure your Node.js applications can scale and handle tasks more efficiently in the future.