CPU-Intensive Task Handling with NestJS

A production-ready demonstration of handling CPU-intensive operations (Excel file processing) in Node.js without blocking the main thread, showcasing scalable architecture patterns for high-concurrency applications.

🎯 Project Goals

This project demonstrates:

• ✅ Non-blocking I/O: Processing large files without blocking the main thread
• ✅ Worker Threads: CPU-intensive work runs in separate threads
• ✅ Job Queues: BullMQ for reliable task distribution and retry logic
• ✅ Horizontal Scalability: Architecture supports multiple workers across servers
• ✅ Progress Tracking: Real-time progress updates persisted in PostgreSQL
• ✅ Concurrency Control: Handle multiple simultaneous file uploads
• ✅ Production Patterns: Error handling, retries, and monitoring

🏗️ Architecture

┌─────────────┐
│   Client    │
└──────┬──────┘
       │ Upload Excel
       ▼
┌─────────────────────────────────────────────┐
│           NestJS API (Main Thread)          │
│  ┌──────────────────────────────────────┐   │
│  │  1. Save file to disk                │   │
│  │  2. Create DB record (QUEUED)        │   │
│  │  3. Add job to Redis queue           │   │
│  │  4. Return jobId immediately         │   │
│  └──────────────────────────────────────┘   │
└──────────────────┬──────────────────────────┘
                   │
                   ▼
         ┌──────────────────┐
         │   Redis Queue    │
         │    (BullMQ)      │
         └────────┬─────────┘
                  │
                  ▼
    ┌────────────────────────────────┐
    │   Queue Processor Worker       │
    │  ┌──────────────────────────┐  │
    │  │ 1. Pick job from queue   │  │
    │  │ 2. Update DB (PROCESSING)│  │
    │  │ 3. Spawn Worker Thread   │  │
    │  │ 4. Parse Excel file      │  │
    │  │ 5. Update progress       │  │
    │  │ 6. Save results to DB    │  │
    │  └──────────────────────────┘  │
    └────────────────────────────────┘
                  │
                  ▼
         ┌─────────────────┐
         │   PostgreSQL    │
         │  (Progress DB)  │
         └─────────────────┘

🚀 Key Features

1. Worker Threads for CPU-Intensive Tasks

Excel parsing runs in separate threads, preventing the main event loop from blocking:

const worker = new Worker('./excel.worker.js', {
  workerData: { buffer: Array.from(buffer) }
});

2. Job Queue with BullMQ

• Jobs persist in Redis (survives crashes)
• Automatic retry with exponential backoff
• Configurable concurrency
• Job status tracking

3. Progress Tracking

Real-time progress updates stored in PostgreSQL:

• QUEUED → PROCESSING → COMPLETED / FAILED
• Progress percentage (0-100%)
• Processed row count
• Error messages on failure

4. Scalability

• Vertical: Add more worker threads per server
• Horizontal: Add more servers, all connecting to same Redis queue
• Load balancing: Multiple workers process jobs concurrently

📦 Tech Stack

• Runtime: Node.js with TypeScript
• Framework: NestJS
• Queue: BullMQ (Redis-based)
• Database: PostgreSQL with Prisma ORM
• Excel Processing: ExcelJS
• Containerization: Docker & Docker Compose

🛠️ Setup & Installation

Prerequisites

• Node.js 18+
• Docker & Docker Compose
• npm or yarn

1. Clone the repository

git clone https://github.com/yourusername/cpu-intensive-handling.git
cd cpu-intensive-handling

2. Install dependencies

npm install

3. Configure environment

cp .env.example .env

Edit .env if needed:

# Database
DATABASE_URL="postgresql://root:root@localhost:5432/cpu_intensive_test?schema=public"

# Redis
REDIS_HOST=localhost
REDIS_PORT=6379
REDIS_PASSWORD=
REDIS_DB=0

# Server
PORT=3000
NODE_ENV=development

4. Start infrastructure

docker-compose up -d

This starts:

• PostgreSQL on port 5432
• Redis on port 6379

5. Run migrations

npx prisma migrate dev

6. Start the application

# Development mode with hot reload
npm run start:dev

# Production mode
npm run build
npm run start:prod

📚 API Endpoints

Upload Excel File

POST /excel/upload
Content-Type: multipart/form-data

# Response
{
  "message": "File queued for processing",
  "jobId": "1",
  "fileId": "uuid-here",
  "status": "QUEUED"
}

🧪 Load Testing

Test with the included load test script:

# Basic test (10 concurrent, 50 total)
./test_excel_process.sh

# Heavy load (20 concurrent, 100 total)
CONCURRENT=50 TOTAL=500 ./loadtest.sh

Expected Results

• ✅ API responds immediately with jobId
• ✅ Multiple uploads handled concurrently
• ✅ No request blocking
• ✅ Progress tracked in real-time

🎓 What This Demonstrates

For Backend Engineers:

1. Understanding Node.js limitations: Single-threaded event loop
2. Worker Threads: Moving CPU work off main thread
3. Job Queues: Async task processing patterns
4. Concurrency: Handling multiple operations simultaneously
5. Data persistence: Tracking state across requests
6. Error handling: Retries, failure states, cleanup

For System Architects:

1. Horizontal scaling: Queue-based architecture
2. Separation of concerns: API layer vs processing layer
3. Database design: Status tracking, job metadata
4. Infrastructure: Docker, Redis, PostgreSQL

For DevOps:

1. Container orchestration: Docker Compose
2. Configuration management: Environment variables
3. Resource management: Redis memory policies
4. Monitoring: Job states, progress tracking

🏭 Production Considerations

Implemented:

• ✅ Environment-based configuration
• ✅ Error handling and retries
• ✅ Job progress tracking
• ✅ Database transaction safety
• ✅ File cleanup after processing
• ✅ Redis eviction policy (noeviction)

📁 Project Structure

src/
├── modules/
│   ├── excel/
│   │   ├── excel.controller.ts    # API endpoints
│   │   ├── excel.service.ts       # Business logic
│   │   ├── excel.processor.ts     # Queue job processor
│   │   ├── excel.worker.ts        # Worker thread code
│   │   └── excel.module.ts        # Module configuration
│   └── prisma/
│       └── prisma.service.ts      # Database service
├── app.module.ts                  # Main application module
└── main.ts                        # Application entry point

prisma/
├── schema.prisma                  # Database schema
└── migrations/                    # Database migrations

uploads/                           # Temporary file storage (gitignored)

🔧 Database Schema

enum ExcelStatus {
  QUEUED
  PROCESSING
  COMPLETED
  FAILED
}

model ProcessedExcel {
  id           String      @id @default(uuid())
  name         String      // Original filename
  file         String      // File path
  status       ExcelStatus @default(QUEUED)
  
  jobId        String?     @unique
  progress     Int         @default(0)
  totalRows    Int?
  processedAt  DateTime?
  failedReason String?
  fileSize     Int?
  
  createdAt    DateTime    @default(now())
  updatedAt    DateTime    @updatedAt
}

🤝 Contributing

This is a portfolio/demonstration project, but suggestions are welcome!

📝 License

MIT

👤 Author

Marlon Anthony Laguna Jurado

• GitHub: @AnthonyLJ96
• LinkedIn: marlonlj

---

💡 Learning Resources

If you want to learn more about these concepts:

• Node.js Worker Threads
• BullMQ Documentation
• NestJS Queues
• Event Loop and Thread Pool

---

⭐ If this project helped you understand concurrent processing in Node.js, please give it a star!