Nas-Notification/docs/queue-system-explained.md

# Notification Queue System

## Overview

The notification system now uses a **background queue processor** that runs automatically every 30 seconds to send queued emails.

## How It Works

### 1. **Queue Creation**
When you create a notification:
```
POST /api/public/create-notification
  ↓
Creates notification → status: "sending"
Creates recipients → status: "pending"
Creates queue items → status: "queued"
  ↓
Returns immediately (non-blocking) ✅
```

### 2. **Background Processing**
A background worker runs automatically:
```
Server starts
  ↓
Queue Processor Plugin initializes
  ↓
Processes queue every 30 seconds
  ↓
Sends emails → Updates status → Repeats
```

### 3. **Status Updates**
```
notification.status: "sending" → "sent" (when all recipients done)
recipients.status: "pending" → "sent" or "failed"
queue.status: "queued" → "processing" → "completed" or "failed"
```

## Components

### 1. Queue Processor (`server/utils/queueProcessor.js`)
- **Runs every 30 seconds**
- Processes up to 50 jobs per batch
- Prevents concurrent processing (lock mechanism)
- Logs activity for monitoring

### 2. Nitro Plugin (`server/plugins/queueProcessor.js`)
- **Auto-starts** when server starts
- Runs in background
- Gracefully shuts down with server

### 3. Email Service (`server/utils/emailService.js`)
- `processEmailQueue()` - Processes queued emails
- `sendEmailNotification()` - Sends individual emails
- Updates all statuses (notification, recipient, queue)

## Queue Flow

```
┌─────────────────────────────────────────┐
│  Create Notification API                │
│  - Creates notification (sending)       │
│  - Creates recipients (pending)         │
│  - Creates queue items (queued)         │
│  - Returns immediately ✅                │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│  Background Queue Processor             │
│  (Runs every 30 seconds)                │
│                                         │
│  1. Find queued items (scheduled_for <= now)
│  2. For each item:                      │
│     - Mark as "processing"              │
│     - Send email via SMTP               │
│     - Update recipient status           │
│     - Update queue status               │
│     - Update notification status        │
│  3. Log results                         │
└─────────────────────────────────────────┘
                  ↓
┌─────────────────────────────────────────┐
│  Email Sent!                            │
│  - notification.status → "sent"         │
│  - recipient.status → "sent"            │
│  - queue.status → "completed"           │
│  - notification.actual_sent = count     │
└─────────────────────────────────────────┘
```

## Current Implementation

### ✅ What We Have:
- **Non-blocking API** - Returns immediately after queuing
- **Background processor** - Runs every 30 seconds
- **Auto-start on server boot** - Via Nitro plugin
- **Concurrent processing prevention** - Lock mechanism
- **Status updates** - All tables updated correctly
- **Error handling** - Failed emails marked as failed
- **Logging** - Console logs for monitoring

### ❌ What We Don't Have (Yet):
- **BullMQ** - Not using a production queue system
- **Redis** - No distributed queue
- **Worker pools** - Single-threaded processing
- **Job retries** - No automatic retry on failure
- **Priority queues** - All jobs equal priority
- **Rate limiting** - No per-provider rate limits

## Manual Queue Processing

You can manually trigger queue processing via API:

```bash
POST /api/notifications/queue/process
```

This is useful for:
- Testing
- Debugging
- Force processing without waiting

## Monitoring

### Console Logs:
```
🚀 Starting background queue processor...
✅ Queue processor started (runs every 30 seconds)
⚙️  Processing notification queue...
✅ Queue processed: 5 sent, 0 failed, 5 total
```

### Check Queue Status:
```sql
-- See queued items
SELECT * FROM notification_queue WHERE status = 'queued';

-- See processing status
SELECT
  n.id,
  n.title,
  n.status,
  COUNT(nr.id) as total_recipients,
  SUM(CASE WHEN nr.status = 'sent' THEN 1 ELSE 0 END) as sent,
  SUM(CASE WHEN nr.status = 'failed' THEN 1 ELSE 0 END) as failed,
  SUM(CASE WHEN nr.status = 'pending' THEN 1 ELSE 0 END) as pending
FROM notifications n
LEFT JOIN notification_recipients nr ON n.id = nr.notification_id
GROUP BY n.id, n.title, n.status;
```

## Configuration

### Queue Processing Interval:
Change in `server/utils/queueProcessor.js`:
```javascript
// Current: 30 seconds
processorInterval = setInterval(processQueue, 30000);

// Faster: 10 seconds
processorInterval = setInterval(processQueue, 10000);

// Slower: 1 minute
processorInterval = setInterval(processQueue, 60000);
```

### Batch Size:
Change in `server/utils/emailService.js`:
```javascript
// Current: 50 jobs per batch
take: 50

// Larger batch: 100 jobs
take: 100

// Smaller batch: 10 jobs
take: 10
```

## Future: Upgrading to BullMQ

If you want a production-grade queue system:

### Install BullMQ:
```bash
npm install bullmq ioredis
```

### Benefits:
- ✅ Persistent queue (survives server restart)
- ✅ Automatic retries with exponential backoff
- ✅ Priority queues
- ✅ Concurrent workers
- ✅ Rate limiting per provider
- ✅ Job scheduling
- ✅ Web UI for monitoring
- ✅ Distributed processing

### When to Upgrade:
- High volume (>1000 emails/day)
- Multiple servers
- Need guaranteed delivery
- Need advanced retry logic
- Need monitoring dashboard

## Troubleshooting

### Queue not processing?
1. Check server logs for errors
2. Check if plugin is loaded: `grep "queue processor" logs`
3. Manually trigger: `POST /api/notifications/queue/process`

### Emails not sending?
1. Check SMTP credentials in delivery settings
2. Check notification_queue table for errors
3. Check notification_recipients for failed status
4. Review email service logs

### Status stuck on "sending"?
1. Check if recipients are still "pending"
2. Manually trigger queue processing
3. Check for errors in notification_queue table

## Summary

**Current System:**
- Simple, lightweight queue using database + setInterval
- Runs every 30 seconds automatically
- Non-blocking API responses
- Suitable for low-medium volume

**When to upgrade:**
- Need higher reliability
- Higher volume
- Multiple servers
- Advanced features

The current system works well for most use cases and can handle thousands of notifications per day!