feat: implement real IMAP message parsing with native CouchDB attachments

- Replace placeholder message generation with actual IMAP message fetching using go-message library - Add per-account CouchDB databases for better organization and isolation - Implement native CouchDB attachment storage with proper revision management - Add command line argument parsing with --max-messages flag for controlling message processing limits - Support both sync and archive modes with proper document synchronization - Add comprehensive test environment with Podman containers (GreenMail IMAP server + CouchDB) - Implement full MIME multipart parsing for proper body and attachment extraction - Add TLS and plain IMAP connection support based on port configuration - Update configuration system to support sync vs archive modes - Create test scripts and sample data for development and testing Key technical improvements: - Real email envelope and header processing with go-imap v2 API - MIME Content-Type and Content-Disposition parsing for attachment detection - CouchDB document ID generation using mailbox_uid format for uniqueness - Duplicate detection and prevention to avoid re-storing existing messages - Proper error handling and connection management for IMAP operations 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
2025-08-01 17:04:10 +02:00 · 2025-08-01 17:04:10 +02:00 · ea6235b674
commit ea6235b674
parent 79f19a8877
22 changed files with 1262 additions and 66 deletions
--- a/test/README.md
+++ b/test/README.md
@ -0,0 +1,150 @@
+# mail2couch Test Environment
+
+This directory contains a complete test environment for mail2couch using Podman containers.
+
+## Overview
+
+The test environment provides:
+- **CouchDB**: Database for storing email messages
+- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages
+- **Test Configuration**: Ready-to-use config for testing both sync and archive modes
+
+## Quick Start
+
+### Run Full Integration Tests
+```bash
+./run-tests.sh
+```
+This will:
+1. Start all containers
+2. Populate test data
+3. Run mail2couch
+4. Verify results
+5. Clean up
+
+### Manual Testing
+```bash
+# Start test environment
+./start-test-env.sh
+
+# Run mail2couch manually
+cd ../go
+./mail2couch -config ../test/config-test.json
+
+# Stop test environment when done
+cd ../test
+./stop-test-env.sh
+```
+
+## Test Accounts
+
+The test environment includes these IMAP accounts:
+
+| Username | Password | Mode | Purpose |
+|----------|----------|------|---------|
+| `testuser1` | `password123` | archive | General archive testing |
+| `testuser2` | `password456` | - | Additional test user |
+| `syncuser` | `syncpass` | sync | Testing sync mode (1-to-1) |
+| `archiveuser` | `archivepass` | archive | Testing archive mode |
+
+Each account contains:
+- 10 messages in INBOX (every 3rd has an attachment)
+- 3 messages in Sent folder
+- Various message types for comprehensive testing
+
+## Services
+
+### CouchDB
+- **URL**: http://localhost:5984
+- **Admin**: `admin` / `password`
+- **Web UI**: http://localhost:5984/_utils
+
+### GreenMail Server
+- **Host**: localhost
+- **IMAP Port**: 3143 (plain)
+- **IMAPS Port**: 3993 (SSL)
+- **SMTP Port**: 3025
+- **Server**: GreenMail (Java-based test server)
+
+## Database Structure
+
+mail2couch will create separate databases for each mail source:
+- `test_user_1` - Test User 1 (archive mode)
+- `test_sync_user` - Test Sync User (sync mode)  
+- `test_archive_user` - Test Archive User (archive mode)
+
+## Testing Sync vs Archive Modes
+
+### Sync Mode (`syncuser`)
+- Database exactly matches mail account
+- If messages are deleted from IMAP, they're removed from CouchDB
+- 1-to-1 relationship
+
+### Archive Mode (`archiveuser`, `testuser1`)
+- Database preserves all messages ever seen
+- Messages deleted from IMAP remain in CouchDB
+- Archive/backup behavior
+
+## File Structure
+
+```
+test/
+├── podman-compose.yml          # Container orchestration
+├── config-test.json            # Test configuration
+├── run-tests.sh               # Full integration test
+├── start-test-env.sh          # Start environment
+├── stop-test-env.sh           # Stop environment
+├── generate-ssl.sh            # Generate SSL certificates
+├── populate-test-messages.sh  # Create test messages
+├── dovecot/
+│   ├── dovecot.conf          # Dovecot configuration
+│   ├── users                 # User database
+│   ├── passwd                # Password database
+│   └── ssl/                  # SSL certificates
+└── README.md                 # This file
+```
+
+## Prerequisites
+
+- Podman and podman-compose
+- OpenSSL (for certificate generation)
+- curl and nc (for connectivity checks)
+- Go (for building mail2couch)
+
+## Troubleshooting
+
+### Containers won't start
+```bash
+# Check podman status
+podman ps -a
+
+# View logs
+podman logs mail2couch_test_couchdb
+podman logs mail2couch_test_imap
+```
+
+### CouchDB connection issues
+- Verify CouchDB is running: `curl http://localhost:5984`
+- Check admin credentials: `admin/password`
+
+### IMAP connection issues
+- Test IMAP connection: `nc -z localhost 143`
+- Check Dovecot logs: `podman logs mail2couch_test_imap`
+
+### Permission issues
+- Ensure scripts are executable: `chmod +x *.sh`
+- Check file permissions in dovecot directory
+
+## Advanced Usage
+
+### Add custom test messages
+Edit `populate-test-messages.sh` to create additional test scenarios.
+
+### Modify IMAP configuration
+Edit `dovecot/dovecot.conf` and restart containers.
+
+### Test with SSL
+Update `config-test.json` to use port 993 and enable SSL.
+
+### Custom test scenarios
+Create additional configuration files for specific test cases.
--- a/test/config-test.json
+++ b/test/config-test.json
@ -0,0 +1,55 @@
+{
+  "couchDb": {
+    "url": "http://localhost:5984",
+    "user": "admin",
+    "password": "password",
+    "database": "mail_backup_test"
+  },
+  "mailSources": [
+    {
+      "name": "Test User 1",
+      "enabled": true,
+      "protocol": "imap",
+      "host": "localhost",
+      "port": 3143,
+      "user": "testuser1",
+      "password": "password123",
+      "mode": "archive",
+      "folderFilter": {
+        "include": ["INBOX", "Sent"],
+        "exclude": []
+      },
+      "messageFilter": {}
+    },
+    {
+      "name": "Test Sync User", 
+      "enabled": true,
+      "protocol": "imap",
+      "host": "localhost",
+      "port": 3143,
+      "user": "syncuser",
+      "password": "syncpass",
+      "mode": "sync",
+      "folderFilter": {
+        "include": ["INBOX"],
+        "exclude": []
+      },
+      "messageFilter": {}
+    },
+    {
+      "name": "Test Archive User",
+      "enabled": true,
+      "protocol": "imap",
+      "host": "localhost", 
+      "port": 3143,
+      "user": "archiveuser",
+      "password": "archivepass",
+      "mode": "archive",
+      "folderFilter": {
+        "include": [],
+        "exclude": ["Drafts"]
+      },
+      "messageFilter": {}
+    }
+  ]
+}
--- a/test/dovecot/dovecot.conf
+++ b/test/dovecot/dovecot.conf
@ -0,0 +1,79 @@
+# Dovecot configuration for testing mail2couch
+
+# Basic settings
+protocols = imap
+listen = *
+
+# SSL/TLS settings - make optional for easier testing
+ssl = optional
+ssl_cert = </etc/dovecot/ssl/server.crt
+ssl_key = </etc/dovecot/ssl/server.key
+
+# Authentication
+auth_mechanisms = plain login
+disable_plaintext_auth = no
+
+# User database
+passdb {
+  driver = passwd-file
+  args = scheme=plain username_format=%u /etc/dovecot/passwd
+}
+
+userdb {
+  driver = passwd-file
+  args = username_format=%u /etc/dovecot/users
+}
+
+# Mail location
+mail_location = maildir:/var/mail/%u
+
+# Mailbox settings
+namespace inbox {
+  type = private
+  separator = /
+  prefix = 
+  location = 
+  inbox = yes
+  hidden = no
+  list = yes
+  subscriptions = yes
+
+  mailbox Drafts {
+    special_use = \Drafts
+  }
+  mailbox Junk {
+    special_use = \Junk
+  }
+  mailbox Trash {
+    special_use = \Trash
+  }
+  mailbox Sent {
+    special_use = \Sent
+  }
+}
+
+# Services
+service imap-login {
+  inet_listener imap {
+    port = 143
+  }
+  inet_listener imaps {
+    port = 993
+    ssl = yes
+  }
+}
+
+service auth {
+  unix_listener auth-userdb {
+    mode = 0666
+  }
+}
+
+# Logging
+log_path = /dev/stdout
+info_log_path = /dev/stdout
+debug_log_path = /dev/stdout
+
+# Process limits
+default_process_limit = 10
+default_client_limit = 100
--- a/test/dovecot/entrypoint.sh
+++ b/test/dovecot/entrypoint.sh
@ -0,0 +1,51 @@
+#!/bin/sh
+
+# Entrypoint script for Dovecot test container
+
+set -e
+
+echo "Installing Dovecot..."
+apk add --no-cache dovecot dovecot-lmtpd openssl
+
+echo "Setting up directories..."
+mkdir -p /var/mail
+mkdir -p /var/run/dovecot
+mkdir -p /var/log/dovecot
+
+# Create dovecot user if it doesn't exist
+if ! getent passwd dovecot > /dev/null 2>&1; then
+    addgroup -g 97 dovecot
+    adduser -D -u 97 -G dovecot -s /sbin/nologin dovecot
+fi
+
+# Set proper ownership
+chown -R dovecot:dovecot /var/mail
+chown -R dovecot:dovecot /var/run/dovecot
+chown -R root:dovecot /etc/dovecot
+chmod -R 0640 /etc/dovecot
+chmod 0644 /etc/dovecot/dovecot.conf
+
+# Generate SSL certificates if they don't exist
+if [ ! -f /etc/dovecot/ssl/server.crt ] || [ ! -f /etc/dovecot/ssl/server.key ]; then
+    echo "Generating SSL certificates..."
+    mkdir -p /etc/dovecot/ssl
+    
+    # Generate DH parameters (small for testing)
+    openssl dhparam -out /etc/dovecot/ssl/dh.pem 1024
+    
+    # Generate private key
+    openssl genrsa -out /etc/dovecot/ssl/server.key 2048
+    
+    # Generate certificate
+    openssl req -new -key /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.csr -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
+    openssl x509 -req -days 365 -in /etc/dovecot/ssl/server.csr -signkey /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.crt
+    rm /etc/dovecot/ssl/server.csr
+fi
+
+# Ensure SSL directory permissions
+chown -R dovecot:dovecot /etc/dovecot/ssl
+chmod 600 /etc/dovecot/ssl/server.key
+chmod 644 /etc/dovecot/ssl/server.crt
+
+echo "Starting Dovecot..."
+exec dovecot -F
--- a/test/dovecot/passwd
+++ b/test/dovecot/passwd
@ -0,0 +1,8 @@
+# Password database for Dovecot testing
+# Format: username:password
+
+# Test accounts with simple passwords for testing
+testuser1:password123
+testuser2:password456
+syncuser:syncpass
+archiveuser:archivepass
--- a/test/dovecot/ssl/dh.pem
+++ b/test/dovecot/ssl/dh.pem
@ -0,0 +1,8 @@
+-----BEGIN DH PARAMETERS-----
+MIIBDAKCAQEAjcUSAHFs60qgDRg/cT7byhuhF3vwZQhmm1QToCFgG4VWu/EOVXq2
+kHxjxmo3hBuJCqUZqTAyF91Tum7A2QuQhXFrxOpRF8EiyVSgBabjN/WcEHIow1uh
+Vtb4JOcDl/Q9IJfFT6zyXdQQiHPBOWnpOBKXeQQQIx5plgsrmK0cTO2ZxtyrmHHp
+wxtE3INKYuBlGH3Y0zghc+Hoezpf/hbIHZibGQ0l79EtBDQjqmqoDJCIiv5gsTt8
+9VpkR6FFvjWTNOb5qY10W/PRhLGjioX29bp1B6qW5PNJcd//cqrBLebKlkAoXnyx
+x0uTUy6pmmIt5vdYxx0symrMXZEjrL7uzwIBAgICAOE=
+-----END DH PARAMETERS-----
--- a/test/dovecot/ssl/server.crt
+++ b/test/dovecot/ssl/server.crt
@ -0,0 +1,20 @@
+-----BEGIN CERTIFICATE-----
+MIIDVzCCAj+gAwIBAgIUYvkZSKbVH08s/3B70AW8IEpTB/kwDQYJKoZIhvcNAQEL
+BQAwVDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx
+EzARBgNVBAoMCk1haWwyQ291Y2gxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yNTA4
+MDExNDIyMzVaFw0yNjA4MDExNDIyMzVaMFQxCzAJBgNVBAYTAlVTMQ0wCwYDVQQI
+DARUZXN0MQ0wCwYDVQQHDARUZXN0MRMwEQYDVQQKDApNYWlsMkNvdWNoMRIwEAYD
+VQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs
+leQcVLlKx7/bJGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vx
+Td1mVCm9Lsd9bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDW
+knco7/MEatakliVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331
+HEFdIqqccpimnIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwo
+eN8zBzGFJe9xzqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5
+LGN5xSzXvFIzgP20y5s5AgMBAAGjITAfMB0GA1UdDgQWBBTkgYZGp2s74D+1ltyl
+rudF/o7jODANBgkqhkiG9w0BAQsFAAOCAQEATc6ekhuk32meLuxhalz6lNwBxfDg
+EG3gGUxNwehwgiNCcKIKQFtCwjJde6drOobkRDANtb7g3gSlAxlUCPsO6xnL1c6E
+HhehFn++7HOpXvmEy/mnoqBL6PLzRZRMRlDynlPVV9Y82zsdrQiQEhGyNTfgP5dk
+u9RMIMQl1hIK381V738b5MXfdpYhmRiTGEd6hCxCnzkx0OakCLM9lnJASr0dYPuh
+LYKoClxhr3sV/JsgAmx91BuHGpzaPYQ2zFvCJSqD2ihM7zIl9K2bLIUR87/CznyH
+JuPRbgt6/cxzwdqflP73j+TTZdlI4gckEA3H0WhNN4nB2SEjTgS+kDctMA==
+-----END CERTIFICATE-----
--- a/test/dovecot/ssl/server.key
+++ b/test/dovecot/ssl/server.key
@ -0,0 +1,28 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsleQcVLlKx7/b
+JGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vxTd1mVCm9Lsd9
+bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDWknco7/MEatak
+liVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331HEFdIqqccpim
+nIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwoeN8zBzGFJe9x
+zqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5LGN5xSzXvFIz
+gP20y5s5AgMBAAECggEAOfBaM76ns/iqKpYlamnjfIs7svotEjhrHcsub6fWvEsE
+XLzRmSqHeWP+t55oo2XeL2zOCofvuUDGnQ+rXE2mHwzhP3FJzsOibm2qmtCJvZwe
+ozRj4xTMLILGDnGRhHAJ21cxZM9lPNLnOzri0868DeYimib49xAdroLBLyKRgDGN
+O7OAwA4KWAebZRBU8cowEF87WGAI2hOLJA5WIKX7X9SiRDx5sNnDpTpqJvyvuleR
+D/wKGHzkQiZx8WNJx5A571dilfKEp8S49o4sJdz8DQ/4ruVov2Vi+nSdMEzHnp5m
+M7ZlZ7IcJRJDKPrDYasmxvtM8EiKyJf87/DPANfYFwKBgQDa+ANr2pGg2oMJVJIq
+r/mj8wtRsLRYFgfs96BUwp7e9Vo+Rx+E0uZxGGMul6hn4b2/TQljLfiX/CP8ZTyw
+MStlNlnAXaF425JNuQIFQ2wGGiGdsx2I1WNw7co4UPVVjH11nr3o30g+NDJcp7Tq
+rvpKvGbeoZtHzOj0bF7fA/B7qwKBgQDJxcUUEH5A50n3oUP99aK/DzSA7kcte2Aw
+tjv9hbPnbOmcM24Fm+KU7bsRYt9QPa0PU2lV3O1KrHj4q+QRcPFl2P2mzZC+Hzmx
+H8dEjMmH8YdrjGqethMoUHJCguNfskNwjgWFlxTSBLY+NffghXNzZgiF9d6WqF48
+iqwH+HsAqwKBgQCb/B2D0Xn4WnEKToKpgh6WGmcv1G9EaL1Qo75FYzcFoUaeItBj
+MFIUssjEwiinh/pBssFDM9Zpfqar//pRkVVWjnc1P/3tOI1qbKbx1Ou5FRhpXNVn
+SovCQMLTh2idfq1JAsJKh/TQyyItOxL4M5n9b2Tgp8MUTPaOWDzlJctEbQKBgEVu
+oNq+sjNzY6iq/dKubEqC2PZlCGlGQ1t/2jTrhXTlrZ3qtLmJYvcMt4rMEzxxfNQB
+SAYb+CvyHc60l87Ipsj9WovDwUMrS5b/8HpOWCtHmeoQb8Adt4nv5OGuWL/dgAeD
+V7MYwjljFbNiruG8CnZzbgtrCCWf2o3KylgT0X/xAoGAUhSdBge5Vpg0JcT1VDgm
+q5rgc6dD1LJtXfBaq3w4kHYK/iLFcPOLUKcIJXNbhMwWza/JwVYK6hsCIw3/b4va
+NhJ8ABpC3fZqkl28glEF8bnrPAkE1akn2GiBaaEbTCQRMrhZ2SW3JCyjX6yCvvvz
+m7b2ZpDMJEMIBmgrK70E3Oo=
+-----END PRIVATE KEY-----
--- a/test/dovecot/users
+++ b/test/dovecot/users
@ -0,0 +1,8 @@
+# User database for Dovecot testing
+# Format: username:uid:gid:home:shell
+
+# Test user accounts
+testuser1::1001:1001:/var/mail/testuser1::
+testuser2::1002:1002:/var/mail/testuser2::
+syncuser::1003:1003:/var/mail/syncuser::
+archiveuser::1004:1004:/var/mail/archiveuser::
--- a/test/generate-ssl.sh
+++ b/test/generate-ssl.sh
@ -0,0 +1,28 @@
+#!/bin/bash
+
+# Generate SSL certificates for Dovecot testing
+
+set -e
+
+CERT_DIR="dovecot/ssl"
+mkdir -p "$CERT_DIR"
+
+# Generate DH parameters
+echo "Generating DH parameters..."
+openssl dhparam -out "$CERT_DIR/dh.pem" 2048
+
+# Generate private key
+echo "Generating private key..."
+openssl genrsa -out "$CERT_DIR/server.key" 2048
+
+# Generate certificate signing request
+echo "Generating certificate..."
+openssl req -new -key "$CERT_DIR/server.key" -out "$CERT_DIR/server.csr" -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
+
+# Generate self-signed certificate
+openssl x509 -req -days 365 -in "$CERT_DIR/server.csr" -signkey "$CERT_DIR/server.key" -out "$CERT_DIR/server.crt"
+
+# Clean up CSR
+rm "$CERT_DIR/server.csr"
+
+echo "SSL certificates generated successfully in $CERT_DIR/"
--- a/test/podman-compose.yml
+++ b/test/podman-compose.yml
@ -0,0 +1,38 @@
+version: '3.8'
+
+services:
+  # CouchDB for testing
+  couchdb:
+    image: docker.io/couchdb:3.3
+    container_name: mail2couch_test_couchdb
+    environment:
+      - COUCHDB_USER=admin
+      - COUCHDB_PASSWORD=password
+    ports:
+      - "5984:5984"
+    volumes:
+      - couchdb_data:/opt/couchdb/data
+    networks:
+      - mail2couch_test
+
+  # GreenMail IMAP server for testing
+  greenmail:
+    image: docker.io/greenmail/standalone:2.0.1
+    container_name: mail2couch_test_imap
+    ports:
+      - "3143:3143"   # IMAP
+      - "3993:3993"   # IMAPS
+      - "3025:3025"   # SMTP
+    environment:
+      - GREENMAIL_OPTS=-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.users=testuser1:password123@localhost,testuser2:password456@localhost,syncuser:syncpass@localhost,archiveuser:archivepass@localhost
+    networks:
+      - mail2couch_test
+    depends_on:
+      - couchdb
+
+volumes:
+  couchdb_data:
+
+networks:
+  mail2couch_test:
+    driver: bridge
--- a/test/populate-greenmail.py
+++ b/test/populate-greenmail.py
@ -0,0 +1,129 @@
+#!/usr/bin/env python3
+
+import imaplib
+import email
+from email.mime.text import MIMEText
+from email.mime.multipart import MIMEMultipart
+from email.mime.base import MIMEBase
+from email import encoders
+import time
+import sys
+
+def create_simple_message(subject, body, from_addr="test-sender@example.com"):
+    """Create a simple text message"""
+    msg = MIMEText(body)
+    msg['Subject'] = subject
+    msg['From'] = from_addr
+    msg['Date'] = email.utils.formatdate(localtime=True)
+    return msg.as_string()
+
+def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com"):
+    """Create a message with an attachment"""
+    msg = MIMEMultipart()
+    msg['Subject'] = subject
+    msg['From'] = from_addr
+    msg['Date'] = email.utils.formatdate(localtime=True)
+    
+    # Add body
+    msg.attach(MIMEText(body, 'plain'))
+    
+    # Add attachment
+    part = MIMEBase('text', 'plain')
+    part.set_payload(attachment_content)
+    encoders.encode_base64(part)
+    part.add_header('Content-Disposition', 'attachment; filename="attachment.txt"')
+    msg.attach(part)
+    
+    return msg.as_string()
+
+def populate_user_mailbox(username, password, host='localhost', port=3143):
+    """Populate a user's mailbox with test messages"""
+    print(f"Connecting to {username}@{host}:{port}")
+    
+    try:
+        # Connect to IMAP server
+        imap = imaplib.IMAP4(host, port)
+        imap.login(username, password)
+        imap.select('INBOX')
+        
+        print(f"Creating messages for {username}...")
+        
+        # Create 10 regular messages
+        for i in range(1, 11):
+            if i % 3 == 0:
+                # Every 3rd message has attachment
+                msg = create_message_with_attachment(
+                    f"Test Message {i} with Attachment",
+                    f"This is test message {i} for {username} with an attachment.",
+                    f"Sample attachment content for message {i}"
+                )
+                print(f"  Created message {i} (with attachment)")
+            else:
+                # Simple message
+                msg = create_simple_message(
+                    f"Test Message {i}",
+                    f"This is test message {i} for {username}."
+                )
+                print(f"  Created message {i}")
+            
+            # Append message to INBOX
+            imap.append('INBOX', None, None, msg.encode('utf-8'))
+            time.sleep(0.1)  # Small delay to avoid overwhelming
+        
+        # Create additional test messages
+        special_messages = [
+            ("Important Message", "This is an important message for testing sync/archive modes."),
+            ("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀")
+        ]
+        
+        for subject, body in special_messages:
+            msg = create_simple_message(subject, body)
+            imap.append('INBOX', None, None, msg.encode('utf-8'))
+            print(f"  Created special message: {subject}")
+        
+        imap.logout()
+        print(f"✅ Successfully created 12 messages for {username}")
+        return True
+        
+    except Exception as e:
+        print(f"❌ Error populating {username}: {e}")
+        return False
+
+def main():
+    print("🚀 Populating GreenMail with test messages using IMAP...")
+    
+    # Test accounts
+    accounts = [
+        ("testuser1", "password123"),
+        ("testuser2", "password456"), 
+        ("syncuser", "syncpass"),
+        ("archiveuser", "archivepass")
+    ]
+    
+    # Wait for GreenMail to be ready
+    print("Waiting for GreenMail to be ready...")
+    time.sleep(5)
+    
+    success_count = 0
+    for username, password in accounts:
+        if populate_user_mailbox(username, password):
+            success_count += 1
+        time.sleep(1)  # Brief pause between accounts
+    
+    print(f"\n🎉 Successfully populated {success_count}/{len(accounts)} accounts!")
+    
+    if success_count == len(accounts):
+        print("\n✅ All test accounts ready:")
+        for username, password in accounts:
+            print(f"  - {username}:{password}@localhost")
+        print(f"\nGreenMail Services:")
+        print(f"  - IMAP: localhost:3143")
+        print(f"  - IMAPS: localhost:3993") 
+        print(f"  - SMTP: localhost:3025")
+        return 0
+    else:
+        print(f"\n❌ Failed to populate {len(accounts) - success_count} accounts")
+        return 1
+
+if __name__ == "__main__":
+    sys.exit(main())
--- a/test/populate-test-messages.sh
+++ b/test/populate-test-messages.sh
@ -0,0 +1,18 @@
+#!/bin/bash
+
+# Populate GreenMail test server with sample messages using Python script
+
+set -e
+
+cd "$(dirname "$0")"
+
+echo "Populating GreenMail with test messages..."
+
+# Check if Python 3 is available
+if ! command -v python3 &> /dev/null; then
+    echo "❌ Python 3 is required but not installed"
+    exit 1
+fi
+
+# Run the Python script to populate messages
+python3 ./populate-greenmail.py
--- a/test/run-tests.sh
+++ b/test/run-tests.sh
@ -0,0 +1,128 @@
+#!/bin/bash
+
+# Run integration tests with test containers
+
+set -e
+
+cd "$(dirname "$0")"
+
+echo "🚀 Starting mail2couch integration tests..."
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Function to print colored output
+print_status() {
+    echo -e "${GREEN}[INFO]${NC} $1"
+}
+
+print_warning() {
+    echo -e "${YELLOW}[WARN]${NC} $1"
+}
+
+print_error() {
+    echo -e "${RED}[ERROR]${NC} $1"
+}
+
+# Cleanup function
+cleanup() {
+    print_status "Cleaning up test containers..."
+    podman-compose -f podman-compose.yml down -v 2>/dev/null || true
+}
+
+# Set up cleanup trap
+trap cleanup EXIT
+
+# Start containers
+print_status "Starting test containers..."
+podman-compose -f podman-compose.yml up -d
+
+# Wait for containers to be ready
+print_status "Waiting for containers to be ready..."
+sleep 10
+
+# Check if CouchDB is ready
+print_status "Checking CouchDB connectivity..."
+timeout=30
+while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
+    timeout=$((timeout - 1))
+    if [ $timeout -le 0 ]; then
+        print_error "CouchDB failed to start within 30 seconds"
+        exit 1
+    fi
+    sleep 1
+done
+print_status "CouchDB is ready!"
+
+# Check if IMAP server is ready
+print_status "Checking IMAP server connectivity..."
+timeout=30
+while ! nc -z localhost 3143 > /dev/null 2>&1; do
+    timeout=$((timeout - 1))
+    if [ $timeout -le 0 ]; then
+        print_error "IMAP server failed to start within 30 seconds"
+        exit 1
+    fi
+    sleep 1
+done
+print_status "IMAP server is ready!"
+
+# Populate test messages
+print_status "Populating test messages..."
+./populate-test-messages.sh
+
+# Build mail2couch
+print_status "Building mail2couch..."
+cd ../go
+go build -o mail2couch .
+cd ../test
+
+# Run mail2couch with test configuration
+print_status "Running mail2couch with test configuration..."
+../go/mail2couch -config config-test.json
+
+# Verify results
+print_status "Verifying test results..."
+
+# Check CouchDB databases were created
+EXPECTED_DBS=("test_user_1" "test_sync_user" "test_archive_user")
+
+for db in "${EXPECTED_DBS[@]}"; do
+    if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then
+        print_status "✅ Database '$db' created successfully"
+    else
+        print_error "❌ Database '$db' was not created"
+        exit 1
+    fi
+done
+
+# Check document counts
+for db in "${EXPECTED_DBS[@]}"; do
+    doc_count=$(curl -s "http://admin:password@localhost:5984/$db" | grep -o '"doc_count":[0-9]*' | cut -d':' -f2)
+    if [ "$doc_count" -gt 0 ]; then
+        print_status "✅ Database '$db' contains $doc_count documents"
+    else
+        print_warning "⚠️ Database '$db' contains no documents"
+    fi
+done
+
+# Test sync mode by running again (should show removed documents if any)
+print_status "Running mail2couch again to test sync behavior..."
+../go/mail2couch -config config-test.json
+
+print_status "🎉 All tests completed successfully!"
+
+# Show summary
+print_status "Test Summary:"
+echo "  - IMAP Server: localhost:143"
+echo "  - CouchDB: http://localhost:5984"
+echo "  - Test accounts: testuser1, syncuser, archiveuser"
+echo "  - Databases created: ${EXPECTED_DBS[*]}"
+echo ""
+echo "You can now:"
+echo "  - Access CouchDB at http://localhost:5984/_utils"
+echo "  - Connect to IMAP at localhost:143"
+echo "  - Run manual tests with: ../go/mail2couch -config config-test.json"
--- a/test/start-test-env.sh
+++ b/test/start-test-env.sh
@ -0,0 +1,67 @@
+#!/bin/bash
+
+# Start test environment for manual testing
+
+cd "$(dirname "$0")"
+
+echo "🚀 Starting mail2couch test environment..."
+
+# Start containers
+echo "Starting containers..."
+podman-compose -f podman-compose.yml up -d
+
+# Wait for services
+echo "Waiting for services to be ready..."
+sleep 10
+
+# Check CouchDB
+echo "Checking CouchDB..."
+timeout=30
+while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
+    timeout=$((timeout - 1))
+    if [ $timeout -le 0 ]; then
+        echo "❌ CouchDB failed to start"
+        exit 1
+    fi
+    sleep 1
+done
+echo "✅ CouchDB is ready at http://localhost:5984"
+
+# Check IMAP
+echo "Checking IMAP server..."
+timeout=30
+while ! nc -z localhost 3143 > /dev/null 2>&1; do
+    timeout=$((timeout - 1)) 
+    if [ $timeout -le 0 ]; then
+        echo "❌ IMAP server failed to start"
+        exit 1
+    fi
+    sleep 1
+done
+echo "✅ IMAP server is ready at localhost:3143"
+
+# Populate test data
+echo "Populating test messages..."
+./populate-test-messages.sh
+
+echo ""
+echo "🎉 Test environment is ready!"
+echo ""
+echo "Services:"
+echo "  - CouchDB: http://localhost:5984 (admin/password)"
+echo "  - CouchDB Web UI: http://localhost:5984/_utils"
+echo "  - IMAP Server: localhost:3143"
+echo "  - IMAPS Server: localhost:3993"
+echo "  - SMTP Server: localhost:3025"
+echo ""
+echo "Test accounts:"
+echo "  - testuser1:password123"
+echo "  - testuser2:password456"
+echo "  - syncuser:syncpass"
+echo "  - archiveuser:archivepass"
+echo ""
+echo "To run mail2couch:"
+echo "  cd ../go && ./mail2couch -config ../test/config-test.json"
+echo ""
+echo "To stop the environment:"
+echo "  ./stop-test-env.sh"
--- a/test/stop-test-env.sh
+++ b/test/stop-test-env.sh
@ -0,0 +1,12 @@
+#!/bin/bash
+
+# Stop test environment
+
+cd "$(dirname "$0")"
+
+echo "🛑 Stopping mail2couch test environment..."
+
+# Stop and remove containers
+podman-compose -f podman-compose.yml down -v
+
+echo "✅ Test environment stopped and cleaned up!"