mail2couch/test/README.md

# mail2couch Test Environment

This directory contains a complete test environment for mail2couch using Podman containers.

## Overview

The test environment provides:
- **CouchDB**: Database for storing email messages
- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages
- **Test Configuration**: Ready-to-use config for testing both sync and archive modes
- **HTML Webmail Interface**: Beautiful, responsive web interface for viewing archived emails

## Quick Start

### Run Basic Integration Tests
```bash
./run-tests.sh
```
This comprehensive test will:
1. Start all containers with cleanup
2. Populate test data  
3. Build and run mail2couch
4. Verify database creation and document storage
5. Test incremental sync behavior
6. Clean up automatically

### Run Wildcard Pattern Tests
```bash
./test-wildcard-patterns.sh
```
This will test various wildcard folder patterns including:
- `*` (all folders)
- `Work*` (prefix patterns)
- `*/Drafts` (subfolder patterns)
- Complex include/exclude combinations

### Run Incremental Sync Tests
```bash
./test-incremental-sync.sh
```
This will test incremental synchronization functionality:
- First sync establishes baseline
- New messages are added to test accounts
- Second sync should only fetch new messages
- Sync metadata tracking and IMAP SEARCH with SINCE

### Manual Testing Environment
```bash
# Start persistent test environment (for manual experimentation)
./start-test-env.sh

# Run mail2couch manually with different configurations
cd ../go
./mail2couch -config ../test/config-test.json
./mail2couch -config ../test/config-wildcard-examples.json

# Stop test environment when done
cd ../test
./stop-test-env.sh
```

## Test Scripts Overview

### Automated Testing (Recommended)
- **`./run-tests.sh`**: Complete integration test with automatic cleanup
  - Starts containers, populates data, runs mail2couch, verifies results
  - Tests basic functionality, database creation, and incremental sync
  - Cleans up automatically - perfect for CI/CD or quick validation

### Specialized Feature Testing  
- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing
  - Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns
  - Self-contained with own setup/teardown
- **`./test-incremental-sync.sh`**: Incremental synchronization testing
  - Tests sync metadata tracking and IMAP SEARCH with SINCE
  - Multi-step validation: baseline sync → add messages → incremental sync
  - Self-contained with own setup/teardown

### Manual Testing Environment
- **`./start-test-env.sh`**: Start persistent test containers
  - Keeps environment running for manual experimentation
  - Populates test data once
  - Use with different configurations for development
- **`./stop-test-env.sh`**: Clean up manual test environment
  - Only needed after using `start-test-env.sh`

## Test Accounts

The test environment includes these IMAP accounts:

| Username | Password | Mode | Folder Pattern | Purpose |
|----------|----------|------|---------------|---------|
| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test |
| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test |
| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test |
| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test |

Each account contains:
- 10 messages in INBOX (every 3rd has an attachment)
- 3 messages in each additional folder
- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash`
- Various message types for comprehensive wildcard testing

## Services

### CouchDB
- **URL**: http://localhost:5984
- **Admin**: `admin` / `password`
- **Web UI**: http://localhost:5984/_utils

### GreenMail Server
- **Host**: localhost
- **IMAP Port**: 3143 (plain)
- **IMAPS Port**: 3993 (SSL)
- **SMTP Port**: 3025
- **Server**: GreenMail (Java-based test server)

### Accessing Test Data
After running mail2couch, you can access the stored emails via CouchDB's REST API:

**📋 Database Access:**
- All databases: http://localhost:5984/_all_dbs
- Specific database: http://localhost:5984/m2c_specific_folders_only
- All documents: http://localhost:5984/m2c_specific_folders_only/_all_docs
- Individual message: http://localhost:5984/m2c_specific_folders_only/INBOX_12

**🔍 Raw Data Examples:**
- Database info: http://localhost:5984/m2c_specific_folders_only
- Document content: http://localhost:5984/m2c_specific_folders_only/INBOX_1
- Email attachments: http://localhost:5984/m2c_specific_folders_only/INBOX_1/{attachment_name}

## Database Structure

mail2couch will create separate databases for each mail source (with `m2c_` prefix):
- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
- `m2c_work_pattern_test` - Work Pattern Test (sync mode)  
- `m2c_specific_folders_only` - Specific Folders Only (archive mode)
- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode)

Each database contains documents with:
- `mailbox` field indicating the origin folder
- Native CouchDB attachments for email attachments
- Full message headers and body content
- JSON documents accessible via CouchDB REST API

## Testing Sync vs Archive Modes

### Sync Mode (`syncuser`)
- Database exactly matches mail account
- If messages are deleted from IMAP, they're removed from CouchDB
- 1-to-1 relationship

### Archive Mode (`archiveuser`, `testuser1`)
- Database preserves all messages ever seen
- Messages deleted from IMAP remain in CouchDB
- Archive/backup behavior

## Wildcard Pattern Examples

The test environment demonstrates these wildcard patterns:

### All Folders Pattern (`*`)
```json
{
  "folderFilter": {
    "include": ["*"],
    "exclude": ["Drafts", "Trash"]
  }
}
```
Processes all folders except Drafts and Trash.

### Work Pattern (`Work*`)
```json
{
  "folderFilter": {
    "include": ["Work*", "Important*", "INBOX"],
    "exclude": ["*Temp*"]
  }
}
```
Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp.

### Specific Folders
```json
{
  "folderFilter": {
    "include": ["INBOX", "Sent", "Personal"],
    "exclude": []
  }
}
```
Only processes the exact named folders.

### Subfolder Pattern (`Work/*`)
```json
{
  "folderFilter": {
    "include": ["Work/*", "Archive/*"],
    "exclude": ["*/Drafts"]
  }
}
```
Includes all subfolders under Work and Archive, but excludes any Drafts subfolder.

## File Structure

```
test/
├── podman-compose.yml            # Container orchestration (GreenMail + CouchDB)
├── config-test.json              # Main test configuration with wildcard examples
├── config-wildcard-examples.json # Advanced wildcard patterns
├── run-tests.sh                  # Automated integration test (recommended)
├── test-wildcard-patterns.sh     # Specialized wildcard pattern testing
├── test-incremental-sync.sh      # Specialized incremental sync testing
├── start-test-env.sh             # Start persistent test environment
├── stop-test-env.sh              # Stop test environment
├── populate-greenmail.py         # Create test messages across multiple folders
├── dovecot/                      # Dovecot configuration (legacy, unused)
└── README.md                     # This file
```

## Prerequisites

- Podman and podman-compose
- OpenSSL (for certificate generation)
- curl and nc (for connectivity checks)
- Go (for building mail2couch)

## Troubleshooting

### Containers won't start
```bash
# Check podman status
podman ps -a

# View logs
podman logs mail2couch_test_couchdb
podman logs mail2couch_test_imap
```

### CouchDB connection issues
- Verify CouchDB is running: `curl http://localhost:5984`
- Check admin credentials: `admin/password`

### IMAP connection issues
- Test IMAP connection: `nc -z localhost 143`
- Check Dovecot logs: `podman logs mail2couch_test_imap`

### Permission issues
- Ensure scripts are executable: `chmod +x *.sh`
- Check file permissions in dovecot directory

## Advanced Usage

### Add custom test messages
Edit `populate-test-messages.sh` to create additional test scenarios.

### Modify IMAP configuration
Edit `dovecot/dovecot.conf` and restart containers.

### Test with SSL
Update `config-test.json` to use port 993 and enable SSL.

### Custom test scenarios
Create additional configuration files for specific test cases.