olemd/mail2couch
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

feat: add comprehensive README documentation and clean up configuration

Browse source 
## Documentation Enhancements
- Create comprehensive README with installation, configuration, and usage examples
- Add simple, advanced, and provider-specific configuration examples
- Document all features: incremental sync, wildcard patterns, keyword filtering, attachment support
- Include production deployment guidance and troubleshooting section
- Add architecture documentation with database structure and document format examples

## Configuration Cleanup
- Remove unnecessary `database` field from CouchDB configuration
- Add `m2c_` prefix to all CouchDB database names for better namespace isolation
- Update GenerateAccountDBName() to consistently prefix databases with `m2c_`
- Clean up all configuration examples to remove deprecated database field

## Test Environment Simplification
- Simplify test script structure to eliminate confusion and redundancy
- Remove redundant populate-test-messages.sh wrapper script
- Update run-tests.sh to be comprehensive automated test with cleanup
- Maintain clear separation: automated tests vs manual testing environment
- Update all test scripts to expect m2c-prefixed database names

## Configuration Examples Added
- config-simple.json: Basic single Gmail account setup
- config-advanced.json: Multi-account with complex filtering and different providers
- config-providers.json: Real-world configurations for Gmail, Outlook, Yahoo, iCloud

## Benefits
- Clear documentation for users from beginner to advanced
- Namespace isolation prevents database conflicts in shared CouchDB instances
- Simplified test workflow eliminates user confusion about which scripts to use
- Comprehensive examples cover common email provider configurations

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ole-Morten Duesund 2025-08-01 21:26:53 +02:00
commit c2ad55eaaf
17 changed files with 1139 additions and 111 deletions
Download patch file Download diff file Expand all files Collapse all files

87
test/README.md
View file

@ -11,16 +11,17 @@ The test environment provides:
## Quick Start
### Run Full Integration Tests
### Run Basic Integration Tests
```bash
./run-tests.sh
```
This will:
1. Start all containers
2. Populate test data
3. Run mail2couch
4. Verify results
5. Clean up
This comprehensive test will:
1. Start all containers with cleanup
2. Populate test data
3. Build and run mail2couch
4. Verify database creation and document storage
5. Test incremental sync behavior
6. Clean up automatically
### Run Wildcard Pattern Tests
```bash
@ -32,20 +33,56 @@ This will test various wildcard folder patterns including:
- `*/Drafts` (subfolder patterns)
- Complex include/exclude combinations
### Manual Testing
### Run Incremental Sync Tests
```bash
# Start test environment
./test-incremental-sync.sh
```
This will test incremental synchronization functionality:
- First sync establishes baseline
- New messages are added to test accounts
- Second sync should only fetch new messages
- Sync metadata tracking and IMAP SEARCH with SINCE
### Manual Testing Environment
```bash
# Start persistent test environment (for manual experimentation)
./start-test-env.sh
# Run mail2couch manually
# Run mail2couch manually with different configurations
cd ../go
./mail2couch -config ../test/config-test.json
./mail2couch -config ../test/config-wildcard-examples.json
# Stop test environment when done
cd ../test
./stop-test-env.sh
```
## Test Scripts Overview
### Automated Testing (Recommended)
- **`./run-tests.sh`**: Complete integration test with automatic cleanup
- Starts containers, populates data, runs mail2couch, verifies results
- Tests basic functionality, database creation, and incremental sync
- Cleans up automatically - perfect for CI/CD or quick validation
### Specialized Feature Testing
- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing
- Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns
- Self-contained with own setup/teardown
- **`./test-incremental-sync.sh`**: Incremental synchronization testing
- Tests sync metadata tracking and IMAP SEARCH with SINCE
- Multi-step validation: baseline sync → add messages → incremental sync
- Self-contained with own setup/teardown
### Manual Testing Environment
- **`./start-test-env.sh`**: Start persistent test containers
- Keeps environment running for manual experimentation
- Populates test data once
- Use with different configurations for development
- **`./stop-test-env.sh`**: Clean up manual test environment
- Only needed after using `start-test-env.sh`
## Test Accounts
The test environment includes these IMAP accounts:
@ -79,11 +116,11 @@ Each account contains:
## Database Structure
mail2couch will create separate databases for each mail source:
- `wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
- `work_pattern_test` - Work Pattern Test (sync mode)
- `specific_folders_only` - Specific Folders Only (archive mode)
- `subfolder_pattern_test` - Subfolder Pattern Test (archive mode)
mail2couch will create separate databases for each mail source (with `m2c_` prefix):
- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
- `m2c_work_pattern_test` - Work Pattern Test (sync mode)
- `m2c_specific_folders_only` - Specific Folders Only (archive mode)
- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode)
Each database contains documents with:
- `mailbox` field indicating the origin folder
@ -154,17 +191,17 @@ Includes all subfolders under Work and Archive, but excludes any Drafts subfolde
```
test/
├── podman-compose.yml # Container orchestration
├── config-test.json # Main test configuration with wildcard examples
├── podman-compose.yml # Container orchestration (GreenMail + CouchDB)
├── config-test.json # Main test configuration with wildcard examples
├── config-wildcard-examples.json # Advanced wildcard patterns
├── test-wildcard-patterns.sh # Wildcard pattern testing script
├── run-tests.sh # Full integration test
├── start-test-env.sh # Start environment
├── stop-test-env.sh # Stop environment
├── populate-greenmail.py # Create test messages with folders
├── populate-test-messages.sh # Wrapper script
├── dovecot/ # Dovecot configuration (legacy)
└── README.md # This file
├── run-tests.sh # Automated integration test (recommended)
├── test-wildcard-patterns.sh # Specialized wildcard pattern testing
├── test-incremental-sync.sh # Specialized incremental sync testing
├── start-test-env.sh # Start persistent test environment
├── stop-test-env.sh # Stop test environment
├── populate-greenmail.py # Create test messages across multiple folders
├── dovecot/ # Dovecot configuration (legacy, unused)
└── README.md # This file
```
## Prerequisites

3
test/config-test.json
View file

@ -2,8 +2,7 @@
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password",
"database": "mail_backup_test"
"password": "password"
},
"mailSources": [
{

3
test/config-wildcard-examples.json
View file

@ -2,8 +2,7 @@
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password",
"database": "mail_backup_test"
"password": "password"
},
"mailSources": [
{

18
test/populate-test-messages.sh
View file

@ -1,18 +0,0 @@
#!/bin/bash
# Populate GreenMail test server with sample messages using Python script
set -e
cd "$(dirname "$0")"
echo "Populating GreenMail with test messages..."
# Check if Python 3 is available
if ! command -v python3 &> /dev/null; then
echo "❌ Python 3 is required but not installed"
exit 1
fi
# Run the Python script to populate messages
python3 ./populate-greenmail.py

30
test/run-tests.sh
View file

@ -1,12 +1,13 @@
#!/bin/bash
# Run integration tests with test containers
# Run basic integration tests with test containers
# This is a comprehensive test that handles its own setup and teardown
set -e
cd "$(dirname "$0")"
echo "🚀 Starting mail2couch integration tests..."
echo "🚀 Running basic integration tests..."
# Colors for output
RED='\033[0;31m'
@ -72,7 +73,7 @@ print_status "IMAP server is ready!"
# Populate test messages
print_status "Populating test messages..."
./populate-test-messages.sh
python3 ./populate-greenmail.py
# Build mail2couch
print_status "Building mail2couch..."
@ -82,13 +83,13 @@ cd ../test
# Run mail2couch with test configuration
print_status "Running mail2couch with test configuration..."
../go/mail2couch -config config-test.json
../go/mail2couch -config config-test.json -max-messages 3
# Verify results
print_status "Verifying test results..."
# Check CouchDB databases were created
EXPECTED_DBS=("test_user_1" "test_sync_user" "test_archive_user")
# Check CouchDB databases were created (using correct database names with m2c prefix)
EXPECTED_DBS=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
for db in "${EXPECTED_DBS[@]}"; do
if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then
@ -109,20 +110,19 @@ for db in "${EXPECTED_DBS[@]}"; do
fi
done
# Test sync mode by running again (should show removed documents if any)
print_status "Running mail2couch again to test sync behavior..."
../go/mail2couch -config config-test.json
# Test sync mode by running again (should show incremental behavior)
print_status "Running mail2couch again to test incremental sync..."
../go/mail2couch -config config-test.json -max-messages 3
print_status "🎉 All tests completed successfully!"
print_status "🎉 Basic integration tests completed successfully!"
# Show summary
print_status "Test Summary:"
echo " - IMAP Server: localhost:143"
echo " - IMAP Server: localhost:3143"
echo " - CouchDB: http://localhost:5984"
echo " - Test accounts: testuser1, syncuser, archiveuser"
echo " - Databases created: ${EXPECTED_DBS[*]}"
echo ""
echo "You can now:"
echo " - Access CouchDB at http://localhost:5984/_utils"
echo " - Connect to IMAP at localhost:143"
echo " - Run manual tests with: ../go/mail2couch -config config-test.json"
echo "For more comprehensive tests, run:"
echo " - ./test-wildcard-patterns.sh (test folder pattern matching)"
echo " - ./test-incremental-sync.sh (test incremental synchronization)"

2
test/start-test-env.sh
View file

@ -42,7 +42,7 @@ echo "✅ IMAP server is ready at localhost:3143"
# Populate test data
echo "Populating test messages..."
./populate-test-messages.sh
python3 ./populate-greenmail.py
echo ""
echo "🎉 Test environment is ready!"

242
test/test-incremental-sync.sh Executable file
View file

@ -0,0 +1,242 @@
#!/bin/bash
# Test script to validate incremental sync functionality
# This script tests that mail2couch properly implements incremental synchronization
set -e
echo "🔄 Testing Incremental Sync Functionality"
echo "=========================================="
# Make sure we're in the right directory
cd "$(dirname "$0")/.."
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to check if containers are running
check_containers() {
echo "🔍 Checking if test containers are running..."
if ! podman ps | grep -q "greenmail"; then
echo -e "${RED}❌ GreenMail container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
if ! podman ps | grep -q "couchdb"; then
echo -e "${RED}❌ CouchDB container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
echo -e "${GREEN}✅ Test containers are running${NC}"
}
# Function to populate initial test data
populate_initial_data() {
echo "📧 Populating initial test data..."
cd test
if python3 populate-greenmail.py; then
echo -e "${GREEN}✅ Initial test data populated${NC}"
else
echo -e "${RED}❌ Failed to populate initial test data${NC}"
exit 1
fi
cd ..
}
# Function to build the application
build_app() {
echo "🔨 Building mail2couch..."
cd go
if go build -o mail2couch .; then
echo -e "${GREEN}✅ Build successful${NC}"
else
echo -e "${RED}❌ Build failed${NC}"
exit 1
fi
cd ..
}
# Function to run first sync
run_first_sync() {
echo -e "\n${BLUE}Running first sync...${NC}"
cd go
./mail2couch -config ../test/config-test.json -max-messages 5
cd ..
}
# Function to add new messages to test incremental sync
add_new_messages() {
echo -e "\n${YELLOW}Adding new messages for incremental sync test...${NC}"
# Create a simple Python script to add messages directly to GreenMail
cat > test/add_incremental_messages.py << 'EOF'
#!/usr/bin/env python3
import imaplib
import time
from test.populate_greenmail import create_simple_message
def add_new_messages():
"""Add new messages to test incremental sync"""
accounts = [
("testuser1", "password123"),
("syncuser", "syncpass"),
("archiveuser", "archivepass")
]
for username, password in accounts:
try:
print(f"Adding new messages to {username}...")
imap = imaplib.IMAP4('localhost', 3143)
imap.login(username, password)
imap.select('INBOX')
# Add 3 new messages with timestamps after the first sync
for i in range(1, 4):
subject = f"Incremental Sync Test Message {i}"
body = f"This message was added after the first sync for incremental testing. Message {i} for {username}."
msg = create_simple_message(subject, body, f"incremental-test@example.com", f"{username}@example.com")
imap.append('INBOX', None, None, msg.encode('utf-8'))
print(f" Added: {subject}")
time.sleep(0.1)
imap.logout()
print(f"✅ Added 3 new messages to {username}")
except Exception as e:
print(f"❌ Error adding messages to {username}: {e}")
if __name__ == "__main__":
add_new_messages()
EOF
# Add the parent directory to Python path and run the script
cd test
PYTHONPATH=.. python3 add_incremental_messages.py
cd ..
}
# Function to run second sync (incremental)
run_incremental_sync() {
echo -e "\n${BLUE}Running incremental sync...${NC}"
cd go
./mail2couch -config ../test/config-test.json -max-messages 10
cd ..
}
# Function to verify incremental sync results
verify_results() {
echo -e "\n${YELLOW}Verifying incremental sync results...${NC}"
# Check CouchDB for sync metadata documents
echo "Checking for sync metadata in CouchDB databases..."
# List of expected databases based on test config (with m2c prefix)
databases=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
for db in "${databases[@]}"; do
echo " Checking database: $db"
# Check if database exists
if curl -s -f "http://admin:password@localhost:5984/$db" > /dev/null; then
echo " ✅ Database exists"
# Look for sync metadata documents
metadata_docs=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"" | grep -o '"total_rows":[0-9]*' | cut -d: -f2 || echo "0")
if [ "$metadata_docs" -gt 0 ]; then
echo " ✅ Found sync metadata documents: $metadata_docs"
# Get a sample sync metadata document
sample_doc=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"&include_docs=true&limit=1")
echo " Sample sync metadata:"
echo "$sample_doc" | python3 -m json.tool | grep -E "(lastSyncTime|lastMessageUID|messageCount)" | head -3
else
echo " ⚠️ No sync metadata documents found"
fi
else
echo " ❌ Database does not exist"
fi
done
}
# Main test execution
main() {
echo "Starting incremental sync tests..."
# Pre-test setup
check_containers
build_app
# Clean up any existing data
echo "🧹 Cleaning up existing test data..."
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_wildcard_all_folders_test" > /dev/null || true
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_work_pattern_test" > /dev/null || true
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_specific_folders_only" > /dev/null || true
# Step 1: Populate initial test data
populate_initial_data
# Wait for data to settle
echo "⏳ Waiting for initial data to settle..."
sleep 5
# Step 2: Run first sync to establish baseline
echo -e "\n${YELLOW}=== STEP 1: First Sync (Baseline) ===${NC}"
run_first_sync
# Wait between syncs
echo "⏳ Waiting between syncs..."
sleep 3
# Step 3: Add new messages for incremental sync test
echo -e "\n${YELLOW}=== STEP 2: Add New Messages ===${NC}"
add_new_messages
# Wait for new messages to be ready
echo "⏳ Waiting for new messages to be ready..."
sleep 2
# Step 4: Run incremental sync
echo -e "\n${YELLOW}=== STEP 3: Incremental Sync ===${NC}"
run_incremental_sync
# Step 5: Verify results
echo -e "\n${YELLOW}=== STEP 4: Verification ===${NC}"
verify_results
echo -e "\n${GREEN}🎉 Incremental sync test completed!${NC}"
echo ""
echo "Key features tested:"
echo " ✅ Sync metadata storage and retrieval"
echo " ✅ IMAP SEARCH with SINCE for efficient incremental fetching"
echo " ✅ Last sync timestamp tracking per mailbox"
echo " ✅ Proper handling of first sync vs incremental sync"
echo ""
echo "To verify results manually:"
echo " - Check CouchDB: http://localhost:5984/_utils"
echo " - Look for 'sync_metadata_*' documents in each database"
echo " - Verify incremental messages were added after baseline sync"
}
# Cleanup function
cleanup() {
echo "🧹 Cleaning up test artifacts..."
rm -f test/add_incremental_messages.py
}
# Set trap to cleanup on exit
trap cleanup EXIT
# Run main function if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi