olemd/mail2couch
1
0
Fork 0
Code Issues Pull requests Projects Releases 1 Packages Wiki Activity Actions

feat: implement comprehensive wildcard folder selection and keyword filtering

Browse source 
## Wildcard Folder Selection
- Add support for wildcard patterns (`*`, `?`, `[abc]`) using filepath.Match
- Implement special case: `"*"` selects ALL available folders
- Support for complex include/exclude pattern combinations
- Maintain backwards compatibility with exact string matching
- Enable subfolder pattern matching (e.g., `Work/*`, `*/Drafts`)

## Keyword Filtering
- Add SubjectKeywords, SenderKeywords, RecipientKeywords to MessageFilter config
- Implement case-insensitive keyword matching across message fields
- Support multiple keywords per filter type with inclusive OR logic
- Add ShouldProcessMessage method for message-level filtering

## Enhanced Test Environment
- Create comprehensive wildcard pattern test scenarios
- Add 12 test folders covering various pattern types: Work/*, Important/*, Archive/*, exact matches
- Implement dedicated wildcard test script (test-wildcard-patterns.sh)
- Update test configurations to demonstrate real-world wildcard usage patterns
- Enhance test data generation with folder-specific messages for validation

## Documentation
- Create FOLDER_PATTERNS.md with comprehensive wildcard examples and use cases
- Update CLAUDE.md to reflect all implemented features and current status
- Enhance test README with detailed wildcard pattern explanations
- Provide configuration examples for common email organization scenarios

## Message Origin Tracking
- Verify all messages in CouchDB properly tagged with origin folder in `mailbox` field
- Maintain per-account database isolation for better organization
- Document ID format: `{folder}_{uid}` ensures uniqueness across folders

Key patterns supported:
- `["*"]` - All folders (with excludes)
- `["Work*", "Important*"]` - Prefix matching
- `["Work/*", "Archive/*"]` - Subfolder patterns
- `["INBOX", "Sent"]` - Exact matches
- Complex include/exclude combinations

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
Ole-Morten Duesund 2025-08-01 17:24:02 +02:00
commit 357cd06264
10 changed files with 602 additions and 84 deletions
Download patch file Download diff file Expand all files Collapse all files

101
test/README.md
View file

@ -22,6 +22,16 @@ This will:
4. Verify results
5. Clean up
### Run Wildcard Pattern Tests
```bash
./test-wildcard-patterns.sh
```
This will test various wildcard folder patterns including:
- `*` (all folders)
- `Work*` (prefix patterns)
- `*/Drafts` (subfolder patterns)
- Complex include/exclude combinations
### Manual Testing
```bash
# Start test environment
@ -40,17 +50,18 @@ cd ../test
The test environment includes these IMAP accounts:
| Username | Password | Mode | Purpose |
|----------|----------|------|---------|
| `testuser1` | `password123` | archive | General archive testing |
| `testuser2` | `password456` | - | Additional test user |
| `syncuser` | `syncpass` | sync | Testing sync mode (1-to-1) |
| `archiveuser` | `archivepass` | archive | Testing archive mode |
| Username | Password | Mode | Folder Pattern | Purpose |
|----------|----------|------|---------------|---------|
| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test |
| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test |
| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test |
| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test |
Each account contains:
- 10 messages in INBOX (every 3rd has an attachment)
- 3 messages in Sent folder
- Various message types for comprehensive testing
- 3 messages in each additional folder
- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash`
- Various message types for comprehensive wildcard testing
## Services
@ -69,9 +80,15 @@ Each account contains:
## Database Structure
mail2couch will create separate databases for each mail source:
- `test_user_1` - Test User 1 (archive mode)
- `test_sync_user` - Test Sync User (sync mode)
- `test_archive_user` - Test Archive User (archive mode)
- `wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
- `work_pattern_test` - Work Pattern Test (sync mode)
- `specific_folders_only` - Specific Folders Only (archive mode)
- `subfolder_pattern_test` - Subfolder Pattern Test (archive mode)
Each database contains documents with:
- `mailbox` field indicating the origin folder
- Native CouchDB attachments for email attachments
- Full message headers and body content
## Testing Sync vs Archive Modes
@ -85,22 +102,68 @@ mail2couch will create separate databases for each mail source:
- Messages deleted from IMAP remain in CouchDB
- Archive/backup behavior
## Wildcard Pattern Examples
The test environment demonstrates these wildcard patterns:
### All Folders Pattern (`*`)
```json
{
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash"]
}
}
```
Processes all folders except Drafts and Trash.
### Work Pattern (`Work*`)
```json
{
"folderFilter": {
"include": ["Work*", "Important*", "INBOX"],
"exclude": ["*Temp*"]
}
}
```
Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp.
### Specific Folders
```json
{
"folderFilter": {
"include": ["INBOX", "Sent", "Personal"],
"exclude": []
}
}
```
Only processes the exact named folders.
### Subfolder Pattern (`Work/*`)
```json
{
"folderFilter": {
"include": ["Work/*", "Archive/*"],
"exclude": ["*/Drafts"]
}
}
```
Includes all subfolders under Work and Archive, but excludes any Drafts subfolder.
## File Structure
```
test/
├── podman-compose.yml # Container orchestration
├── config-test.json # Test configuration
├── config-test.json # Main test configuration with wildcard examples
├── config-wildcard-examples.json # Advanced wildcard patterns
├── test-wildcard-patterns.sh # Wildcard pattern testing script
├── run-tests.sh # Full integration test
├── start-test-env.sh # Start environment
├── stop-test-env.sh # Stop environment
├── generate-ssl.sh # Generate SSL certificates
├── populate-test-messages.sh # Create test messages
├── dovecot/
│ ├── dovecot.conf # Dovecot configuration
│ ├── users # User database
│ ├── passwd # Password database
│ └── ssl/ # SSL certificates
├── populate-greenmail.py # Create test messages with folders
├── populate-test-messages.sh # Wrapper script
├── dovecot/ # Dovecot configuration (legacy)
└── README.md # This file
```

42
test/config-test.json
View file

@ -7,7 +7,7 @@
},
"mailSources": [
{
"name": "Test User 1",
"name": "Wildcard All Folders Test",
"enabled": true,
"protocol": "imap",
"host": "localhost",
@ -16,13 +16,16 @@
"password": "password123",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": []
"include": ["*"],
"exclude": ["Drafts", "Trash"]
},
"messageFilter": {}
"messageFilter": {
"subjectKeywords": ["meeting", "important"],
"senderKeywords": ["@company.com"]
}
},
{
"name": "Test Sync User",
"name": "Work Pattern Test",
"enabled": true,
"protocol": "imap",
"host": "localhost",
@ -31,13 +34,15 @@
"password": "syncpass",
"mode": "sync",
"folderFilter": {
"include": ["INBOX"],
"exclude": []
"include": ["Work*", "Important*", "INBOX"],
"exclude": ["*Temp*"]
},
"messageFilter": {}
"messageFilter": {
"recipientKeywords": ["support@", "team@"]
}
},
{
"name": "Test Archive User",
"name": "Specific Folders Only",
"enabled": true,
"protocol": "imap",
"host": "localhost",
@ -46,8 +51,23 @@
"password": "archivepass",
"mode": "archive",
"folderFilter": {
"include": [],
"exclude": ["Drafts"]
"include": ["INBOX", "Sent", "Personal"],
"exclude": []
},
"messageFilter": {}
},
{
"name": "Subfolder Pattern Test",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser2",
"password": "password456",
"mode": "archive",
"folderFilter": {
"include": ["Work/*", "Archive/*"],
"exclude": ["*/Drafts"]
},
"messageFilter": {}
}

72
test/config-wildcard-examples.json Normal file
View file

@ -0,0 +1,72 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password",
"database": "mail_backup_test"
},
"mailSources": [
{
"name": "All Folders Example",
"enabled": true,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser1",
"password": "password123",
"mode": "archive",
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash", "Spam"]
},
"messageFilter": {}
},
{
"name": "Inbox and Sent Only",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser2",
"password": "password456",
"mode": "sync",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": []
},
"messageFilter": {}
},
{
"name": "Work Folders Pattern",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "workuser",
"password": "workpass",
"mode": "archive",
"folderFilter": {
"include": ["Work*", "Projects*", "INBOX"],
"exclude": ["*Temp*", "*Draft*"]
},
"messageFilter": {
"senderKeywords": ["@company.com"]
}
},
{
"name": "Archive Pattern Example",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "archiveuser",
"password": "archivepass",
"mode": "archive",
"folderFilter": {
"include": ["Archive*", "*Important*", "INBOX"],
"exclude": ["*Temp"]
},
"messageFilter": {}
}
]
}

122
test/populate-greenmail.py
View file

@ -9,19 +9,21 @@ from email import encoders
import time
import sys
def create_simple_message(subject, body, from_addr="test-sender@example.com"):
def create_simple_message(subject, body, from_addr="test-sender@example.com", to_addr="user@example.com"):
"""Create a simple text message"""
msg = MIMEText(body)
msg['Subject'] = subject
msg['From'] = from_addr
msg['To'] = to_addr
msg['Date'] = email.utils.formatdate(localtime=True)
return msg.as_string()
def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com"):
def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com", to_addr="user@example.com"):
"""Create a message with an attachment"""
msg = MIMEMultipart()
msg['Subject'] = subject
msg['From'] = from_addr
msg['To'] = to_addr
msg['Date'] = email.utils.formatdate(localtime=True)
# Add body
@ -44,45 +46,93 @@ def populate_user_mailbox(username, password, host='localhost', port=3143):
# Connect to IMAP server
imap = imaplib.IMAP4(host, port)
imap.login(username, password)
imap.select('INBOX')
print(f"Creating messages for {username}...")
# Create 10 regular messages
for i in range(1, 11):
if i % 3 == 0:
# Every 3rd message has attachment
msg = create_message_with_attachment(
f"Test Message {i} with Attachment",
f"This is test message {i} for {username} with an attachment.",
f"Sample attachment content for message {i}"
)
print(f" Created message {i} (with attachment)")
else:
# Simple message
msg = create_simple_message(
f"Test Message {i}",
f"This is test message {i} for {username}."
)
print(f" Created message {i}")
# Append message to INBOX
imap.append('INBOX', None, None, msg.encode('utf-8'))
time.sleep(0.1) # Small delay to avoid overwhelming
# Create additional test messages
special_messages = [
("Important Message", "This is an important message for testing sync/archive modes."),
("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀")
# Create additional folders for testing wildcards
# These folders are designed to test various wildcard patterns
test_folders = [
'Sent', # Exact match
'Work/Projects', # Work/* pattern
'Work/Archive', # Work/* pattern
'Work/Temp', # Work/* but excluded by *Temp*
'Personal', # Exact match
'Important/Urgent', # Important/* pattern
'Important/Meetings', # Important/* pattern
'Archive/2024', # Archive/* pattern
'Archive/Projects', # Archive/* pattern
'Archive/Drafts', # Archive/* but excluded by */Drafts
'Drafts', # Should be excluded
'Trash' # Should be excluded
]
created_folders = ['INBOX'] # INBOX always exists
for subject, body in special_messages:
msg = create_simple_message(subject, body)
imap.append('INBOX', None, None, msg.encode('utf-8'))
print(f" Created special message: {subject}")
for folder in test_folders:
try:
imap.create(folder)
created_folders.append(folder)
print(f" Created folder: {folder}")
except Exception as e:
print(f" Folder {folder} may already exist or creation failed: {e}")
# Try to select it to see if it exists
try:
imap.select(folder)
created_folders.append(folder)
print(f" Folder {folder} already exists")
except:
pass
print(f"Available folders for {username}: {created_folders}")
# Populate each folder with messages
for folder in created_folders[:3]: # Limit to first 3 folders to avoid too many messages
imap.select(folder)
print(f"Creating messages in {folder} for {username}...")
# Create fewer messages per folder for testing
message_count = 3 if folder != 'INBOX' else 10
for i in range(1, message_count + 1):
if i % 3 == 0 and folder == 'INBOX':
# Every 3rd message has attachment (only in INBOX)
msg = create_message_with_attachment(
f"[{folder}] Test Message {i} with Attachment",
f"This is test message {i} in {folder} for {username} with an attachment.",
f"Sample attachment content for message {i} in {folder}",
"test-sender@example.com",
f"{username}@example.com"
)
print(f" Created message {i} (with attachment)")
else:
# Simple message
msg = create_simple_message(
f"[{folder}] Test Message {i}",
f"This is test message {i} in {folder} for {username}.",
"test-sender@example.com",
f"{username}@example.com"
)
print(f" Created message {i}")
# Append message to current folder
imap.append(folder, None, None, msg.encode('utf-8'))
time.sleep(0.1) # Small delay to avoid overwhelming
# Add special messages only to INBOX for keyword filtering tests
if folder == 'INBOX':
special_messages = [
("Important Meeting Reminder", "This is an important meeting message for testing keyword filters.", "manager@company.com", "team@company.com"),
("Urgent: System Maintenance", "Important notification about system maintenance.", "admin@company.com", f"{username}@example.com"),
("Regular Newsletter", "This is a regular newsletter message.", "newsletter@external.com", f"{username}@example.com"),
("Team Meeting Notes", "Meeting notes from the team.", "secretary@company.com", "support@company.com"),
("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀", "test-sender@example.com", f"{username}@example.com")
]
for subject, body, sender, recipient in special_messages:
msg = create_simple_message(subject, body, sender, recipient)
imap.append(folder, None, None, msg.encode('utf-8'))
print(f" Created special message: {subject} from {sender} to {recipient}")
time.sleep(0.1)
imap.logout()
print(f"✅ Successfully created 12 messages for {username}")
print(f"✅ Successfully created messages across {len(created_folders[:3])} folders for {username}")
return True
except Exception as e:

137
test/test-wildcard-patterns.sh Executable file
View file

@ -0,0 +1,137 @@
#!/bin/bash
# Test script to validate wildcard folder pattern functionality
# This script tests the various wildcard patterns against the test environment
set -e
echo "🧪 Testing Wildcard Folder Pattern Functionality"
echo "================================================"
# Make sure we're in the right directory
cd "$(dirname "$0")/.."
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to run mail2couch with a specific config and capture output
run_test() {
local test_name="$1"
local config_file="$2"
local max_messages="$3"
echo -e "\n${BLUE}Testing: $test_name${NC}"
echo "Config: $config_file"
echo "Max messages: $max_messages"
echo "----------------------------------------"
# Run mail2couch and capture output
cd go
if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then
echo -e "${GREEN}✅ Test completed successfully${NC}"
else
echo -e "${RED}❌ Test failed${NC}"
return 1
fi
cd ..
}
# Function to check if containers are running
check_containers() {
echo "🔍 Checking if test containers are running..."
if ! podman ps | grep -q "greenmail"; then
echo -e "${RED}❌ GreenMail container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
if ! podman ps | grep -q "couchdb"; then
echo -e "${RED}❌ CouchDB container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
echo -e "${GREEN}✅ Test containers are running${NC}"
}
# Function to populate test data
populate_test_data() {
echo "📧 Populating test data..."
cd test
if python3 populate-greenmail.py; then
echo -e "${GREEN}✅ Test data populated successfully${NC}"
else
echo -e "${RED}❌ Failed to populate test data${NC}"
exit 1
fi
cd ..
}
# Function to build the application
build_app() {
echo "🔨 Building mail2couch..."
cd go
if go build -o mail2couch .; then
echo -e "${GREEN}✅ Build successful${NC}"
else
echo -e "${RED}❌ Build failed${NC}"
exit 1
fi
cd ..
}
# Main test execution
main() {
echo "Starting wildcard pattern tests..."
# Pre-test checks
check_containers
build_app
populate_test_data
# Wait a moment for test data to be fully ready
echo "⏳ Waiting for test data to settle..."
sleep 3
# Test 1: Wildcard all folders (*)
echo -e "\n${YELLOW}Test 1: Wildcard All Folders Pattern (*)${NC}"
echo "Expected: Should process all folders except Drafts and Trash"
run_test "Wildcard All Folders" "config-test.json" 3
# Test 2: Work pattern (Work*)
echo -e "\n${YELLOW}Test 2: Work Pattern (Work*)${NC}"
echo "Expected: Should process Work/Projects, Work/Archive but not Work/Temp (excluded by *Temp*)"
run_test "Work Pattern" "config-test.json" 3
# Test 3: Specific folders only
echo -e "\n${YELLOW}Test 3: Specific Folders Only${NC}"
echo "Expected: Should only process INBOX, Sent, and Personal folders"
run_test "Specific Folders" "config-test.json" 3
# Test 4: Advanced wildcard examples
echo -e "\n${YELLOW}Test 4: Advanced Wildcard Examples${NC}"
echo "Expected: Various complex patterns should work correctly"
run_test "Advanced Patterns" "config-wildcard-examples.json" 2
echo -e "\n${GREEN}🎉 All wildcard pattern tests completed!${NC}"
echo ""
echo "To verify results, check the CouchDB databases:"
echo " http://localhost:5984/_utils"
echo ""
echo "Expected databases should be created for each account:"
echo " - wildcard_all_folders_test"
echo " - work_pattern_test"
echo " - specific_folders_only"
echo ""
echo "Each database should contain documents with 'mailbox' field showing origin folder."
}
# Run main function if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi