diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..fce9b14 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,161 @@ +# CLAUDE.md + +This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. + +## Project Overview + +mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations: +- **Go implementation**: Located in `/go/` directory (currently the active implementation) +- **Rust implementation**: Planned but not yet implemented + +## Development Commands + +### Go Implementation (Primary) + +```bash +# Build the application +cd go && go build -o mail2couch . + +# Run the application with automatic config discovery +cd go && ./mail2couch + +# Run with specific config file +cd go && ./mail2couch -config /path/to/config.json + +# Run with message limit (useful for large mailboxes) +cd go && ./mail2couch -max-messages 100 + +# Run with both config and message limit +cd go && ./mail2couch -config /path/to/config.json -max-messages 50 + +# Run linting/static analysis +cd go && go vet ./... + +# Run integration tests with Podman containers +cd test && ./run-tests.sh + +# Run specialized tests +cd test && ./test-wildcard-patterns.sh +cd test && ./test-incremental-sync.sh + +# Run unit tests (none currently implemented) +cd go && go test ./... + +# Check dependencies +cd go && go mod tidy +``` + +## Architecture + +### Core Components + +1. **Configuration (`config/`)**: JSON-based configuration system + - Supports multiple mail sources with filtering options + - CouchDB connection settings + - Each source can have folder and message filters + +2. **Mail Handling (`mail/`)**: IMAP client implementation + - Uses `github.com/emersion/go-imap/v2` for IMAP operations + - Supports TLS connections + - Currently only lists mailboxes (backup functionality not yet implemented) + +3. **CouchDB Integration (`couch/`)**: Database operations + - Uses `github.com/go-kivik/kivik/v4` as CouchDB driver + - Handles database creation and document management + - Defines `MailDocument` structure for email storage + +### Configuration Structure + +The application uses `config.json` for configuration with the following structure: +- `couchDb`: Database connection settings (URL, credentials) +- `mailSources`: Array of mail sources with individual settings: + - Protocol support (currently only IMAP) + - Connection details (host, port, credentials) + - `mode`: Either "sync" or "archive" (defaults to "archive" if not specified) + - **sync**: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB) + - **archive**: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents) + - Filtering options for folders and messages with wildcard support + - Enable/disable per source + +### Configuration File Discovery + +The application automatically searches for configuration files in the following order: +1. Path specified by `-config` command line flag +2. `./config.json` (current working directory) +3. `./config/config.json` (config subdirectory) +4. `~/.config/mail2couch/config.json` (user XDG config directory) +5. `~/.mail2couch.json` (user home directory) + +This design ensures the same `config.json` format will work for both Go and Rust implementations. + +### Current Implementation Status + +- ✅ Configuration loading with automatic file discovery +- ✅ Command line flag support for config file path +- ✅ Per-account CouchDB database creation and management +- ✅ IMAP connection and mailbox listing +- ✅ Build error fixes +- ✅ Email message retrieval framework (with placeholder data) +- ✅ Email storage to CouchDB framework with native attachments +- ✅ Folder filtering logic with wildcard support (`*`, `?`, `[abc]` patterns) +- ✅ Date filtering support +- ✅ Keyword filtering support (subject, sender, recipient keywords) +- ✅ Duplicate detection and prevention +- ✅ Sync vs Archive mode implementation +- ✅ CouchDB attachment storage for email attachments +- ✅ Real IMAP message parsing with go-message library +- ✅ Full message body and attachment handling with MIME multipart support +- ✅ Command line argument support (--max-messages flag) +- ✅ Per-account CouchDB databases for better organization +- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking +- ❌ Rust implementation + +### Key Dependencies + +- `github.com/emersion/go-imap/v2`: IMAP client library +- `github.com/go-kivik/kivik/v4`: CouchDB client library + +### Incremental Sync Implementation + +The application implements intelligent incremental synchronization to avoid re-processing messages: + +- **Sync Metadata Storage**: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed +- **IMAP SEARCH Integration**: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages +- **Per-Mailbox Tracking**: Sync state is tracked independently for each mailbox in each account +- **Fallback Behavior**: Gracefully falls back to fetching recent messages if IMAP SEARCH fails +- **First Sync Handling**: Initial sync can use config `since` date or perform full sync + +Sync metadata documents are stored in CouchDB with ID format: `sync_metadata_{mailbox}` and include: +- `lastSyncTime`: When this mailbox was last successfully synced +- `lastMessageUID`: Highest UID processed in the last sync +- `messageCount`: Number of messages processed in the last sync + +### Development Notes + +- The main entry point is `main.go` which orchestrates the configuration loading, CouchDB setup, and mail source processing +- Each mail source gets its own CouchDB database named using `GenerateAccountDBName()` function with `m2c_` prefix +- Each mail source is processed sequentially with proper error handling +- The application uses real IMAP message parsing with go-message library for full email processing +- Message filtering by folder (wildcard patterns), date (since), and keywords is implemented +- Duplicate detection prevents re-storing existing messages +- Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account +- Email attachments are stored as native CouchDB attachments linked to the email document +- Comprehensive test environment with Podman containers and automated test scripts +- The application uses automatic config file discovery as documented above + +### Next Steps + +The following enhancements could further improve the implementation: + +1. **Error Recovery**: Add retry logic for network failures and partial sync recovery +2. **Performance Optimization**: Add batch operations for better CouchDB insertion performance +3. **Unit Testing**: Add comprehensive unit tests for all major components +4. **Advanced Filtering**: Add support for more complex filter expressions and regex patterns +5. **Monitoring**: Add metrics and logging for production deployment +6. **Configuration Validation**: Enhanced validation for configuration files +7. **Multi-threading**: Parallel processing of multiple mailboxes or accounts + +## Development Guidelines + +### Code Quality and Standards +- All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly \ No newline at end of file diff --git a/FOLDER_PATTERNS.md b/FOLDER_PATTERNS.md new file mode 100644 index 0000000..dc96b8c --- /dev/null +++ b/FOLDER_PATTERNS.md @@ -0,0 +1,102 @@ +# Folder Pattern Matching in mail2couch + +mail2couch supports powerful wildcard patterns for selecting which folders to process. This allows flexible configuration for different mail backup scenarios. + +## Pattern Syntax + +The folder filtering uses Go's `filepath.Match` syntax, which supports: + +- `*` matches any sequence of characters (including none) +- `?` matches any single character +- `[abc]` matches any character within the brackets +- `[a-z]` matches any character in the range +- `\` escapes special characters + +## Special Cases + +- `"*"` in the include list means **ALL available folders** will be processed +- Empty include list with exclude patterns will process all folders except excluded ones +- Exact string matching is supported for backwards compatibility + +## Examples + +### Include All Folders +```json +{ + "folderFilter": { + "include": ["*"], + "exclude": ["Drafts", "Trash", "Spam"] + } +} +``` +This processes all folders except Drafts, Trash, and Spam. + +### Work-Related Folders Only +```json +{ + "folderFilter": { + "include": ["Work*", "Projects*", "INBOX"], + "exclude": ["*Temp*", "*Draft*"] + } +} +``` +This includes folders starting with "Work" or "Projects", plus INBOX, but excludes any folder containing "Temp" or "Draft". + +### Archive Patterns +```json +{ + "folderFilter": { + "include": ["Archive*", "*Important*", "INBOX"], + "exclude": ["*Temp"] + } +} +``` +This includes folders starting with "Archive", any folder containing "Important", and INBOX, excluding temporary folders. + +### Specific Folders Only +```json +{ + "folderFilter": { + "include": ["INBOX", "Sent", "Important"], + "exclude": [] + } +} +``` +This processes only the exact folders: INBOX, Sent, and Important. + +### Subfolder Patterns +```json +{ + "folderFilter": { + "include": ["Work/*", "Personal/*"], + "exclude": ["*/Drafts"] + } +} +``` +This includes all subfolders under Work and Personal, but excludes any Drafts subfolder. + +## Folder Hierarchy + +Different IMAP servers use different separators for folder hierarchies: +- Most servers use `/` (e.g., `Work/Projects`, `Archive/2024`) +- Some use `.` (e.g., `Work.Projects`, `Archive.2024`) + +The patterns work with whatever separator your IMAP server uses. + +## Common Use Cases + +1. **Corporate Email**: `["*"]` with exclude `["Drafts", "Trash", "Spam"]` for complete backup +2. **Selective Backup**: `["INBOX", "Sent", "Important"]` for essential folders only +3. **Project-based**: `["Project*", "Client*"]` to backup work-related folders +4. **Archive Mode**: `["Archive*", "*Important*"]` for long-term storage +5. **Sync Mode**: `["INBOX"]` for real-time synchronization + +## Message Origin Tracking + +All messages stored in CouchDB include a `mailbox` field that records the original folder name. This ensures you can always identify which folder a message came from, regardless of how it was selected by the folder filter. + +## Performance Considerations + +- Using `"*"` processes all folders, which may be slow for accounts with many folders +- Specific folder names are faster than wildcard patterns +- Consider using exclude patterns to filter out large, unimportant folders like Trash or Spam \ No newline at end of file diff --git a/README.md b/README.md index 631d01c..f81e050 100644 --- a/README.md +++ b/README.md @@ -1,5 +1,389 @@ # mail2couch -A utility to back up mail from various sources to couchdb +A powerful email backup utility that synchronizes mail from IMAP accounts to CouchDB databases with intelligent incremental sync, comprehensive filtering, and native attachment support. -At least two implementations will be available, on in Rust and one in Go. +## Features + +### Core Functionality +- **IMAP Email Backup**: Connect to any IMAP server (Gmail, Outlook, self-hosted) +- **CouchDB Storage**: Store emails as JSON documents with native CouchDB attachments +- **Incremental Sync**: Efficiently sync only new messages using IMAP SEARCH with timestamp tracking +- **Per-Account Databases**: Each mail source gets its own CouchDB database for better organization +- **Duplicate Prevention**: Automatic detection and prevention of duplicate message storage + +### Sync Modes +- **Archive Mode**: Preserve all messages ever seen, even if deleted from mail server (default) +- **Sync Mode**: Maintain 1-to-1 relationship with mail server (removes deleted messages from CouchDB) + +### Advanced Filtering +- **Wildcard Folder Patterns**: Use `*`, `?`, `[abc]` patterns for flexible folder selection +- **Keyword Filtering**: Filter messages by keywords in subjects, senders, or recipients +- **Date Filtering**: Process only messages since a specific date +- **Include/Exclude Logic**: Combine multiple filter types for precise control + +### Message Processing +- **Full MIME Support**: Parse multipart messages, HTML/plain text, and embedded content +- **Native Attachments**: Store email attachments as CouchDB native attachments with compression +- **Complete Headers**: Preserve all email headers and metadata +- **UTF-8 Support**: Handle international characters and special content + +### Operational Features +- **Automatic Config Discovery**: Finds configuration files in standard locations +- **Command Line Control**: Override settings with `--max-messages` and `--config` flags +- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting +- **Error Resilience**: Graceful handling of network issues and server problems + +## Quick Start + +### Installation + +1. **Install dependencies**: + ```bash + # Go 1.21+ required + go version + ``` + +2. **Clone and build**: + ```bash + git clone + cd mail2couch/go + go build -o mail2couch . + ``` + +### Basic Usage + +1. **Create configuration file** (`config.json`): + ```json + { + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "your-email@gmail.com", + "password": "your-app-password", + "mode": "archive", + "folderFilter": { + "include": ["*"], + "exclude": ["[Gmail]/Trash", "[Gmail]/Spam"] + } + } + ] + } + ``` + +2. **Run mail2couch**: + ```bash + ./mail2couch + ``` + +The application will: +- Create a CouchDB database named `m2c_personal_gmail` +- Sync all folders except Trash and Spam +- Store messages with native attachments +- Track sync state for efficient incremental updates + +## Configuration + +### Configuration File Discovery + +mail2couch automatically searches for configuration files in this order: +1. Path specified by `--config` flag +2. `./config.json` (current directory) +3. `./config/config.json` (config subdirectory) +4. `~/.config/mail2couch/config.json` (user config directory) +5. `~/.mail2couch.json` (user home directory) + +### Command Line Options + +```bash +./mail2couch [options] + +Options: + --config PATH Specify configuration file path + --max-messages N Limit messages processed per mailbox per run (0 = unlimited) +``` + +### Folder Pattern Examples + +| Pattern | Description | Matches | +|---------|-------------|---------| +| `"*"` | All folders | `INBOX`, `Sent`, `Work/Projects`, etc. | +| `"INBOX"` | Exact match | `INBOX` only | +| `"Work*"` | Prefix match | `Work`, `Work/Projects`, `WorkStuff` | +| `"*/Archive"` | Suffix match | `Personal/Archive`, `Work/Archive` | +| `"Work/*"` | Subfolder match | `Work/Projects`, `Work/Clients` | + +### Keyword Filtering Examples + +```json +{ + "messageFilter": { + "subjectKeywords": ["urgent", "meeting", "invoice"], + "senderKeywords": ["@company.com", "noreply@"], + "recipientKeywords": ["team@", "support@"] + } +} +``` + +## Advanced Configuration Examples + +See the [example configurations](#example-configurations) section below for detailed configuration scenarios. + +## Testing + +A comprehensive test environment is included with Podman containers: + +```bash +cd test + +# Quick automated testing (recommended) +./run-tests.sh # Complete integration test with automatic cleanup + +# Specialized feature testing +./test-wildcard-patterns.sh # Test folder pattern matching +./test-incremental-sync.sh # Test incremental synchronization + +# Manual testing environment +./start-test-env.sh # Start persistent test environment +# ... manual testing with various configurations ... +./stop-test-env.sh # Clean up when done +``` + +## Architecture + +### Database Structure +- **Per-Account Databases**: Each mail source creates its own CouchDB database with `m2c_` prefix +- **Message Documents**: Each email becomes a CouchDB document with metadata +- **Native Attachments**: Email attachments stored as CouchDB attachments (compressed) +- **Sync Metadata**: Tracks incremental sync state per mailbox + +### Document Structure +```json +{ + "_id": "INBOX_12345", + "sourceUid": "12345", + "mailbox": "INBOX", + "from": ["sender@example.com"], + "to": ["recipient@example.com"], + "subject": "Sample Email", + "date": "2024-01-15T10:30:00Z", + "body": "Email content...", + "headers": {"Content-Type": ["text/plain"]}, + "storedAt": "2024-01-15T10:35:00Z", + "docType": "mail", + "hasAttachments": true, + "_attachments": { + "document.pdf": { + "content_type": "application/pdf", + "length": 54321 + } + } +} +``` + +## Example Configurations + +### Simple Configuration +Basic setup for a single Gmail account: + +```json +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "your-email@gmail.com", + "password": "your-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": [] + }, + "messageFilter": { + "since": "2024-01-01" + } + } + ] +} +``` + +### Advanced Multi-Account Configuration +Complex setup with multiple accounts, filtering, and different sync modes: + +```json +{ + "couchDb": { + "url": "https://your-couchdb.example.com:5984", + "user": "backup_user", + "password": "secure_password" + }, + "mailSources": [ + { + "name": "Work Email", + "enabled": true, + "protocol": "imap", + "host": "outlook.office365.com", + "port": 993, + "user": "you@company.com", + "password": "app-password", + "mode": "sync", + "folderFilter": { + "include": ["*"], + "exclude": ["Deleted Items", "Junk Email", "Drafts"] + }, + "messageFilter": { + "since": "2023-01-01", + "subjectKeywords": ["project", "meeting", "urgent"], + "senderKeywords": ["@company.com", "@client.com"] + } + }, + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "personal@gmail.com", + "password": "gmail-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Important", "Work/*", "Personal/*"], + "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"] + }, + "messageFilter": { + "recipientKeywords": ["family@", "personal@"] + } + }, + { + "name": "Self-Hosted Mail", + "enabled": true, + "protocol": "imap", + "host": "mail.yourdomain.com", + "port": 143, + "user": "admin@yourdomain.com", + "password": "mail-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Archive/*", "Projects/*"], + "exclude": ["*/Drafts", "Trash"] + }, + "messageFilter": { + "since": "2023-06-01", + "subjectKeywords": ["invoice", "receipt", "statement"] + } + }, + { + "name": "Legacy Account", + "enabled": false, + "protocol": "imap", + "host": "legacy.mailserver.com", + "port": 993, + "user": "old@account.com", + "password": "legacy-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX"], + "exclude": [] + }, + "messageFilter": {} + } + ] +} +``` + +### Configuration Options Reference + +#### CouchDB Configuration +- `url`: CouchDB server URL with protocol and port +- `user`: CouchDB username with database access +- `password`: CouchDB password + +#### Mail Source Configuration +- `name`: Descriptive name (used for database naming) +- `enabled`: Boolean to enable/disable this source +- `protocol`: Only `"imap"` currently supported +- `host`: IMAP server hostname +- `port`: IMAP port (993 for TLS, 143 for plain, 3143 for testing) +- `user`: Email account username +- `password`: Email account password (use app passwords for Gmail/Outlook) +- `mode`: `"sync"` (mirror server) or `"archive"` (preserve all messages) + +#### Folder Filter Configuration +- `include`: Array of folder patterns to process (empty = all folders) +- `exclude`: Array of folder patterns to skip + +#### Message Filter Configuration +- `since`: Date string (YYYY-MM-DD) to process messages from +- `subjectKeywords`: Array of keywords that must appear in subject line +- `senderKeywords`: Array of keywords that must appear in sender addresses +- `recipientKeywords`: Array of keywords that must appear in recipient addresses + +## Production Deployment + +### Security Considerations +- Use app passwords instead of account passwords +- Store configuration files with restricted permissions (600) +- Use HTTPS for CouchDB connections in production +- Consider encrypting sensitive configuration data + +### Monitoring and Maintenance +- Review sync metadata documents for sync health +- Monitor CouchDB database sizes and compaction +- Set up log rotation for application output +- Schedule regular backups of CouchDB databases + +### Performance Tuning +- Use `--max-messages` to limit processing load +- Run during off-peak hours for large initial syncs +- Monitor IMAP server rate limits and connection limits +- Consider running multiple instances for different accounts + +## Troubleshooting + +### Common Issues + +**Connection Errors**: +- Verify IMAP server settings and credentials +- Check firewall and network connectivity +- Ensure correct ports (993 for TLS, 143 for plain) + +**Authentication Failures**: +- Use app passwords for Gmail, Outlook, and other providers +- Enable "Less Secure Apps" if required by provider +- Verify account permissions and 2FA settings + +**Sync Issues**: +- Check CouchDB connectivity and permissions +- Review sync metadata documents for error states +- Verify folder names and patterns match server structure + +**Performance Problems**: +- Use date filtering (`since`) for large mailboxes +- Implement `--max-messages` limits for initial syncs +- Monitor server-side rate limiting + +For detailed troubleshooting, see the [test environment documentation](test/README.md). + +## Contributing + +This project welcomes contributions! Please see [CLAUDE.md](CLAUDE.md) for development setup and architecture details. + +## License + +[License information to be added] diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..e004c00 --- /dev/null +++ b/TODO.md @@ -0,0 +1,47 @@ +# mail2couch TODO and Feature Requests + +## Planned Features + +### Keyword Filtering for Messages + +Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration. + +**Proposed Configuration Extension:** + +```json +{ + "messageFilter": { + "since": "2024-01-01", + "subjectKeywords": ["urgent", "important", "meeting"], + "senderKeywords": ["@company.com", "notifications"], + "recipientKeywords": ["team@company.com", "all@"] + } +} +``` + +**Implementation Details:** + +- `subjectKeywords`: Array of keywords to match in email subject lines +- `senderKeywords`: Array of keywords to match in sender email addresses or names +- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names +- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns +- Case-insensitive matching by default +- Support for simple wildcards or regex patterns + +**Use Cases:** + +1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords +2. **Project-based Archiving**: Filter emails related to specific projects or clients +3. **Notification Management**: Exclude or include automated notifications based on sender patterns +4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders + +**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails. + +## Other Planned Improvements + +1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content +2. **Message Body Extraction**: Support for HTML/plain text and multipart messages +3. **Attachment Handling**: Optional support for email attachments +4. **Batch Operations**: Improve CouchDB insertion performance +5. **Error Recovery**: Retry logic and partial sync recovery +6. **Testing**: Comprehensive unit test coverage \ No newline at end of file diff --git a/config-advanced.json b/config-advanced.json new file mode 100644 index 0000000..a54f350 --- /dev/null +++ b/config-advanced.json @@ -0,0 +1,78 @@ +{ + "couchDb": { + "url": "https://your-couchdb.example.com:5984", + "user": "backup_user", + "password": "secure_password" + }, + "mailSources": [ + { + "name": "Work Email", + "enabled": true, + "protocol": "imap", + "host": "outlook.office365.com", + "port": 993, + "user": "you@company.com", + "password": "app-password", + "mode": "sync", + "folderFilter": { + "include": ["*"], + "exclude": ["Deleted Items", "Junk Email", "Drafts"] + }, + "messageFilter": { + "since": "2023-01-01", + "subjectKeywords": ["project", "meeting", "urgent"], + "senderKeywords": ["@company.com", "@client.com"] + } + }, + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "personal@gmail.com", + "password": "gmail-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Important", "Work/*", "Personal/*"], + "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"] + }, + "messageFilter": { + "recipientKeywords": ["family@", "personal@"] + } + }, + { + "name": "Self-Hosted Mail", + "enabled": true, + "protocol": "imap", + "host": "mail.yourdomain.com", + "port": 143, + "user": "admin@yourdomain.com", + "password": "mail-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Archive/*", "Projects/*"], + "exclude": ["*/Drafts", "Trash"] + }, + "messageFilter": { + "since": "2023-06-01", + "subjectKeywords": ["invoice", "receipt", "statement"] + } + }, + { + "name": "Legacy Account", + "enabled": false, + "protocol": "imap", + "host": "legacy.mailserver.com", + "port": 993, + "user": "old@account.com", + "password": "legacy-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX"], + "exclude": [] + }, + "messageFilter": {} + } + ] +} \ No newline at end of file diff --git a/config-providers.json b/config-providers.json new file mode 100644 index 0000000..577294c --- /dev/null +++ b/config-providers.json @@ -0,0 +1,90 @@ +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Gmail Account", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "your-email@gmail.com", + "password": "your-16-character-app-password", + "mode": "archive", + "folderFilter": { + "include": ["*"], + "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "[Gmail]/Drafts"] + }, + "messageFilter": { + "since": "2024-01-01" + } + }, + { + "name": "Outlook 365", + "enabled": true, + "protocol": "imap", + "host": "outlook.office365.com", + "port": 993, + "user": "you@outlook.com", + "password": "your-app-password", + "mode": "sync", + "folderFilter": { + "include": ["INBOX", "Sent Items", "Archive"], + "exclude": ["Deleted Items", "Junk Email"] + }, + "messageFilter": { + "since": "2023-06-01" + } + }, + { + "name": "Yahoo Mail", + "enabled": false, + "protocol": "imap", + "host": "imap.mail.yahoo.com", + "port": 993, + "user": "your-email@yahoo.com", + "password": "your-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": ["Trash", "Spam"] + }, + "messageFilter": {} + }, + { + "name": "iCloud Mail", + "enabled": false, + "protocol": "imap", + "host": "imap.mail.me.com", + "port": 993, + "user": "your-email@icloud.com", + "password": "your-app-specific-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent Messages"], + "exclude": ["Deleted Messages", "Junk"] + }, + "messageFilter": {} + }, + { + "name": "Custom IMAP Server", + "enabled": false, + "protocol": "imap", + "host": "mail.example.com", + "port": 993, + "user": "username@example.com", + "password": "password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": ["Trash"] + }, + "messageFilter": { + "since": "2024-01-01" + } + } + ] +} \ No newline at end of file diff --git a/config-simple.json b/config-simple.json new file mode 100644 index 0000000..4c9cb50 --- /dev/null +++ b/config-simple.json @@ -0,0 +1,26 @@ +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "your-email@gmail.com", + "password": "your-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": [] + }, + "messageFilter": { + "since": "2024-01-01" + } + } + ] +} \ No newline at end of file diff --git a/config.json b/config.json new file mode 100644 index 0000000..39f4af0 --- /dev/null +++ b/config.json @@ -0,0 +1,41 @@ +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Personal Gmail", + "enabled": true, + "protocol": "imap", + "host": "imap.gmail.com", + "port": 993, + "user": "your-email@gmail.com", + "password": "your-app-password", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": ["Spam", "Trash"] + }, + "messageFilter": { + "since": "2024-01-01" + } + }, + { + "name": "Work Account", + "enabled": true, + "protocol": "imap", + "host": "imap.work.com", + "port": 993, + "user": "user@work.com", + "password": "password", + "mode": "sync", + "folderFilter": { + "include": [], + "exclude": [] + }, + "messageFilter": {} + } + ] +} diff --git a/go/config/config.go b/go/config/config.go new file mode 100644 index 0000000..5581094 --- /dev/null +++ b/go/config/config.go @@ -0,0 +1,153 @@ +package config + +import ( + "encoding/json" + "flag" + "fmt" + "os" + "path/filepath" +) + +type Config struct { + CouchDb CouchDbConfig `json:"couchDb"` + MailSources []MailSource `json:"mailSources"` +} + +type CouchDbConfig struct { + URL string `json:"url"` + User string `json:"user"` + Password string `json:"password"` +} + +type MailSource struct { + Name string `json:"name"` + Enabled bool `json:"enabled"` + Protocol string `json:"protocol"` + Host string `json:"host"` + Port int `json:"port"` + User string `json:"user"` + Password string `json:"password"` + Mode string `json:"mode"` // "sync" or "archive" + FolderFilter FolderFilter `json:"folderFilter"` + MessageFilter MessageFilter `json:"messageFilter"` +} + +type FolderFilter struct { + Include []string `json:"include"` + Exclude []string `json:"exclude"` +} + +type MessageFilter struct { + Since string `json:"since,omitempty"` + SubjectKeywords []string `json:"subjectKeywords,omitempty"` // Filter by keywords in subject + SenderKeywords []string `json:"senderKeywords,omitempty"` // Filter by keywords in sender addresses + RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses +} + +func LoadConfig(path string) (*Config, error) { + configFile, err := os.Open(path) + if err != nil { + return nil, err + } + defer configFile.Close() + + var config Config + jsonParser := json.NewDecoder(configFile) + if err = jsonParser.Decode(&config); err != nil { + return nil, err + } + + // Validate and set defaults for mail sources + for i := range config.MailSources { + source := &config.MailSources[i] + if source.Mode == "" { + source.Mode = "archive" // Default to archive mode + } + if source.Mode != "sync" && source.Mode != "archive" { + return nil, fmt.Errorf("invalid mode '%s' for mail source '%s': must be 'sync' or 'archive'", source.Mode, source.Name) + } + } + + return &config, nil +} + +// IsSyncMode returns true if the mail source is in sync mode +func (ms *MailSource) IsSyncMode() bool { + return ms.Mode == "sync" +} + +// IsArchiveMode returns true if the mail source is in archive mode +func (ms *MailSource) IsArchiveMode() bool { + return ms.Mode == "archive" || ms.Mode == "" // Default to archive +} + +// CommandLineArgs holds parsed command line arguments +type CommandLineArgs struct { + ConfigPath string + MaxMessages int +} + +// ParseCommandLine parses command line arguments +func ParseCommandLine() *CommandLineArgs { + configFlag := flag.String("config", "", "Path to configuration file") + maxMessagesFlag := flag.Int("max-messages", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)") + flag.Parse() + + return &CommandLineArgs{ + ConfigPath: *configFlag, + MaxMessages: *maxMessagesFlag, + } +} + +// FindConfigFile searches for config.json in the following order: +// 1. Path specified by -config flag +// 2. ./config.json (current directory) +// 3. ~/.config/mail2couch/config.json (user config directory) +// 4. ~/.mail2couch.json (user home directory) +func FindConfigFile(args *CommandLineArgs) (string, error) { + if args.ConfigPath != "" { + if _, err := os.Stat(args.ConfigPath); err == nil { + return args.ConfigPath, nil + } + return "", fmt.Errorf("specified config file not found: %s", args.ConfigPath) + } + + // List of possible config file locations in order of preference + candidates := []string{ + "config.json", // Current directory + "config/config.json", // Config subdirectory + } + + // Add user directory paths + if homeDir, err := os.UserHomeDir(); err == nil { + candidates = append(candidates, + filepath.Join(homeDir, ".config", "mail2couch", "config.json"), + filepath.Join(homeDir, ".mail2couch.json"), + ) + } + + // Try each candidate location + for _, candidate := range candidates { + if _, err := os.Stat(candidate); err == nil { + return candidate, nil + } + } + + return "", fmt.Errorf("no configuration file found. Searched locations: %v", candidates) +} + +// LoadConfigWithDiscovery loads configuration using automatic file discovery +func LoadConfigWithDiscovery(args *CommandLineArgs) (*Config, error) { + configPath, err := FindConfigFile(args) + if err != nil { + return nil, err + } + + fmt.Printf("Using configuration file: %s\n", configPath) + if args.MaxMessages > 0 { + fmt.Printf("Maximum messages per mailbox: %d\n", args.MaxMessages) + } else { + fmt.Printf("Maximum messages per mailbox: unlimited\n") + } + return LoadConfig(configPath) +} diff --git a/go/couch/couch.go b/go/couch/couch.go new file mode 100644 index 0000000..7a3c5ab --- /dev/null +++ b/go/couch/couch.go @@ -0,0 +1,374 @@ +package couch + +import ( + "context" + "fmt" + "io" + "net/url" + "regexp" + "strings" + "time" + + "github.com/go-kivik/kivik/v4" + _ "github.com/go-kivik/kivik/v4/couchdb" // The CouchDB driver + "mail2couch/config" + "mail2couch/mail" +) + +// Client wraps the Kivik client +type Client struct { + *kivik.Client +} + +// MailDocument represents an email message stored in CouchDB +type MailDocument struct { + ID string `json:"_id,omitempty"` + Rev string `json:"_rev,omitempty"` + Attachments map[string]AttachmentStub `json:"_attachments,omitempty"` // CouchDB attachments + SourceUID string `json:"sourceUid"` // Unique ID from the mail source (e.g., IMAP UID) + Mailbox string `json:"mailbox"` // Source mailbox name + From []string `json:"from"` + To []string `json:"to"` + Subject string `json:"subject"` + Date time.Time `json:"date"` + Body string `json:"body"` + Headers map[string][]string `json:"headers"` + StoredAt time.Time `json:"storedAt"` // When the document was stored + DocType string `json:"docType"` // Always "mail" + HasAttachments bool `json:"hasAttachments"` // Indicates if message has attachments +} + +// AttachmentStub represents metadata for a CouchDB attachment +type AttachmentStub struct { + ContentType string `json:"content_type"` + Length int64 `json:"length,omitempty"` + Stub bool `json:"stub,omitempty"` +} + +// SyncMetadata represents sync state information stored in CouchDB +type SyncMetadata struct { + ID string `json:"_id,omitempty"` + Rev string `json:"_rev,omitempty"` + DocType string `json:"docType"` // Always "sync_metadata" + Mailbox string `json:"mailbox"` // Mailbox name + LastSyncTime time.Time `json:"lastSyncTime"` // When this mailbox was last synced + LastMessageUID uint32 `json:"lastMessageUID"` // Highest UID processed in last sync + MessageCount int `json:"messageCount"` // Number of messages processed in last sync + UpdatedAt time.Time `json:"updatedAt"` // When this metadata was last updated +} + +// NewClient creates a new CouchDB client from the configuration +func NewClient(cfg *config.CouchDbConfig) (*Client, error) { + parsedURL, err := url.Parse(cfg.URL) + if err != nil { + return nil, fmt.Errorf("invalid couchdb url: %w", err) + } + + parsedURL.User = url.UserPassword(cfg.User, cfg.Password) + dsn := parsedURL.String() + + client, err := kivik.New("couch", dsn) + if err != nil { + return nil, err + } + + return &Client{client}, nil +} + +// EnsureDB ensures that the configured database exists. +func (c *Client) EnsureDB(ctx context.Context, dbName string) error { + exists, err := c.DBExists(ctx, dbName) + if err != nil { + return err + } + if !exists { + return c.CreateDB(ctx, dbName) + } + return nil +} + +// GenerateAccountDBName creates a CouchDB-compatible database name from account info +func GenerateAccountDBName(accountName, userEmail string) string { + // Use account name if available, otherwise fall back to email + name := accountName + if name == "" { + name = userEmail + } + + // Convert to lowercase and replace invalid characters with underscores + name = strings.ToLower(name) + // CouchDB database names must match: ^[a-z][a-z0-9_$()+/-]*$ + validName := regexp.MustCompile(`[^a-z0-9_$()+/-]`).ReplaceAllString(name, "_") + + // Ensure it starts with a letter and add m2c prefix + if len(validName) > 0 && (validName[0] < 'a' || validName[0] > 'z') { + validName = "m2c_mail_" + validName + } else { + validName = "m2c_" + validName + } + + return validName +} + +// ConvertMessage converts an IMAP message to a MailDocument +func ConvertMessage(msg *mail.Message, mailbox string) *MailDocument { + docID := fmt.Sprintf("%s_%d", mailbox, msg.UID) + + doc := &MailDocument{ + ID: docID, + SourceUID: fmt.Sprintf("%d", msg.UID), + Mailbox: mailbox, + From: msg.From, + To: msg.To, + Subject: msg.Subject, + Date: msg.Date, + Body: msg.Body, + Headers: msg.Headers, + StoredAt: time.Now(), + DocType: "mail", + HasAttachments: len(msg.Attachments) > 0, + } + + // Don't add attachment metadata here - CouchDB will handle this when we store attachments + // We'll add the attachment metadata after successful document creation + + return doc +} + +// StoreMessage stores a mail message in CouchDB with attachments +func (c *Client) StoreMessage(ctx context.Context, dbName string, doc *MailDocument, msg *mail.Message) error { + db := c.DB(dbName) + if db.Err() != nil { + return db.Err() + } + + // Check if document already exists + exists, err := c.DocumentExists(ctx, dbName, doc.ID) + if err != nil { + return fmt.Errorf("failed to check if document exists: %w", err) + } + + if exists { + return nil // Document already exists, skip + } + + // Store the document first (without attachments) + rev, err := db.Put(ctx, doc.ID, doc) + if err != nil { + return fmt.Errorf("failed to store document: %w", err) + } + + // If there are attachments, store them as CouchDB attachments + if msg != nil && len(msg.Attachments) > 0 { + currentRev := rev + for _, att := range msg.Attachments { + newRev, err := c.StoreAttachment(ctx, dbName, doc.ID, currentRev, att.Filename, att.ContentType, att.Content) + if err != nil { + return fmt.Errorf("failed to store attachment %s: %w", att.Filename, err) + } + currentRev = newRev // Update revision for next attachment + } + } + + return nil +} + +// StoreAttachment stores an attachment to an existing CouchDB document +func (c *Client) StoreAttachment(ctx context.Context, dbName, docID, rev, filename, contentType string, content []byte) (string, error) { + db := c.DB(dbName) + if db.Err() != nil { + return "", db.Err() + } + + att := &kivik.Attachment{ + Filename: filename, + ContentType: contentType, + Content: io.NopCloser(strings.NewReader(string(content))), + } + + newRev, err := db.PutAttachment(ctx, docID, att, kivik.Rev(rev)) + if err != nil { + return "", fmt.Errorf("failed to store attachment: %w", err) + } + + return newRev, nil +} + +// StoreMessages stores multiple mail messages in CouchDB with their corresponding attachments +func (c *Client) StoreMessages(ctx context.Context, dbName string, docs []*MailDocument, messages []*mail.Message) error { + for i, doc := range docs { + var msg *mail.Message + if i < len(messages) { + msg = messages[i] + } + if err := c.StoreMessage(ctx, dbName, doc, msg); err != nil { + return err + } + } + return nil +} + +// DocumentExists checks if a document with the given ID already exists. +func (c *Client) DocumentExists(ctx context.Context, dbName, docID string) (bool, error) { + db := c.DB(dbName) + if db.Err() != nil { + return false, db.Err() + } + + row := db.Get(ctx, docID) + return row.Err() == nil, nil +} + +// GetAllMailDocumentIDs returns all mail document IDs from a database for a specific mailbox +func (c *Client) GetAllMailDocumentIDs(ctx context.Context, dbName, mailbox string) (map[string]bool, error) { + db := c.DB(dbName) + if db.Err() != nil { + return nil, db.Err() + } + + // Create a view query to get all document IDs for the specified mailbox + rows := db.AllDocs(ctx) + + docIDs := make(map[string]bool) + for rows.Next() { + docID, err := rows.ID() + if err != nil { + continue + } + // Filter by mailbox prefix (documents are named like "INBOX_123") + if mailbox == "" || strings.HasPrefix(docID, mailbox+"_") { + docIDs[docID] = true + } + } + + if rows.Err() != nil { + return nil, rows.Err() + } + + return docIDs, nil +} + +// DeleteDocument removes a document from CouchDB +func (c *Client) DeleteDocument(ctx context.Context, dbName, docID string) error { + db := c.DB(dbName) + if db.Err() != nil { + return db.Err() + } + + // Get the current revision + row := db.Get(ctx, docID) + if row.Err() != nil { + return row.Err() // Document doesn't exist or other error + } + + var doc struct { + Rev string `json:"_rev"` + } + if err := row.ScanDoc(&doc); err != nil { + return err + } + + // Delete the document + _, err := db.Delete(ctx, docID, doc.Rev) + return err +} + +// SyncMailbox synchronizes a mailbox between mail server and CouchDB +// In sync mode: removes documents from CouchDB that are no longer in the mail account +// In archive mode: keeps all documents (no removal) +func (c *Client) SyncMailbox(ctx context.Context, dbName, mailbox string, currentMessageUIDs map[uint32]bool, syncMode bool) error { + if !syncMode { + return nil // Archive mode - don't remove anything + } + + // Get all existing document IDs for this mailbox from CouchDB + existingDocs, err := c.GetAllMailDocumentIDs(ctx, dbName, mailbox) + if err != nil { + return fmt.Errorf("failed to get existing documents: %w", err) + } + + // Find documents that should be removed (exist in CouchDB but not in mail account) + var toDelete []string + for docID := range existingDocs { + // Extract UID from document ID (format: "mailbox_uid") + parts := strings.Split(docID, "_") + if len(parts) < 2 { + continue + } + + uidStr := parts[len(parts)-1] + uid := uint32(0) + if _, err := fmt.Sscanf(uidStr, "%d", &uid); err != nil { + continue + } + + // If this UID is not in the current mail account, mark for deletion + if !currentMessageUIDs[uid] { + toDelete = append(toDelete, docID) + } + } + + // Delete documents that are no longer in the mail account + for _, docID := range toDelete { + if err := c.DeleteDocument(ctx, dbName, docID); err != nil { + return fmt.Errorf("failed to delete document %s: %w", docID, err) + } + } + + if len(toDelete) > 0 { + fmt.Printf(" Sync mode: Removed %d documents no longer in mail account\n", len(toDelete)) + } + + return nil +} + +// GetSyncMetadata retrieves the sync metadata for a specific mailbox +func (c *Client) GetSyncMetadata(ctx context.Context, dbName, mailbox string) (*SyncMetadata, error) { + db := c.DB(dbName) + if db.Err() != nil { + return nil, db.Err() + } + + metadataID := fmt.Sprintf("sync_metadata_%s", mailbox) + row := db.Get(ctx, metadataID) + if row.Err() != nil { + // If metadata doesn't exist, return nil (not an error for first sync) + return nil, nil + } + + var metadata SyncMetadata + if err := row.ScanDoc(&metadata); err != nil { + return nil, fmt.Errorf("failed to scan sync metadata: %w", err) + } + + return &metadata, nil +} + +// StoreSyncMetadata stores or updates sync metadata for a mailbox +func (c *Client) StoreSyncMetadata(ctx context.Context, dbName string, metadata *SyncMetadata) error { + db := c.DB(dbName) + if db.Err() != nil { + return db.Err() + } + + metadata.ID = fmt.Sprintf("sync_metadata_%s", metadata.Mailbox) + metadata.DocType = "sync_metadata" + metadata.UpdatedAt = time.Now() + + // Check if metadata already exists to get current revision + existing, err := c.GetSyncMetadata(ctx, dbName, metadata.Mailbox) + if err != nil { + return fmt.Errorf("failed to check existing sync metadata: %w", err) + } + + if existing != nil { + metadata.Rev = existing.Rev + } + + _, err = db.Put(ctx, metadata.ID, metadata) + if err != nil { + return fmt.Errorf("failed to store sync metadata: %w", err) + } + + return nil +} diff --git a/go/go.mod b/go/go.mod new file mode 100644 index 0000000..377160a --- /dev/null +++ b/go/go.mod @@ -0,0 +1,16 @@ +module mail2couch + +go 1.24.4 + +require ( + github.com/emersion/go-imap/v2 v2.0.0-beta.5 + github.com/go-kivik/kivik/v4 v4.4.0 +) + +require ( + github.com/emersion/go-message v0.18.1 // indirect + github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 // indirect + github.com/google/uuid v1.6.0 // indirect + golang.org/x/net v0.25.0 // indirect + golang.org/x/sync v0.10.0 // indirect +) diff --git a/go/go.sum b/go/go.sum new file mode 100644 index 0000000..7851761 --- /dev/null +++ b/go/go.sum @@ -0,0 +1,65 @@ +github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0= +github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ= +github.com/cenkalti/backoff/v4 v4.1.3 h1:cFAlzYUlVYDysBEH2T5hyJZMh3+5+WCBvSnK6Q8UtC4= +github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= +github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/emersion/go-imap/v2 v2.0.0-beta.5 h1:H3858DNmBuXyMK1++YrQIRdpKE1MwBc+ywBtg3n+0wA= +github.com/emersion/go-imap/v2 v2.0.0-beta.5/go.mod h1:BZTFHsS1hmgBkFlHqbxGLXk2hnRqTItUgwjSSCsYNAk= +github.com/emersion/go-message v0.18.1 h1:tfTxIoXFSFRwWaZsgnqS1DSZuGpYGzSmCZD8SK3QA2E= +github.com/emersion/go-message v0.18.1/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA= +github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 h1:hH4PQfOndHDlpzYfLAAfl63E8Le6F2+EL/cdhlkyRJY= +github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ= +github.com/go-kivik/kivik/v4 v4.4.0 h1:1YMqNvRMIIC+CJUtyldD7c4Czl6SqdUcnbusCoFOTfk= +github.com/go-kivik/kivik/v4 v4.4.0/go.mod h1:DnPzIEO7CcLOqJNuqxuo7EMZeK4bPsEbUSSmAfi+tL4= +github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= +github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g= +github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k= +github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0 h1:nHoRIX8iXob3Y2kdt9KsjyIb7iApSvb3vgsd93xb5Ow= +github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0/go.mod h1:c1tRKs5Tx7E2+uHGSyyncziFjvGpgv4H2HrqXeUQ/Uk= +github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= +github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= +github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= +gitlab.com/flimzy/testy v0.14.0 h1:2nZV4Wa1OSJb3rOKHh0GJqvvhtE03zT+sKnPCI0owfQ= +gitlab.com/flimzy/testy v0.14.0/go.mod h1:m3aGuwdXc+N3QgnH+2Ar2zf1yg0UxNdIaXKvC5SlfMk= +golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= +golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= +golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= +golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= +golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= +golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= +golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= +golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= +golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac= +golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= +golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= +golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ= +golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= +golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= +golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= +golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= +golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= +golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= +golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= +golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= +golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= +golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= +golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= +golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= +golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= +golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= +golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/go/mail/imap.go b/go/mail/imap.go new file mode 100644 index 0000000..63c7d7e --- /dev/null +++ b/go/mail/imap.go @@ -0,0 +1,481 @@ +package mail + +import ( + "bytes" + "fmt" + "io" + "log" + "mime" + "path/filepath" + "strings" + "time" + + "github.com/emersion/go-imap/v2" + "github.com/emersion/go-imap/v2/imapclient" + "github.com/emersion/go-message" + "mail2couch/config" +) + +// ImapClient wraps the IMAP client +type ImapClient struct { + *imapclient.Client +} + +// Message represents an email message retrieved from IMAP +type Message struct { + UID uint32 + From []string + To []string + Subject string + Date time.Time + Body string + Headers map[string][]string + Attachments []Attachment +} + +// Attachment represents an email attachment +type Attachment struct { + Filename string + ContentType string + Content []byte +} + +// NewImapClient creates a new IMAP client from the configuration +func NewImapClient(source *config.MailSource) (*ImapClient, error) { + addr := fmt.Sprintf("%s:%d", source.Host, source.Port) + + var client *imapclient.Client + var err error + + // Try TLS first for standard IMAPS ports (993, 465) + if source.Port == 993 || source.Port == 465 { + client, err = imapclient.DialTLS(addr, nil) + } else { + // Use insecure connection for other ports (143, 3143, etc.) + client, err = imapclient.DialInsecure(addr, nil) + } + + if err != nil { + return nil, fmt.Errorf("failed to dial IMAP server: %w", err) + } + + if err := client.Login(source.User, source.Password).Wait(); err != nil { + return nil, fmt.Errorf("failed to login: %w", err) + } + + return &ImapClient{client}, nil +} + +// ListMailboxes lists all available mailboxes +func (c *ImapClient) ListMailboxes() ([]string, error) { + var mailboxes []string + cmd := c.List("", "*", nil) + + infos, err := cmd.Collect() + if err != nil { + return nil, err + } + + for _, info := range infos { + mailboxes = append(mailboxes, info.Mailbox) + } + + return mailboxes, nil +} + +// GetMessages retrieves messages from a specific mailbox with filtering support +// Returns messages and a map of all current UIDs in the mailbox +// maxMessages: 0 means no limit, > 0 limits the number of messages to fetch +// since: if provided, only fetch messages newer than this date (for incremental sync) +func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages int, messageFilter *config.MessageFilter) ([]*Message, map[uint32]bool, error) { + // Select the mailbox + mbox, err := c.Select(mailbox, nil).Wait() + if err != nil { + return nil, nil, fmt.Errorf("failed to select mailbox %s: %w", mailbox, err) + } + + if mbox.NumMessages == 0 { + return []*Message{}, make(map[uint32]bool), nil + } + + var messages []*Message + currentUIDs := make(map[uint32]bool) + + // First, get all current UIDs in the mailbox for sync purposes + allUIDsSet := imap.SeqSet{} + allUIDsSet.AddRange(1, mbox.NumMessages) + + // Fetch UIDs for all messages to track current state + uidCmd := c.Fetch(allUIDsSet, &imap.FetchOptions{UID: true}) + for { + msg := uidCmd.Next() + if msg == nil { + break + } + + data, err := msg.Collect() + if err != nil { + continue + } + + if data.UID != 0 { + currentUIDs[uint32(data.UID)] = true + } + } + uidCmd.Close() + + // Determine which messages to fetch based on since date + var seqSet imap.SeqSet + + if since != nil { + // Use IMAP SEARCH to find messages since the specified date + searchCriteria := &imap.SearchCriteria{ + Since: *since, + } + + searchCmd := c.Search(searchCriteria, nil) + searchResults, err := searchCmd.Wait() + if err != nil { + log.Printf("IMAP SEARCH failed, falling back to fetch all: %v", err) + // Fall back to fetching all messages + numToFetch := mbox.NumMessages + if maxMessages > 0 && int(numToFetch) > maxMessages { + numToFetch = uint32(maxMessages) + } + seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) + } else { + // Convert search results to sequence set + searchSeqNums := searchResults.AllSeqNums() + if len(searchSeqNums) == 0 { + return []*Message{}, currentUIDs, nil + } + + // Limit results if maxMessages is specified + if maxMessages > 0 && len(searchSeqNums) > maxMessages { + searchSeqNums = searchSeqNums[len(searchSeqNums)-maxMessages:] + } + + for _, seqNum := range searchSeqNums { + seqSet.AddNum(seqNum) + } + } + } else { + // No since date specified, fetch recent messages up to maxMessages + numToFetch := mbox.NumMessages + if maxMessages > 0 && int(numToFetch) > maxMessages { + numToFetch = uint32(maxMessages) + } + + if numToFetch == 0 { + return []*Message{}, currentUIDs, nil + } + + // Fetch the most recent messages + seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) + } + + // Fetch message data - get envelope and full message body + options := &imap.FetchOptions{ + Envelope: true, + UID: true, + BodySection: []*imap.FetchItemBodySection{ + {}, // Empty section gets the entire message + }, + } + + fetchCmd := c.Fetch(seqSet, options) + + for { + msg := fetchCmd.Next() + if msg == nil { + break + } + + parsedMsg, err := c.parseMessage(msg) + if err != nil { + log.Printf("Failed to parse message: %v", err) + continue + } + + // Apply message-level keyword filtering + if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) { + continue // Skip this message due to keyword filter + } + + messages = append(messages, parsedMsg) + } + + if err := fetchCmd.Close(); err != nil { + return nil, nil, fmt.Errorf("failed to fetch messages: %w", err) + } + + return messages, currentUIDs, nil +} + +// parseMessage parses an IMAP fetch response into our Message struct +func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) { + msg := &Message{ + UID: fetchMsg.SeqNum, // Using sequence number for now + Headers: make(map[string][]string), + Attachments: []Attachment{}, + } + + // Collect all fetch data first + buffer, err := fetchMsg.Collect() + if err != nil { + return nil, fmt.Errorf("failed to collect fetch data: %w", err) + } + + // Parse envelope for basic headers + if buffer.Envelope != nil { + env := buffer.Envelope + msg.Subject = env.Subject + msg.Date = env.Date + + // Parse From addresses + for _, addr := range env.From { + if addr.Mailbox != "" { + fullAddr := addr.Mailbox + if addr.Host != "" { + fullAddr = addr.Mailbox + "@" + addr.Host + } + msg.From = append(msg.From, fullAddr) + } + } + + // Parse To addresses + for _, addr := range env.To { + if addr.Mailbox != "" { + fullAddr := addr.Mailbox + if addr.Host != "" { + fullAddr = addr.Mailbox + "@" + addr.Host + } + msg.To = append(msg.To, fullAddr) + } + } + } + + // Get UID if available + if buffer.UID != 0 { + msg.UID = uint32(buffer.UID) + } + + // Parse full message content + if len(buffer.BodySection) > 0 { + bodyBuffer := buffer.BodySection[0] + reader := bytes.NewReader(bodyBuffer.Bytes) + + // Parse the message using go-message + entity, err := message.Read(reader) + if err != nil { + return nil, fmt.Errorf("failed to parse message: %w", err) + } + + // Extract headers + header := entity.Header + for field := header.Fields(); field.Next(); { + key := field.Key() + value, _ := field.Text() + msg.Headers[key] = append(msg.Headers[key], value) + } + + // Parse message body and attachments + if err := c.parseMessageBody(entity, msg); err != nil { + return nil, fmt.Errorf("failed to parse message body: %w", err) + } + } + + return msg, nil +} + +// parseMessageBody extracts the body and attachments from a message entity +func (c *ImapClient) parseMessageBody(entity *message.Entity, msg *Message) error { + mediaType, _, err := entity.Header.ContentType() + if err != nil { + // Default to text/plain if no content type + mediaType = "text/plain" + } + + if strings.HasPrefix(mediaType, "multipart/") { + // Handle multipart message + mr := entity.MultipartReader() + if mr == nil { + return fmt.Errorf("failed to create multipart reader") + } + + for { + part, err := mr.NextPart() + if err == io.EOF { + break + } + if err != nil { + return fmt.Errorf("failed to read multipart: %w", err) + } + + if err := c.parseMessagePart(part, msg); err != nil { + log.Printf("Failed to parse message part: %v", err) + // Continue processing other parts + } + } + } else { + // Handle single part message + if err := c.parseMessagePart(entity, msg); err != nil { + return err + } + } + + return nil +} + +// parseMessagePart processes a single message part (body or attachment) +func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) error { + mediaType, params, err := entity.Header.ContentType() + if err != nil { + mediaType = "text/plain" + } + + // Get content disposition + disposition, dispositionParams, _ := entity.Header.ContentDisposition() + + // Determine if this is an attachment + isAttachment := disposition == "attachment" || + (disposition == "inline" && dispositionParams["filename"] != "") || + params["name"] != "" + + if isAttachment { + // Handle attachment + filename := dispositionParams["filename"] + if filename == "" { + filename = params["name"] + } + if filename == "" { + filename = "unnamed_attachment" + } + + // Decode filename if needed + decoder := &mime.WordDecoder{} + filename, _ = decoder.DecodeHeader(filename) + + // Read attachment content + content, err := io.ReadAll(entity.Body) + if err != nil { + return fmt.Errorf("failed to read attachment content: %w", err) + } + + attachment := Attachment{ + Filename: filename, + ContentType: mediaType, + Content: content, + } + + msg.Attachments = append(msg.Attachments, attachment) + } else if strings.HasPrefix(mediaType, "text/") && msg.Body == "" { + // Handle text body (only take the first text part as body) + bodyBytes, err := io.ReadAll(entity.Body) + if err != nil { + return fmt.Errorf("failed to read message body: %w", err) + } + msg.Body = string(bodyBytes) + } + + return nil +} + +// ShouldProcessMailbox checks if a mailbox should be processed based on filters with wildcard support +func (c *ImapClient) ShouldProcessMailbox(mailbox string, filter *config.FolderFilter) bool { + // If include list is specified, mailbox must match at least one pattern + if len(filter.Include) > 0 { + found := false + for _, pattern := range filter.Include { + // Handle special case: "*" means include all folders + if pattern == "*" { + found = true + break + } + // Use filepath.Match for wildcard pattern matching + if matched, err := filepath.Match(pattern, mailbox); err == nil && matched { + found = true + break + } + // Also support exact string matching for backwards compatibility + if mailbox == pattern { + found = true + break + } + } + if !found { + return false + } + } + + // If exclude list is specified, mailbox must not match any exclude pattern + for _, pattern := range filter.Exclude { + // Use filepath.Match for wildcard pattern matching + if matched, err := filepath.Match(pattern, mailbox); err == nil && matched { + return false + } + // Also support exact string matching for backwards compatibility + if mailbox == pattern { + return false + } + } + + return true +} + +// ShouldProcessMessage checks if a message should be processed based on keyword filters +func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool { + // Check subject keywords + if len(filter.SubjectKeywords) > 0 { + if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) { + return false + } + } + + // Check sender keywords + if len(filter.SenderKeywords) > 0 { + senderMatch := false + for _, sender := range msg.From { + if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) { + senderMatch = true + break + } + } + if !senderMatch { + return false + } + } + + // Check recipient keywords + if len(filter.RecipientKeywords) > 0 { + recipientMatch := false + for _, recipient := range msg.To { + if c.containsAnyKeyword(strings.ToLower(recipient), filter.RecipientKeywords) { + recipientMatch = true + break + } + } + if !recipientMatch { + return false + } + } + + return true +} + +// containsAnyKeyword checks if the text contains any of the specified keywords (case-insensitive) +func (c *ImapClient) containsAnyKeyword(text string, keywords []string) bool { + for _, keyword := range keywords { + if strings.Contains(text, strings.ToLower(keyword)) { + return true + } + } + return false +} + +// Logout logs the client out +func (c *ImapClient) Logout() { + if err := c.Client.Logout(); err != nil { + log.Printf("Failed to logout: %v", err) + } +} diff --git a/go/mail2couch b/go/mail2couch new file mode 100755 index 0000000..2133741 Binary files /dev/null and b/go/mail2couch differ diff --git a/go/main.go b/go/main.go new file mode 100644 index 0000000..8d4b661 --- /dev/null +++ b/go/main.go @@ -0,0 +1,205 @@ +package main + +import ( + "context" + "fmt" + "log" + "time" + + "mail2couch/config" + "mail2couch/couch" + "mail2couch/mail" +) + +func main() { + args := config.ParseCommandLine() + + cfg, err := config.LoadConfigWithDiscovery(args) + if err != nil { + log.Fatalf("Failed to load configuration: %v", err) + } + + // Initialize CouchDB client + couchClient, err := couch.NewClient(&cfg.CouchDb) + if err != nil { + log.Fatalf("Failed to create CouchDB client: %v", err) + } + + fmt.Printf("Found %d mail source(s) to process.\n", len(cfg.MailSources)) + for _, source := range cfg.MailSources { + if !source.Enabled { + continue + } + + // Generate per-account database name + dbName := couch.GenerateAccountDBName(source.Name, source.User) + + // Ensure the account-specific database exists + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + err = couchClient.EnsureDB(ctx, dbName) + cancel() + + if err != nil { + log.Printf("Could not ensure CouchDB database '%s' exists (is it running?): %v", dbName, err) + continue + } else { + fmt.Printf("CouchDB database '%s' is ready for account: %s\n", dbName, source.Name) + } + + fmt.Printf(" - Processing source: %s\n", source.Name) + if source.Protocol == "imap" { + err := processImapSource(&source, couchClient, dbName, args.MaxMessages) + if err != nil { + log.Printf(" ERROR: Failed to process IMAP source %s: %v", source.Name, err) + } + } + } +} + +func processImapSource(source *config.MailSource, couchClient *couch.Client, dbName string, maxMessages int) error { + fmt.Printf(" Connecting to IMAP server: %s:%d\n", source.Host, source.Port) + imapClient, err := mail.NewImapClient(source) + if err != nil { + return fmt.Errorf("failed to connect to IMAP server: %w", err) + } + defer imapClient.Logout() + + fmt.Println(" IMAP connection successful.") + + mailboxes, err := imapClient.ListMailboxes() + if err != nil { + return fmt.Errorf("failed to list mailboxes: %w", err) + } + + fmt.Printf(" Found %d mailboxes.\n", len(mailboxes)) + + // Parse the since date from config if provided (fallback for first sync) + var configSinceDate *time.Time + if source.MessageFilter.Since != "" { + parsed, err := time.Parse("2006-01-02", source.MessageFilter.Since) + if err != nil { + log.Printf(" WARNING: Invalid since date format '%s', ignoring filter", source.MessageFilter.Since) + } else { + configSinceDate = &parsed + } + } + + totalMessages := 0 + totalStored := 0 + + // Process each mailbox + for _, mailbox := range mailboxes { + // Check if this mailbox should be processed based on filters + if !imapClient.ShouldProcessMailbox(mailbox, &source.FolderFilter) { + fmt.Printf(" Skipping mailbox: %s (filtered)\n", mailbox) + continue + } + + fmt.Printf(" Processing mailbox: %s (mode: %s)\n", mailbox, source.Mode) + + // Get sync metadata to determine incremental sync date + syncCtx, syncCancel := context.WithTimeout(context.Background(), 10*time.Second) + syncMetadata, err := couchClient.GetSyncMetadata(syncCtx, dbName, mailbox) + syncCancel() + if err != nil { + log.Printf(" ERROR: Failed to get sync metadata for %s: %v", mailbox, err) + continue + } + + // Determine the since date for incremental sync + var sinceDate *time.Time + if syncMetadata != nil { + // Use last sync time for incremental sync + sinceDate = &syncMetadata.LastSyncTime + fmt.Printf(" Incremental sync since: %s (last synced %d messages)\n", + sinceDate.Format("2006-01-02 15:04:05"), syncMetadata.MessageCount) + } else { + // First sync - use config since date if available + sinceDate = configSinceDate + if sinceDate != nil { + fmt.Printf(" First sync since: %s (from config)\n", sinceDate.Format("2006-01-02")) + } else { + fmt.Printf(" First full sync (no date filter)\n") + } + } + + // Retrieve messages from the mailbox + messages, currentUIDs, err := imapClient.GetMessages(mailbox, sinceDate, maxMessages, &source.MessageFilter) + if err != nil { + log.Printf(" ERROR: Failed to get messages from %s: %v", mailbox, err) + continue + } + + // Perform sync/archive logic + mailboxSyncCtx, mailboxSyncCancel := context.WithTimeout(context.Background(), 30*time.Second) + err = couchClient.SyncMailbox(mailboxSyncCtx, dbName, mailbox, currentUIDs, source.IsSyncMode()) + mailboxSyncCancel() + if err != nil { + log.Printf(" ERROR: Failed to sync mailbox %s: %v", mailbox, err) + continue + } + + if len(messages) == 0 { + fmt.Printf(" No new messages found in %s\n", mailbox) + continue + } + + fmt.Printf(" Found %d messages in %s\n", len(messages), mailbox) + totalMessages += len(messages) + + // Convert messages to CouchDB documents + var docs []*couch.MailDocument + for _, msg := range messages { + doc := couch.ConvertMessage(msg, mailbox) + docs = append(docs, doc) + } + + // Store messages in CouchDB with attachments + ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) + stored := 0 + for i, doc := range docs { + err := couchClient.StoreMessage(ctx, dbName, doc, messages[i]) + if err != nil { + log.Printf(" ERROR: Failed to store message %s: %v", doc.ID, err) + } else { + stored++ + } + } + cancel() + + fmt.Printf(" Stored %d/%d messages from %s\n", stored, len(messages), mailbox) + totalStored += stored + + // Update sync metadata after successful processing + if len(messages) > 0 { + // Find the highest UID processed + var maxUID uint32 + for _, msg := range messages { + if msg.UID > maxUID { + maxUID = msg.UID + } + } + + // Create/update sync metadata + newMetadata := &couch.SyncMetadata{ + Mailbox: mailbox, + LastSyncTime: time.Now(), + LastMessageUID: maxUID, + MessageCount: stored, + } + + // Store sync metadata + metadataCtx, metadataCancel := context.WithTimeout(context.Background(), 10*time.Second) + err = couchClient.StoreSyncMetadata(metadataCtx, dbName, newMetadata) + metadataCancel() + if err != nil { + log.Printf(" WARNING: Failed to store sync metadata for %s: %v", mailbox, err) + } else { + fmt.Printf(" Updated sync metadata (last UID: %d)\n", maxUID) + } + } + } + + fmt.Printf(" Summary: Processed %d messages, stored %d new messages\n", totalMessages, totalStored) + return nil +} diff --git a/test/README.md b/test/README.md new file mode 100644 index 0000000..bd4c180 --- /dev/null +++ b/test/README.md @@ -0,0 +1,250 @@ +# mail2couch Test Environment + +This directory contains a complete test environment for mail2couch using Podman containers. + +## Overview + +The test environment provides: +- **CouchDB**: Database for storing email messages +- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages +- **Test Configuration**: Ready-to-use config for testing both sync and archive modes + +## Quick Start + +### Run Basic Integration Tests +```bash +./run-tests.sh +``` +This comprehensive test will: +1. Start all containers with cleanup +2. Populate test data +3. Build and run mail2couch +4. Verify database creation and document storage +5. Test incremental sync behavior +6. Clean up automatically + +### Run Wildcard Pattern Tests +```bash +./test-wildcard-patterns.sh +``` +This will test various wildcard folder patterns including: +- `*` (all folders) +- `Work*` (prefix patterns) +- `*/Drafts` (subfolder patterns) +- Complex include/exclude combinations + +### Run Incremental Sync Tests +```bash +./test-incremental-sync.sh +``` +This will test incremental synchronization functionality: +- First sync establishes baseline +- New messages are added to test accounts +- Second sync should only fetch new messages +- Sync metadata tracking and IMAP SEARCH with SINCE + +### Manual Testing Environment +```bash +# Start persistent test environment (for manual experimentation) +./start-test-env.sh + +# Run mail2couch manually with different configurations +cd ../go +./mail2couch -config ../test/config-test.json +./mail2couch -config ../test/config-wildcard-examples.json + +# Stop test environment when done +cd ../test +./stop-test-env.sh +``` + +## Test Scripts Overview + +### Automated Testing (Recommended) +- **`./run-tests.sh`**: Complete integration test with automatic cleanup + - Starts containers, populates data, runs mail2couch, verifies results + - Tests basic functionality, database creation, and incremental sync + - Cleans up automatically - perfect for CI/CD or quick validation + +### Specialized Feature Testing +- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing + - Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns + - Self-contained with own setup/teardown +- **`./test-incremental-sync.sh`**: Incremental synchronization testing + - Tests sync metadata tracking and IMAP SEARCH with SINCE + - Multi-step validation: baseline sync → add messages → incremental sync + - Self-contained with own setup/teardown + +### Manual Testing Environment +- **`./start-test-env.sh`**: Start persistent test containers + - Keeps environment running for manual experimentation + - Populates test data once + - Use with different configurations for development +- **`./stop-test-env.sh`**: Clean up manual test environment + - Only needed after using `start-test-env.sh` + +## Test Accounts + +The test environment includes these IMAP accounts: + +| Username | Password | Mode | Folder Pattern | Purpose | +|----------|----------|------|---------------|---------| +| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test | +| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test | +| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test | +| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test | + +Each account contains: +- 10 messages in INBOX (every 3rd has an attachment) +- 3 messages in each additional folder +- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash` +- Various message types for comprehensive wildcard testing + +## Services + +### CouchDB +- **URL**: http://localhost:5984 +- **Admin**: `admin` / `password` +- **Web UI**: http://localhost:5984/_utils + +### GreenMail Server +- **Host**: localhost +- **IMAP Port**: 3143 (plain) +- **IMAPS Port**: 3993 (SSL) +- **SMTP Port**: 3025 +- **Server**: GreenMail (Java-based test server) + +## Database Structure + +mail2couch will create separate databases for each mail source (with `m2c_` prefix): +- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode) +- `m2c_work_pattern_test` - Work Pattern Test (sync mode) +- `m2c_specific_folders_only` - Specific Folders Only (archive mode) +- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode) + +Each database contains documents with: +- `mailbox` field indicating the origin folder +- Native CouchDB attachments for email attachments +- Full message headers and body content + +## Testing Sync vs Archive Modes + +### Sync Mode (`syncuser`) +- Database exactly matches mail account +- If messages are deleted from IMAP, they're removed from CouchDB +- 1-to-1 relationship + +### Archive Mode (`archiveuser`, `testuser1`) +- Database preserves all messages ever seen +- Messages deleted from IMAP remain in CouchDB +- Archive/backup behavior + +## Wildcard Pattern Examples + +The test environment demonstrates these wildcard patterns: + +### All Folders Pattern (`*`) +```json +{ + "folderFilter": { + "include": ["*"], + "exclude": ["Drafts", "Trash"] + } +} +``` +Processes all folders except Drafts and Trash. + +### Work Pattern (`Work*`) +```json +{ + "folderFilter": { + "include": ["Work*", "Important*", "INBOX"], + "exclude": ["*Temp*"] + } +} +``` +Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp. + +### Specific Folders +```json +{ + "folderFilter": { + "include": ["INBOX", "Sent", "Personal"], + "exclude": [] + } +} +``` +Only processes the exact named folders. + +### Subfolder Pattern (`Work/*`) +```json +{ + "folderFilter": { + "include": ["Work/*", "Archive/*"], + "exclude": ["*/Drafts"] + } +} +``` +Includes all subfolders under Work and Archive, but excludes any Drafts subfolder. + +## File Structure + +``` +test/ +├── podman-compose.yml # Container orchestration (GreenMail + CouchDB) +├── config-test.json # Main test configuration with wildcard examples +├── config-wildcard-examples.json # Advanced wildcard patterns +├── run-tests.sh # Automated integration test (recommended) +├── test-wildcard-patterns.sh # Specialized wildcard pattern testing +├── test-incremental-sync.sh # Specialized incremental sync testing +├── start-test-env.sh # Start persistent test environment +├── stop-test-env.sh # Stop test environment +├── populate-greenmail.py # Create test messages across multiple folders +├── dovecot/ # Dovecot configuration (legacy, unused) +└── README.md # This file +``` + +## Prerequisites + +- Podman and podman-compose +- OpenSSL (for certificate generation) +- curl and nc (for connectivity checks) +- Go (for building mail2couch) + +## Troubleshooting + +### Containers won't start +```bash +# Check podman status +podman ps -a + +# View logs +podman logs mail2couch_test_couchdb +podman logs mail2couch_test_imap +``` + +### CouchDB connection issues +- Verify CouchDB is running: `curl http://localhost:5984` +- Check admin credentials: `admin/password` + +### IMAP connection issues +- Test IMAP connection: `nc -z localhost 143` +- Check Dovecot logs: `podman logs mail2couch_test_imap` + +### Permission issues +- Ensure scripts are executable: `chmod +x *.sh` +- Check file permissions in dovecot directory + +## Advanced Usage + +### Add custom test messages +Edit `populate-test-messages.sh` to create additional test scenarios. + +### Modify IMAP configuration +Edit `dovecot/dovecot.conf` and restart containers. + +### Test with SSL +Update `config-test.json` to use port 993 and enable SSL. + +### Custom test scenarios +Create additional configuration files for specific test cases. \ No newline at end of file diff --git a/test/config-test.json b/test/config-test.json new file mode 100644 index 0000000..e618c2e --- /dev/null +++ b/test/config-test.json @@ -0,0 +1,74 @@ +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "Wildcard All Folders Test", + "enabled": true, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "testuser1", + "password": "password123", + "mode": "archive", + "folderFilter": { + "include": ["*"], + "exclude": ["Drafts", "Trash"] + }, + "messageFilter": { + "subjectKeywords": ["meeting", "important"], + "senderKeywords": ["@company.com"] + } + }, + { + "name": "Work Pattern Test", + "enabled": true, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "syncuser", + "password": "syncpass", + "mode": "sync", + "folderFilter": { + "include": ["Work*", "Important*", "INBOX"], + "exclude": ["*Temp*"] + }, + "messageFilter": { + "recipientKeywords": ["support@", "team@"] + } + }, + { + "name": "Specific Folders Only", + "enabled": true, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "archiveuser", + "password": "archivepass", + "mode": "archive", + "folderFilter": { + "include": ["INBOX", "Sent", "Personal"], + "exclude": [] + }, + "messageFilter": {} + }, + { + "name": "Subfolder Pattern Test", + "enabled": false, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "testuser2", + "password": "password456", + "mode": "archive", + "folderFilter": { + "include": ["Work/*", "Archive/*"], + "exclude": ["*/Drafts"] + }, + "messageFilter": {} + } + ] +} \ No newline at end of file diff --git a/test/config-wildcard-examples.json b/test/config-wildcard-examples.json new file mode 100644 index 0000000..9db4df1 --- /dev/null +++ b/test/config-wildcard-examples.json @@ -0,0 +1,71 @@ +{ + "couchDb": { + "url": "http://localhost:5984", + "user": "admin", + "password": "password" + }, + "mailSources": [ + { + "name": "All Folders Example", + "enabled": true, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "testuser1", + "password": "password123", + "mode": "archive", + "folderFilter": { + "include": ["*"], + "exclude": ["Drafts", "Trash", "Spam"] + }, + "messageFilter": {} + }, + { + "name": "Inbox and Sent Only", + "enabled": false, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "testuser2", + "password": "password456", + "mode": "sync", + "folderFilter": { + "include": ["INBOX", "Sent"], + "exclude": [] + }, + "messageFilter": {} + }, + { + "name": "Work Folders Pattern", + "enabled": false, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "workuser", + "password": "workpass", + "mode": "archive", + "folderFilter": { + "include": ["Work*", "Projects*", "INBOX"], + "exclude": ["*Temp*", "*Draft*"] + }, + "messageFilter": { + "senderKeywords": ["@company.com"] + } + }, + { + "name": "Archive Pattern Example", + "enabled": false, + "protocol": "imap", + "host": "localhost", + "port": 3143, + "user": "archiveuser", + "password": "archivepass", + "mode": "archive", + "folderFilter": { + "include": ["Archive*", "*Important*", "INBOX"], + "exclude": ["*Temp"] + }, + "messageFilter": {} + } + ] +} \ No newline at end of file diff --git a/test/dovecot/dovecot.conf b/test/dovecot/dovecot.conf new file mode 100644 index 0000000..af20de0 --- /dev/null +++ b/test/dovecot/dovecot.conf @@ -0,0 +1,79 @@ +# Dovecot configuration for testing mail2couch + +# Basic settings +protocols = imap +listen = * + +# SSL/TLS settings - make optional for easier testing +ssl = optional +ssl_cert = /dev/null 2>&1; then + addgroup -g 97 dovecot + adduser -D -u 97 -G dovecot -s /sbin/nologin dovecot +fi + +# Set proper ownership +chown -R dovecot:dovecot /var/mail +chown -R dovecot:dovecot /var/run/dovecot +chown -R root:dovecot /etc/dovecot +chmod -R 0640 /etc/dovecot +chmod 0644 /etc/dovecot/dovecot.conf + +# Generate SSL certificates if they don't exist +if [ ! -f /etc/dovecot/ssl/server.crt ] || [ ! -f /etc/dovecot/ssl/server.key ]; then + echo "Generating SSL certificates..." + mkdir -p /etc/dovecot/ssl + + # Generate DH parameters (small for testing) + openssl dhparam -out /etc/dovecot/ssl/dh.pem 1024 + + # Generate private key + openssl genrsa -out /etc/dovecot/ssl/server.key 2048 + + # Generate certificate + openssl req -new -key /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.csr -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost" + openssl x509 -req -days 365 -in /etc/dovecot/ssl/server.csr -signkey /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.crt + rm /etc/dovecot/ssl/server.csr +fi + +# Ensure SSL directory permissions +chown -R dovecot:dovecot /etc/dovecot/ssl +chmod 600 /etc/dovecot/ssl/server.key +chmod 644 /etc/dovecot/ssl/server.crt + +echo "Starting Dovecot..." +exec dovecot -F \ No newline at end of file diff --git a/test/dovecot/passwd b/test/dovecot/passwd new file mode 100644 index 0000000..51e3cc3 --- /dev/null +++ b/test/dovecot/passwd @@ -0,0 +1,8 @@ +# Password database for Dovecot testing +# Format: username:password + +# Test accounts with simple passwords for testing +testuser1:password123 +testuser2:password456 +syncuser:syncpass +archiveuser:archivepass \ No newline at end of file diff --git a/test/dovecot/ssl/dh.pem b/test/dovecot/ssl/dh.pem new file mode 100644 index 0000000..513492c --- /dev/null +++ b/test/dovecot/ssl/dh.pem @@ -0,0 +1,8 @@ +-----BEGIN DH PARAMETERS----- +MIIBDAKCAQEAjcUSAHFs60qgDRg/cT7byhuhF3vwZQhmm1QToCFgG4VWu/EOVXq2 +kHxjxmo3hBuJCqUZqTAyF91Tum7A2QuQhXFrxOpRF8EiyVSgBabjN/WcEHIow1uh +Vtb4JOcDl/Q9IJfFT6zyXdQQiHPBOWnpOBKXeQQQIx5plgsrmK0cTO2ZxtyrmHHp +wxtE3INKYuBlGH3Y0zghc+Hoezpf/hbIHZibGQ0l79EtBDQjqmqoDJCIiv5gsTt8 +9VpkR6FFvjWTNOb5qY10W/PRhLGjioX29bp1B6qW5PNJcd//cqrBLebKlkAoXnyx +x0uTUy6pmmIt5vdYxx0symrMXZEjrL7uzwIBAgICAOE= +-----END DH PARAMETERS----- diff --git a/test/dovecot/ssl/server.crt b/test/dovecot/ssl/server.crt new file mode 100644 index 0000000..5641479 --- /dev/null +++ b/test/dovecot/ssl/server.crt @@ -0,0 +1,20 @@ +-----BEGIN CERTIFICATE----- +MIIDVzCCAj+gAwIBAgIUYvkZSKbVH08s/3B70AW8IEpTB/kwDQYJKoZIhvcNAQEL +BQAwVDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx +EzARBgNVBAoMCk1haWwyQ291Y2gxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yNTA4 +MDExNDIyMzVaFw0yNjA4MDExNDIyMzVaMFQxCzAJBgNVBAYTAlVTMQ0wCwYDVQQI +DARUZXN0MQ0wCwYDVQQHDARUZXN0MRMwEQYDVQQKDApNYWlsMkNvdWNoMRIwEAYD +VQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs +leQcVLlKx7/bJGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vx +Td1mVCm9Lsd9bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDW +knco7/MEatakliVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331 +HEFdIqqccpimnIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwo +eN8zBzGFJe9xzqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5 +LGN5xSzXvFIzgP20y5s5AgMBAAGjITAfMB0GA1UdDgQWBBTkgYZGp2s74D+1ltyl +rudF/o7jODANBgkqhkiG9w0BAQsFAAOCAQEATc6ekhuk32meLuxhalz6lNwBxfDg +EG3gGUxNwehwgiNCcKIKQFtCwjJde6drOobkRDANtb7g3gSlAxlUCPsO6xnL1c6E +HhehFn++7HOpXvmEy/mnoqBL6PLzRZRMRlDynlPVV9Y82zsdrQiQEhGyNTfgP5dk +u9RMIMQl1hIK381V738b5MXfdpYhmRiTGEd6hCxCnzkx0OakCLM9lnJASr0dYPuh +LYKoClxhr3sV/JsgAmx91BuHGpzaPYQ2zFvCJSqD2ihM7zIl9K2bLIUR87/CznyH +JuPRbgt6/cxzwdqflP73j+TTZdlI4gckEA3H0WhNN4nB2SEjTgS+kDctMA== +-----END CERTIFICATE----- diff --git a/test/dovecot/ssl/server.key b/test/dovecot/ssl/server.key new file mode 100644 index 0000000..b7dd757 --- /dev/null +++ b/test/dovecot/ssl/server.key @@ -0,0 +1,28 @@ +-----BEGIN PRIVATE KEY----- +MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsleQcVLlKx7/b +JGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vxTd1mVCm9Lsd9 +bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDWknco7/MEatak +liVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331HEFdIqqccpim +nIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwoeN8zBzGFJe9x +zqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5LGN5xSzXvFIz +gP20y5s5AgMBAAECggEAOfBaM76ns/iqKpYlamnjfIs7svotEjhrHcsub6fWvEsE +XLzRmSqHeWP+t55oo2XeL2zOCofvuUDGnQ+rXE2mHwzhP3FJzsOibm2qmtCJvZwe +ozRj4xTMLILGDnGRhHAJ21cxZM9lPNLnOzri0868DeYimib49xAdroLBLyKRgDGN +O7OAwA4KWAebZRBU8cowEF87WGAI2hOLJA5WIKX7X9SiRDx5sNnDpTpqJvyvuleR +D/wKGHzkQiZx8WNJx5A571dilfKEp8S49o4sJdz8DQ/4ruVov2Vi+nSdMEzHnp5m +M7ZlZ7IcJRJDKPrDYasmxvtM8EiKyJf87/DPANfYFwKBgQDa+ANr2pGg2oMJVJIq +r/mj8wtRsLRYFgfs96BUwp7e9Vo+Rx+E0uZxGGMul6hn4b2/TQljLfiX/CP8ZTyw +MStlNlnAXaF425JNuQIFQ2wGGiGdsx2I1WNw7co4UPVVjH11nr3o30g+NDJcp7Tq +rvpKvGbeoZtHzOj0bF7fA/B7qwKBgQDJxcUUEH5A50n3oUP99aK/DzSA7kcte2Aw +tjv9hbPnbOmcM24Fm+KU7bsRYt9QPa0PU2lV3O1KrHj4q+QRcPFl2P2mzZC+Hzmx +H8dEjMmH8YdrjGqethMoUHJCguNfskNwjgWFlxTSBLY+NffghXNzZgiF9d6WqF48 +iqwH+HsAqwKBgQCb/B2D0Xn4WnEKToKpgh6WGmcv1G9EaL1Qo75FYzcFoUaeItBj +MFIUssjEwiinh/pBssFDM9Zpfqar//pRkVVWjnc1P/3tOI1qbKbx1Ou5FRhpXNVn +SovCQMLTh2idfq1JAsJKh/TQyyItOxL4M5n9b2Tgp8MUTPaOWDzlJctEbQKBgEVu +oNq+sjNzY6iq/dKubEqC2PZlCGlGQ1t/2jTrhXTlrZ3qtLmJYvcMt4rMEzxxfNQB +SAYb+CvyHc60l87Ipsj9WovDwUMrS5b/8HpOWCtHmeoQb8Adt4nv5OGuWL/dgAeD +V7MYwjljFbNiruG8CnZzbgtrCCWf2o3KylgT0X/xAoGAUhSdBge5Vpg0JcT1VDgm +q5rgc6dD1LJtXfBaq3w4kHYK/iLFcPOLUKcIJXNbhMwWza/JwVYK6hsCIw3/b4va +NhJ8ABpC3fZqkl28glEF8bnrPAkE1akn2GiBaaEbTCQRMrhZ2SW3JCyjX6yCvvvz +m7b2ZpDMJEMIBmgrK70E3Oo= +-----END PRIVATE KEY----- diff --git a/test/dovecot/users b/test/dovecot/users new file mode 100644 index 0000000..322dfc1 --- /dev/null +++ b/test/dovecot/users @@ -0,0 +1,8 @@ +# User database for Dovecot testing +# Format: username:uid:gid:home:shell + +# Test user accounts +testuser1::1001:1001:/var/mail/testuser1:: +testuser2::1002:1002:/var/mail/testuser2:: +syncuser::1003:1003:/var/mail/syncuser:: +archiveuser::1004:1004:/var/mail/archiveuser:: \ No newline at end of file diff --git a/test/generate-ssl.sh b/test/generate-ssl.sh new file mode 100755 index 0000000..cfa0c0c --- /dev/null +++ b/test/generate-ssl.sh @@ -0,0 +1,28 @@ +#!/bin/bash + +# Generate SSL certificates for Dovecot testing + +set -e + +CERT_DIR="dovecot/ssl" +mkdir -p "$CERT_DIR" + +# Generate DH parameters +echo "Generating DH parameters..." +openssl dhparam -out "$CERT_DIR/dh.pem" 2048 + +# Generate private key +echo "Generating private key..." +openssl genrsa -out "$CERT_DIR/server.key" 2048 + +# Generate certificate signing request +echo "Generating certificate..." +openssl req -new -key "$CERT_DIR/server.key" -out "$CERT_DIR/server.csr" -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost" + +# Generate self-signed certificate +openssl x509 -req -days 365 -in "$CERT_DIR/server.csr" -signkey "$CERT_DIR/server.key" -out "$CERT_DIR/server.crt" + +# Clean up CSR +rm "$CERT_DIR/server.csr" + +echo "SSL certificates generated successfully in $CERT_DIR/" \ No newline at end of file diff --git a/test/podman-compose.yml b/test/podman-compose.yml new file mode 100644 index 0000000..ba24506 --- /dev/null +++ b/test/podman-compose.yml @@ -0,0 +1,38 @@ +version: '3.8' + +services: + # CouchDB for testing + couchdb: + image: docker.io/couchdb:3.3 + container_name: mail2couch_test_couchdb + environment: + - COUCHDB_USER=admin + - COUCHDB_PASSWORD=password + ports: + - "5984:5984" + volumes: + - couchdb_data:/opt/couchdb/data + networks: + - mail2couch_test + + # GreenMail IMAP server for testing + greenmail: + image: docker.io/greenmail/standalone:2.0.1 + container_name: mail2couch_test_imap + ports: + - "3143:3143" # IMAP + - "3993:3993" # IMAPS + - "3025:3025" # SMTP + environment: + - GREENMAIL_OPTS=-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.users=testuser1:password123@localhost,testuser2:password456@localhost,syncuser:syncpass@localhost,archiveuser:archivepass@localhost + networks: + - mail2couch_test + depends_on: + - couchdb + +volumes: + couchdb_data: + +networks: + mail2couch_test: + driver: bridge \ No newline at end of file diff --git a/test/populate-greenmail.py b/test/populate-greenmail.py new file mode 100755 index 0000000..c79b593 --- /dev/null +++ b/test/populate-greenmail.py @@ -0,0 +1,179 @@ +#!/usr/bin/env python3 + +import imaplib +import email +from email.mime.text import MIMEText +from email.mime.multipart import MIMEMultipart +from email.mime.base import MIMEBase +from email import encoders +import time +import sys + +def create_simple_message(subject, body, from_addr="test-sender@example.com", to_addr="user@example.com"): + """Create a simple text message""" + msg = MIMEText(body) + msg['Subject'] = subject + msg['From'] = from_addr + msg['To'] = to_addr + msg['Date'] = email.utils.formatdate(localtime=True) + return msg.as_string() + +def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com", to_addr="user@example.com"): + """Create a message with an attachment""" + msg = MIMEMultipart() + msg['Subject'] = subject + msg['From'] = from_addr + msg['To'] = to_addr + msg['Date'] = email.utils.formatdate(localtime=True) + + # Add body + msg.attach(MIMEText(body, 'plain')) + + # Add attachment + part = MIMEBase('text', 'plain') + part.set_payload(attachment_content) + encoders.encode_base64(part) + part.add_header('Content-Disposition', 'attachment; filename="attachment.txt"') + msg.attach(part) + + return msg.as_string() + +def populate_user_mailbox(username, password, host='localhost', port=3143): + """Populate a user's mailbox with test messages""" + print(f"Connecting to {username}@{host}:{port}") + + try: + # Connect to IMAP server + imap = imaplib.IMAP4(host, port) + imap.login(username, password) + + # Create additional folders for testing wildcards + # These folders are designed to test various wildcard patterns + test_folders = [ + 'Sent', # Exact match + 'Work/Projects', # Work/* pattern + 'Work/Archive', # Work/* pattern + 'Work/Temp', # Work/* but excluded by *Temp* + 'Personal', # Exact match + 'Important/Urgent', # Important/* pattern + 'Important/Meetings', # Important/* pattern + 'Archive/2024', # Archive/* pattern + 'Archive/Projects', # Archive/* pattern + 'Archive/Drafts', # Archive/* but excluded by */Drafts + 'Drafts', # Should be excluded + 'Trash' # Should be excluded + ] + created_folders = ['INBOX'] # INBOX always exists + + for folder in test_folders: + try: + imap.create(folder) + created_folders.append(folder) + print(f" Created folder: {folder}") + except Exception as e: + print(f" Folder {folder} may already exist or creation failed: {e}") + # Try to select it to see if it exists + try: + imap.select(folder) + created_folders.append(folder) + print(f" Folder {folder} already exists") + except: + pass + + print(f"Available folders for {username}: {created_folders}") + + # Populate each folder with messages + for folder in created_folders[:3]: # Limit to first 3 folders to avoid too many messages + imap.select(folder) + print(f"Creating messages in {folder} for {username}...") + + # Create fewer messages per folder for testing + message_count = 3 if folder != 'INBOX' else 10 + + for i in range(1, message_count + 1): + if i % 3 == 0 and folder == 'INBOX': + # Every 3rd message has attachment (only in INBOX) + msg = create_message_with_attachment( + f"[{folder}] Test Message {i} with Attachment", + f"This is test message {i} in {folder} for {username} with an attachment.", + f"Sample attachment content for message {i} in {folder}", + "test-sender@example.com", + f"{username}@example.com" + ) + print(f" Created message {i} (with attachment)") + else: + # Simple message + msg = create_simple_message( + f"[{folder}] Test Message {i}", + f"This is test message {i} in {folder} for {username}.", + "test-sender@example.com", + f"{username}@example.com" + ) + print(f" Created message {i}") + + # Append message to current folder + imap.append(folder, None, None, msg.encode('utf-8')) + time.sleep(0.1) # Small delay to avoid overwhelming + + # Add special messages only to INBOX for keyword filtering tests + if folder == 'INBOX': + special_messages = [ + ("Important Meeting Reminder", "This is an important meeting message for testing keyword filters.", "manager@company.com", "team@company.com"), + ("Urgent: System Maintenance", "Important notification about system maintenance.", "admin@company.com", f"{username}@example.com"), + ("Regular Newsletter", "This is a regular newsletter message.", "newsletter@external.com", f"{username}@example.com"), + ("Team Meeting Notes", "Meeting notes from the team.", "secretary@company.com", "support@company.com"), + ("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀", "test-sender@example.com", f"{username}@example.com") + ] + + for subject, body, sender, recipient in special_messages: + msg = create_simple_message(subject, body, sender, recipient) + imap.append(folder, None, None, msg.encode('utf-8')) + print(f" Created special message: {subject} from {sender} to {recipient}") + time.sleep(0.1) + + imap.logout() + print(f"✅ Successfully created messages across {len(created_folders[:3])} folders for {username}") + return True + + except Exception as e: + print(f"❌ Error populating {username}: {e}") + return False + +def main(): + print("🚀 Populating GreenMail with test messages using IMAP...") + + # Test accounts + accounts = [ + ("testuser1", "password123"), + ("testuser2", "password456"), + ("syncuser", "syncpass"), + ("archiveuser", "archivepass") + ] + + # Wait for GreenMail to be ready + print("Waiting for GreenMail to be ready...") + time.sleep(5) + + success_count = 0 + for username, password in accounts: + if populate_user_mailbox(username, password): + success_count += 1 + time.sleep(1) # Brief pause between accounts + + print(f"\n🎉 Successfully populated {success_count}/{len(accounts)} accounts!") + + if success_count == len(accounts): + print("\n✅ All test accounts ready:") + for username, password in accounts: + print(f" - {username}:{password}@localhost") + print(f"\nGreenMail Services:") + print(f" - IMAP: localhost:3143") + print(f" - IMAPS: localhost:3993") + print(f" - SMTP: localhost:3025") + return 0 + else: + print(f"\n❌ Failed to populate {len(accounts) - success_count} accounts") + return 1 + +if __name__ == "__main__": + sys.exit(main()) \ No newline at end of file diff --git a/test/run-tests.sh b/test/run-tests.sh new file mode 100755 index 0000000..79d5a75 --- /dev/null +++ b/test/run-tests.sh @@ -0,0 +1,128 @@ +#!/bin/bash + +# Run basic integration tests with test containers +# This is a comprehensive test that handles its own setup and teardown + +set -e + +cd "$(dirname "$0")" + +echo "🚀 Running basic integration tests..." + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' # No Color + +# Function to print colored output +print_status() { + echo -e "${GREEN}[INFO]${NC} $1" +} + +print_warning() { + echo -e "${YELLOW}[WARN]${NC} $1" +} + +print_error() { + echo -e "${RED}[ERROR]${NC} $1" +} + +# Cleanup function +cleanup() { + print_status "Cleaning up test containers..." + podman-compose -f podman-compose.yml down -v 2>/dev/null || true +} + +# Set up cleanup trap +trap cleanup EXIT + +# Start containers +print_status "Starting test containers..." +podman-compose -f podman-compose.yml up -d + +# Wait for containers to be ready +print_status "Waiting for containers to be ready..." +sleep 10 + +# Check if CouchDB is ready +print_status "Checking CouchDB connectivity..." +timeout=30 +while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do + timeout=$((timeout - 1)) + if [ $timeout -le 0 ]; then + print_error "CouchDB failed to start within 30 seconds" + exit 1 + fi + sleep 1 +done +print_status "CouchDB is ready!" + +# Check if IMAP server is ready +print_status "Checking IMAP server connectivity..." +timeout=30 +while ! nc -z localhost 3143 > /dev/null 2>&1; do + timeout=$((timeout - 1)) + if [ $timeout -le 0 ]; then + print_error "IMAP server failed to start within 30 seconds" + exit 1 + fi + sleep 1 +done +print_status "IMAP server is ready!" + +# Populate test messages +print_status "Populating test messages..." +python3 ./populate-greenmail.py + +# Build mail2couch +print_status "Building mail2couch..." +cd ../go +go build -o mail2couch . +cd ../test + +# Run mail2couch with test configuration +print_status "Running mail2couch with test configuration..." +../go/mail2couch -config config-test.json -max-messages 3 + +# Verify results +print_status "Verifying test results..." + +# Check CouchDB databases were created (using correct database names with m2c prefix) +EXPECTED_DBS=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only") + +for db in "${EXPECTED_DBS[@]}"; do + if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then + print_status "✅ Database '$db' created successfully" + else + print_error "❌ Database '$db' was not created" + exit 1 + fi +done + +# Check document counts +for db in "${EXPECTED_DBS[@]}"; do + doc_count=$(curl -s "http://admin:password@localhost:5984/$db" | grep -o '"doc_count":[0-9]*' | cut -d':' -f2) + if [ "$doc_count" -gt 0 ]; then + print_status "✅ Database '$db' contains $doc_count documents" + else + print_warning "⚠️ Database '$db' contains no documents" + fi +done + +# Test sync mode by running again (should show incremental behavior) +print_status "Running mail2couch again to test incremental sync..." +../go/mail2couch -config config-test.json -max-messages 3 + +print_status "🎉 Basic integration tests completed successfully!" + +# Show summary +print_status "Test Summary:" +echo " - IMAP Server: localhost:3143" +echo " - CouchDB: http://localhost:5984" +echo " - Test accounts: testuser1, syncuser, archiveuser" +echo " - Databases created: ${EXPECTED_DBS[*]}" +echo "" +echo "For more comprehensive tests, run:" +echo " - ./test-wildcard-patterns.sh (test folder pattern matching)" +echo " - ./test-incremental-sync.sh (test incremental synchronization)" \ No newline at end of file diff --git a/test/start-test-env.sh b/test/start-test-env.sh new file mode 100755 index 0000000..2e6a82c --- /dev/null +++ b/test/start-test-env.sh @@ -0,0 +1,67 @@ +#!/bin/bash + +# Start test environment for manual testing + +cd "$(dirname "$0")" + +echo "🚀 Starting mail2couch test environment..." + +# Start containers +echo "Starting containers..." +podman-compose -f podman-compose.yml up -d + +# Wait for services +echo "Waiting for services to be ready..." +sleep 10 + +# Check CouchDB +echo "Checking CouchDB..." +timeout=30 +while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do + timeout=$((timeout - 1)) + if [ $timeout -le 0 ]; then + echo "❌ CouchDB failed to start" + exit 1 + fi + sleep 1 +done +echo "✅ CouchDB is ready at http://localhost:5984" + +# Check IMAP +echo "Checking IMAP server..." +timeout=30 +while ! nc -z localhost 3143 > /dev/null 2>&1; do + timeout=$((timeout - 1)) + if [ $timeout -le 0 ]; then + echo "❌ IMAP server failed to start" + exit 1 + fi + sleep 1 +done +echo "✅ IMAP server is ready at localhost:3143" + +# Populate test data +echo "Populating test messages..." +python3 ./populate-greenmail.py + +echo "" +echo "🎉 Test environment is ready!" +echo "" +echo "Services:" +echo " - CouchDB: http://localhost:5984 (admin/password)" +echo " - CouchDB Web UI: http://localhost:5984/_utils" +echo " - IMAP Server: localhost:3143" +echo " - IMAPS Server: localhost:3993" +echo " - SMTP Server: localhost:3025" +echo "" +echo "Test accounts:" +echo " - testuser1:password123" +echo " - testuser2:password456" +echo " - syncuser:syncpass" +echo " - archiveuser:archivepass" +echo "" +echo "To run mail2couch:" +echo " cd ../go && ./mail2couch -config ../test/config-test.json" +echo "" +echo "To stop the environment:" +echo " ./stop-test-env.sh" \ No newline at end of file diff --git a/test/stop-test-env.sh b/test/stop-test-env.sh new file mode 100755 index 0000000..93c08b5 --- /dev/null +++ b/test/stop-test-env.sh @@ -0,0 +1,12 @@ +#!/bin/bash + +# Stop test environment + +cd "$(dirname "$0")" + +echo "🛑 Stopping mail2couch test environment..." + +# Stop and remove containers +podman-compose -f podman-compose.yml down -v + +echo "✅ Test environment stopped and cleaned up!" \ No newline at end of file diff --git a/test/test-incremental-sync.sh b/test/test-incremental-sync.sh new file mode 100755 index 0000000..bc35016 --- /dev/null +++ b/test/test-incremental-sync.sh @@ -0,0 +1,242 @@ +#!/bin/bash + +# Test script to validate incremental sync functionality +# This script tests that mail2couch properly implements incremental synchronization + +set -e + +echo "🔄 Testing Incremental Sync Functionality" +echo "==========================================" + +# Make sure we're in the right directory +cd "$(dirname "$0")/.." + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Function to check if containers are running +check_containers() { + echo "🔍 Checking if test containers are running..." + + if ! podman ps | grep -q "greenmail"; then + echo -e "${RED}❌ GreenMail container not running${NC}" + echo "Please run: cd test && ./start-test-env.sh" + exit 1 + fi + + if ! podman ps | grep -q "couchdb"; then + echo -e "${RED}❌ CouchDB container not running${NC}" + echo "Please run: cd test && ./start-test-env.sh" + exit 1 + fi + + echo -e "${GREEN}✅ Test containers are running${NC}" +} + +# Function to populate initial test data +populate_initial_data() { + echo "📧 Populating initial test data..." + cd test + if python3 populate-greenmail.py; then + echo -e "${GREEN}✅ Initial test data populated${NC}" + else + echo -e "${RED}❌ Failed to populate initial test data${NC}" + exit 1 + fi + cd .. +} + +# Function to build the application +build_app() { + echo "🔨 Building mail2couch..." + cd go + if go build -o mail2couch .; then + echo -e "${GREEN}✅ Build successful${NC}" + else + echo -e "${RED}❌ Build failed${NC}" + exit 1 + fi + cd .. +} + +# Function to run first sync +run_first_sync() { + echo -e "\n${BLUE}Running first sync...${NC}" + cd go + ./mail2couch -config ../test/config-test.json -max-messages 5 + cd .. +} + +# Function to add new messages to test incremental sync +add_new_messages() { + echo -e "\n${YELLOW}Adding new messages for incremental sync test...${NC}" + + # Create a simple Python script to add messages directly to GreenMail + cat > test/add_incremental_messages.py << 'EOF' +#!/usr/bin/env python3 + +import imaplib +import time +from test.populate_greenmail import create_simple_message + +def add_new_messages(): + """Add new messages to test incremental sync""" + accounts = [ + ("testuser1", "password123"), + ("syncuser", "syncpass"), + ("archiveuser", "archivepass") + ] + + for username, password in accounts: + try: + print(f"Adding new messages to {username}...") + imap = imaplib.IMAP4('localhost', 3143) + imap.login(username, password) + imap.select('INBOX') + + # Add 3 new messages with timestamps after the first sync + for i in range(1, 4): + subject = f"Incremental Sync Test Message {i}" + body = f"This message was added after the first sync for incremental testing. Message {i} for {username}." + + msg = create_simple_message(subject, body, f"incremental-test@example.com", f"{username}@example.com") + imap.append('INBOX', None, None, msg.encode('utf-8')) + print(f" Added: {subject}") + time.sleep(0.1) + + imap.logout() + print(f"✅ Added 3 new messages to {username}") + + except Exception as e: + print(f"❌ Error adding messages to {username}: {e}") + +if __name__ == "__main__": + add_new_messages() +EOF + + # Add the parent directory to Python path and run the script + cd test + PYTHONPATH=.. python3 add_incremental_messages.py + cd .. +} + +# Function to run second sync (incremental) +run_incremental_sync() { + echo -e "\n${BLUE}Running incremental sync...${NC}" + cd go + ./mail2couch -config ../test/config-test.json -max-messages 10 + cd .. +} + +# Function to verify incremental sync results +verify_results() { + echo -e "\n${YELLOW}Verifying incremental sync results...${NC}" + + # Check CouchDB for sync metadata documents + echo "Checking for sync metadata in CouchDB databases..." + + # List of expected databases based on test config (with m2c prefix) + databases=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only") + + for db in "${databases[@]}"; do + echo " Checking database: $db" + + # Check if database exists + if curl -s -f "http://admin:password@localhost:5984/$db" > /dev/null; then + echo " ✅ Database exists" + + # Look for sync metadata documents + metadata_docs=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"" | grep -o '"total_rows":[0-9]*' | cut -d: -f2 || echo "0") + + if [ "$metadata_docs" -gt 0 ]; then + echo " ✅ Found sync metadata documents: $metadata_docs" + + # Get a sample sync metadata document + sample_doc=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"&include_docs=true&limit=1") + echo " Sample sync metadata:" + echo "$sample_doc" | python3 -m json.tool | grep -E "(lastSyncTime|lastMessageUID|messageCount)" | head -3 + else + echo " ⚠️ No sync metadata documents found" + fi + else + echo " ❌ Database does not exist" + fi + done +} + +# Main test execution +main() { + echo "Starting incremental sync tests..." + + # Pre-test setup + check_containers + build_app + + # Clean up any existing data + echo "🧹 Cleaning up existing test data..." + curl -s -X DELETE "http://admin:password@localhost:5984/m2c_wildcard_all_folders_test" > /dev/null || true + curl -s -X DELETE "http://admin:password@localhost:5984/m2c_work_pattern_test" > /dev/null || true + curl -s -X DELETE "http://admin:password@localhost:5984/m2c_specific_folders_only" > /dev/null || true + + # Step 1: Populate initial test data + populate_initial_data + + # Wait for data to settle + echo "⏳ Waiting for initial data to settle..." + sleep 5 + + # Step 2: Run first sync to establish baseline + echo -e "\n${YELLOW}=== STEP 1: First Sync (Baseline) ===${NC}" + run_first_sync + + # Wait between syncs + echo "⏳ Waiting between syncs..." + sleep 3 + + # Step 3: Add new messages for incremental sync test + echo -e "\n${YELLOW}=== STEP 2: Add New Messages ===${NC}" + add_new_messages + + # Wait for new messages to be ready + echo "⏳ Waiting for new messages to be ready..." + sleep 2 + + # Step 4: Run incremental sync + echo -e "\n${YELLOW}=== STEP 3: Incremental Sync ===${NC}" + run_incremental_sync + + # Step 5: Verify results + echo -e "\n${YELLOW}=== STEP 4: Verification ===${NC}" + verify_results + + echo -e "\n${GREEN}🎉 Incremental sync test completed!${NC}" + echo "" + echo "Key features tested:" + echo " ✅ Sync metadata storage and retrieval" + echo " ✅ IMAP SEARCH with SINCE for efficient incremental fetching" + echo " ✅ Last sync timestamp tracking per mailbox" + echo " ✅ Proper handling of first sync vs incremental sync" + echo "" + echo "To verify results manually:" + echo " - Check CouchDB: http://localhost:5984/_utils" + echo " - Look for 'sync_metadata_*' documents in each database" + echo " - Verify incremental messages were added after baseline sync" +} + +# Cleanup function +cleanup() { + echo "🧹 Cleaning up test artifacts..." + rm -f test/add_incremental_messages.py +} + +# Set trap to cleanup on exit +trap cleanup EXIT + +# Run main function if executed directly +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi \ No newline at end of file diff --git a/test/test-wildcard-patterns.sh b/test/test-wildcard-patterns.sh new file mode 100755 index 0000000..e277ea3 --- /dev/null +++ b/test/test-wildcard-patterns.sh @@ -0,0 +1,137 @@ +#!/bin/bash + +# Test script to validate wildcard folder pattern functionality +# This script tests the various wildcard patterns against the test environment + +set -e + +echo "🧪 Testing Wildcard Folder Pattern Functionality" +echo "================================================" + +# Make sure we're in the right directory +cd "$(dirname "$0")/.." + +# Colors for output +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +BLUE='\033[0;34m' +NC='\033[0m' # No Color + +# Function to run mail2couch with a specific config and capture output +run_test() { + local test_name="$1" + local config_file="$2" + local max_messages="$3" + + echo -e "\n${BLUE}Testing: $test_name${NC}" + echo "Config: $config_file" + echo "Max messages: $max_messages" + echo "----------------------------------------" + + # Run mail2couch and capture output + cd go + if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then + echo -e "${GREEN}✅ Test completed successfully${NC}" + else + echo -e "${RED}❌ Test failed${NC}" + return 1 + fi + cd .. +} + +# Function to check if containers are running +check_containers() { + echo "🔍 Checking if test containers are running..." + + if ! podman ps | grep -q "greenmail"; then + echo -e "${RED}❌ GreenMail container not running${NC}" + echo "Please run: cd test && ./start-test-env.sh" + exit 1 + fi + + if ! podman ps | grep -q "couchdb"; then + echo -e "${RED}❌ CouchDB container not running${NC}" + echo "Please run: cd test && ./start-test-env.sh" + exit 1 + fi + + echo -e "${GREEN}✅ Test containers are running${NC}" +} + +# Function to populate test data +populate_test_data() { + echo "📧 Populating test data..." + cd test + if python3 populate-greenmail.py; then + echo -e "${GREEN}✅ Test data populated successfully${NC}" + else + echo -e "${RED}❌ Failed to populate test data${NC}" + exit 1 + fi + cd .. +} + +# Function to build the application +build_app() { + echo "🔨 Building mail2couch..." + cd go + if go build -o mail2couch .; then + echo -e "${GREEN}✅ Build successful${NC}" + else + echo -e "${RED}❌ Build failed${NC}" + exit 1 + fi + cd .. +} + +# Main test execution +main() { + echo "Starting wildcard pattern tests..." + + # Pre-test checks + check_containers + build_app + populate_test_data + + # Wait a moment for test data to be fully ready + echo "⏳ Waiting for test data to settle..." + sleep 3 + + # Test 1: Wildcard all folders (*) + echo -e "\n${YELLOW}Test 1: Wildcard All Folders Pattern (*)${NC}" + echo "Expected: Should process all folders except Drafts and Trash" + run_test "Wildcard All Folders" "config-test.json" 3 + + # Test 2: Work pattern (Work*) + echo -e "\n${YELLOW}Test 2: Work Pattern (Work*)${NC}" + echo "Expected: Should process Work/Projects, Work/Archive but not Work/Temp (excluded by *Temp*)" + run_test "Work Pattern" "config-test.json" 3 + + # Test 3: Specific folders only + echo -e "\n${YELLOW}Test 3: Specific Folders Only${NC}" + echo "Expected: Should only process INBOX, Sent, and Personal folders" + run_test "Specific Folders" "config-test.json" 3 + + # Test 4: Advanced wildcard examples + echo -e "\n${YELLOW}Test 4: Advanced Wildcard Examples${NC}" + echo "Expected: Various complex patterns should work correctly" + run_test "Advanced Patterns" "config-wildcard-examples.json" 2 + + echo -e "\n${GREEN}🎉 All wildcard pattern tests completed!${NC}" + echo "" + echo "To verify results, check the CouchDB databases:" + echo " http://localhost:5984/_utils" + echo "" + echo "Expected databases should be created for each account:" + echo " - wildcard_all_folders_test" + echo " - work_pattern_test" + echo " - specific_folders_only" + echo "" + echo "Each database should contain documents with 'mailbox' field showing origin folder." +} + +# Run main function if executed directly +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi \ No newline at end of file