diff --git a/CLAUDE.md b/CLAUDE.md deleted file mode 100644 index fce9b14..0000000 --- a/CLAUDE.md +++ /dev/null @@ -1,161 +0,0 @@ -# CLAUDE.md - -This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository. - -## Project Overview - -mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations: -- **Go implementation**: Located in `/go/` directory (currently the active implementation) -- **Rust implementation**: Planned but not yet implemented - -## Development Commands - -### Go Implementation (Primary) - -```bash -# Build the application -cd go && go build -o mail2couch . - -# Run the application with automatic config discovery -cd go && ./mail2couch - -# Run with specific config file -cd go && ./mail2couch -config /path/to/config.json - -# Run with message limit (useful for large mailboxes) -cd go && ./mail2couch -max-messages 100 - -# Run with both config and message limit -cd go && ./mail2couch -config /path/to/config.json -max-messages 50 - -# Run linting/static analysis -cd go && go vet ./... - -# Run integration tests with Podman containers -cd test && ./run-tests.sh - -# Run specialized tests -cd test && ./test-wildcard-patterns.sh -cd test && ./test-incremental-sync.sh - -# Run unit tests (none currently implemented) -cd go && go test ./... - -# Check dependencies -cd go && go mod tidy -``` - -## Architecture - -### Core Components - -1. **Configuration (`config/`)**: JSON-based configuration system - - Supports multiple mail sources with filtering options - - CouchDB connection settings - - Each source can have folder and message filters - -2. **Mail Handling (`mail/`)**: IMAP client implementation - - Uses `github.com/emersion/go-imap/v2` for IMAP operations - - Supports TLS connections - - Currently only lists mailboxes (backup functionality not yet implemented) - -3. **CouchDB Integration (`couch/`)**: Database operations - - Uses `github.com/go-kivik/kivik/v4` as CouchDB driver - - Handles database creation and document management - - Defines `MailDocument` structure for email storage - -### Configuration Structure - -The application uses `config.json` for configuration with the following structure: -- `couchDb`: Database connection settings (URL, credentials) -- `mailSources`: Array of mail sources with individual settings: - - Protocol support (currently only IMAP) - - Connection details (host, port, credentials) - - `mode`: Either "sync" or "archive" (defaults to "archive" if not specified) - - **sync**: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB) - - **archive**: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents) - - Filtering options for folders and messages with wildcard support - - Enable/disable per source - -### Configuration File Discovery - -The application automatically searches for configuration files in the following order: -1. Path specified by `-config` command line flag -2. `./config.json` (current working directory) -3. `./config/config.json` (config subdirectory) -4. `~/.config/mail2couch/config.json` (user XDG config directory) -5. `~/.mail2couch.json` (user home directory) - -This design ensures the same `config.json` format will work for both Go and Rust implementations. - -### Current Implementation Status - -- ✅ Configuration loading with automatic file discovery -- ✅ Command line flag support for config file path -- ✅ Per-account CouchDB database creation and management -- ✅ IMAP connection and mailbox listing -- ✅ Build error fixes -- ✅ Email message retrieval framework (with placeholder data) -- ✅ Email storage to CouchDB framework with native attachments -- ✅ Folder filtering logic with wildcard support (`*`, `?`, `[abc]` patterns) -- ✅ Date filtering support -- ✅ Keyword filtering support (subject, sender, recipient keywords) -- ✅ Duplicate detection and prevention -- ✅ Sync vs Archive mode implementation -- ✅ CouchDB attachment storage for email attachments -- ✅ Real IMAP message parsing with go-message library -- ✅ Full message body and attachment handling with MIME multipart support -- ✅ Command line argument support (--max-messages flag) -- ✅ Per-account CouchDB databases for better organization -- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking -- ❌ Rust implementation - -### Key Dependencies - -- `github.com/emersion/go-imap/v2`: IMAP client library -- `github.com/go-kivik/kivik/v4`: CouchDB client library - -### Incremental Sync Implementation - -The application implements intelligent incremental synchronization to avoid re-processing messages: - -- **Sync Metadata Storage**: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed -- **IMAP SEARCH Integration**: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages -- **Per-Mailbox Tracking**: Sync state is tracked independently for each mailbox in each account -- **Fallback Behavior**: Gracefully falls back to fetching recent messages if IMAP SEARCH fails -- **First Sync Handling**: Initial sync can use config `since` date or perform full sync - -Sync metadata documents are stored in CouchDB with ID format: `sync_metadata_{mailbox}` and include: -- `lastSyncTime`: When this mailbox was last successfully synced -- `lastMessageUID`: Highest UID processed in the last sync -- `messageCount`: Number of messages processed in the last sync - -### Development Notes - -- The main entry point is `main.go` which orchestrates the configuration loading, CouchDB setup, and mail source processing -- Each mail source gets its own CouchDB database named using `GenerateAccountDBName()` function with `m2c_` prefix -- Each mail source is processed sequentially with proper error handling -- The application uses real IMAP message parsing with go-message library for full email processing -- Message filtering by folder (wildcard patterns), date (since), and keywords is implemented -- Duplicate detection prevents re-storing existing messages -- Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account -- Email attachments are stored as native CouchDB attachments linked to the email document -- Comprehensive test environment with Podman containers and automated test scripts -- The application uses automatic config file discovery as documented above - -### Next Steps - -The following enhancements could further improve the implementation: - -1. **Error Recovery**: Add retry logic for network failures and partial sync recovery -2. **Performance Optimization**: Add batch operations for better CouchDB insertion performance -3. **Unit Testing**: Add comprehensive unit tests for all major components -4. **Advanced Filtering**: Add support for more complex filter expressions and regex patterns -5. **Monitoring**: Add metrics and logging for production deployment -6. **Configuration Validation**: Enhanced validation for configuration files -7. **Multi-threading**: Parallel processing of multiple mailboxes or accounts - -## Development Guidelines - -### Code Quality and Standards -- All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly \ No newline at end of file diff --git a/FOLDER_PATTERNS.md b/FOLDER_PATTERNS.md deleted file mode 100644 index dc96b8c..0000000 --- a/FOLDER_PATTERNS.md +++ /dev/null @@ -1,102 +0,0 @@ -# Folder Pattern Matching in mail2couch - -mail2couch supports powerful wildcard patterns for selecting which folders to process. This allows flexible configuration for different mail backup scenarios. - -## Pattern Syntax - -The folder filtering uses Go's `filepath.Match` syntax, which supports: - -- `*` matches any sequence of characters (including none) -- `?` matches any single character -- `[abc]` matches any character within the brackets -- `[a-z]` matches any character in the range -- `\` escapes special characters - -## Special Cases - -- `"*"` in the include list means **ALL available folders** will be processed -- Empty include list with exclude patterns will process all folders except excluded ones -- Exact string matching is supported for backwards compatibility - -## Examples - -### Include All Folders -```json -{ - "folderFilter": { - "include": ["*"], - "exclude": ["Drafts", "Trash", "Spam"] - } -} -``` -This processes all folders except Drafts, Trash, and Spam. - -### Work-Related Folders Only -```json -{ - "folderFilter": { - "include": ["Work*", "Projects*", "INBOX"], - "exclude": ["*Temp*", "*Draft*"] - } -} -``` -This includes folders starting with "Work" or "Projects", plus INBOX, but excludes any folder containing "Temp" or "Draft". - -### Archive Patterns -```json -{ - "folderFilter": { - "include": ["Archive*", "*Important*", "INBOX"], - "exclude": ["*Temp"] - } -} -``` -This includes folders starting with "Archive", any folder containing "Important", and INBOX, excluding temporary folders. - -### Specific Folders Only -```json -{ - "folderFilter": { - "include": ["INBOX", "Sent", "Important"], - "exclude": [] - } -} -``` -This processes only the exact folders: INBOX, Sent, and Important. - -### Subfolder Patterns -```json -{ - "folderFilter": { - "include": ["Work/*", "Personal/*"], - "exclude": ["*/Drafts"] - } -} -``` -This includes all subfolders under Work and Personal, but excludes any Drafts subfolder. - -## Folder Hierarchy - -Different IMAP servers use different separators for folder hierarchies: -- Most servers use `/` (e.g., `Work/Projects`, `Archive/2024`) -- Some use `.` (e.g., `Work.Projects`, `Archive.2024`) - -The patterns work with whatever separator your IMAP server uses. - -## Common Use Cases - -1. **Corporate Email**: `["*"]` with exclude `["Drafts", "Trash", "Spam"]` for complete backup -2. **Selective Backup**: `["INBOX", "Sent", "Important"]` for essential folders only -3. **Project-based**: `["Project*", "Client*"]` to backup work-related folders -4. **Archive Mode**: `["Archive*", "*Important*"]` for long-term storage -5. **Sync Mode**: `["INBOX"]` for real-time synchronization - -## Message Origin Tracking - -All messages stored in CouchDB include a `mailbox` field that records the original folder name. This ensures you can always identify which folder a message came from, regardless of how it was selected by the folder filter. - -## Performance Considerations - -- Using `"*"` processes all folders, which may be slow for accounts with many folders -- Specific folder names are faster than wildcard patterns -- Consider using exclude patterns to filter out large, unimportant folders like Trash or Spam \ No newline at end of file diff --git a/README.md b/README.md index f81e050..631d01c 100644 --- a/README.md +++ b/README.md @@ -1,389 +1,5 @@ # mail2couch -A powerful email backup utility that synchronizes mail from IMAP accounts to CouchDB databases with intelligent incremental sync, comprehensive filtering, and native attachment support. +A utility to back up mail from various sources to couchdb -## Features - -### Core Functionality -- **IMAP Email Backup**: Connect to any IMAP server (Gmail, Outlook, self-hosted) -- **CouchDB Storage**: Store emails as JSON documents with native CouchDB attachments -- **Incremental Sync**: Efficiently sync only new messages using IMAP SEARCH with timestamp tracking -- **Per-Account Databases**: Each mail source gets its own CouchDB database for better organization -- **Duplicate Prevention**: Automatic detection and prevention of duplicate message storage - -### Sync Modes -- **Archive Mode**: Preserve all messages ever seen, even if deleted from mail server (default) -- **Sync Mode**: Maintain 1-to-1 relationship with mail server (removes deleted messages from CouchDB) - -### Advanced Filtering -- **Wildcard Folder Patterns**: Use `*`, `?`, `[abc]` patterns for flexible folder selection -- **Keyword Filtering**: Filter messages by keywords in subjects, senders, or recipients -- **Date Filtering**: Process only messages since a specific date -- **Include/Exclude Logic**: Combine multiple filter types for precise control - -### Message Processing -- **Full MIME Support**: Parse multipart messages, HTML/plain text, and embedded content -- **Native Attachments**: Store email attachments as CouchDB native attachments with compression -- **Complete Headers**: Preserve all email headers and metadata -- **UTF-8 Support**: Handle international characters and special content - -### Operational Features -- **Automatic Config Discovery**: Finds configuration files in standard locations -- **Command Line Control**: Override settings with `--max-messages` and `--config` flags -- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting -- **Error Resilience**: Graceful handling of network issues and server problems - -## Quick Start - -### Installation - -1. **Install dependencies**: - ```bash - # Go 1.21+ required - go version - ``` - -2. **Clone and build**: - ```bash - git clone - cd mail2couch/go - go build -o mail2couch . - ``` - -### Basic Usage - -1. **Create configuration file** (`config.json`): - ```json - { - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "your-email@gmail.com", - "password": "your-app-password", - "mode": "archive", - "folderFilter": { - "include": ["*"], - "exclude": ["[Gmail]/Trash", "[Gmail]/Spam"] - } - } - ] - } - ``` - -2. **Run mail2couch**: - ```bash - ./mail2couch - ``` - -The application will: -- Create a CouchDB database named `m2c_personal_gmail` -- Sync all folders except Trash and Spam -- Store messages with native attachments -- Track sync state for efficient incremental updates - -## Configuration - -### Configuration File Discovery - -mail2couch automatically searches for configuration files in this order: -1. Path specified by `--config` flag -2. `./config.json` (current directory) -3. `./config/config.json` (config subdirectory) -4. `~/.config/mail2couch/config.json` (user config directory) -5. `~/.mail2couch.json` (user home directory) - -### Command Line Options - -```bash -./mail2couch [options] - -Options: - --config PATH Specify configuration file path - --max-messages N Limit messages processed per mailbox per run (0 = unlimited) -``` - -### Folder Pattern Examples - -| Pattern | Description | Matches | -|---------|-------------|---------| -| `"*"` | All folders | `INBOX`, `Sent`, `Work/Projects`, etc. | -| `"INBOX"` | Exact match | `INBOX` only | -| `"Work*"` | Prefix match | `Work`, `Work/Projects`, `WorkStuff` | -| `"*/Archive"` | Suffix match | `Personal/Archive`, `Work/Archive` | -| `"Work/*"` | Subfolder match | `Work/Projects`, `Work/Clients` | - -### Keyword Filtering Examples - -```json -{ - "messageFilter": { - "subjectKeywords": ["urgent", "meeting", "invoice"], - "senderKeywords": ["@company.com", "noreply@"], - "recipientKeywords": ["team@", "support@"] - } -} -``` - -## Advanced Configuration Examples - -See the [example configurations](#example-configurations) section below for detailed configuration scenarios. - -## Testing - -A comprehensive test environment is included with Podman containers: - -```bash -cd test - -# Quick automated testing (recommended) -./run-tests.sh # Complete integration test with automatic cleanup - -# Specialized feature testing -./test-wildcard-patterns.sh # Test folder pattern matching -./test-incremental-sync.sh # Test incremental synchronization - -# Manual testing environment -./start-test-env.sh # Start persistent test environment -# ... manual testing with various configurations ... -./stop-test-env.sh # Clean up when done -``` - -## Architecture - -### Database Structure -- **Per-Account Databases**: Each mail source creates its own CouchDB database with `m2c_` prefix -- **Message Documents**: Each email becomes a CouchDB document with metadata -- **Native Attachments**: Email attachments stored as CouchDB attachments (compressed) -- **Sync Metadata**: Tracks incremental sync state per mailbox - -### Document Structure -```json -{ - "_id": "INBOX_12345", - "sourceUid": "12345", - "mailbox": "INBOX", - "from": ["sender@example.com"], - "to": ["recipient@example.com"], - "subject": "Sample Email", - "date": "2024-01-15T10:30:00Z", - "body": "Email content...", - "headers": {"Content-Type": ["text/plain"]}, - "storedAt": "2024-01-15T10:35:00Z", - "docType": "mail", - "hasAttachments": true, - "_attachments": { - "document.pdf": { - "content_type": "application/pdf", - "length": 54321 - } - } -} -``` - -## Example Configurations - -### Simple Configuration -Basic setup for a single Gmail account: - -```json -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "your-email@gmail.com", - "password": "your-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": [] - }, - "messageFilter": { - "since": "2024-01-01" - } - } - ] -} -``` - -### Advanced Multi-Account Configuration -Complex setup with multiple accounts, filtering, and different sync modes: - -```json -{ - "couchDb": { - "url": "https://your-couchdb.example.com:5984", - "user": "backup_user", - "password": "secure_password" - }, - "mailSources": [ - { - "name": "Work Email", - "enabled": true, - "protocol": "imap", - "host": "outlook.office365.com", - "port": 993, - "user": "you@company.com", - "password": "app-password", - "mode": "sync", - "folderFilter": { - "include": ["*"], - "exclude": ["Deleted Items", "Junk Email", "Drafts"] - }, - "messageFilter": { - "since": "2023-01-01", - "subjectKeywords": ["project", "meeting", "urgent"], - "senderKeywords": ["@company.com", "@client.com"] - } - }, - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "personal@gmail.com", - "password": "gmail-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Important", "Work/*", "Personal/*"], - "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"] - }, - "messageFilter": { - "recipientKeywords": ["family@", "personal@"] - } - }, - { - "name": "Self-Hosted Mail", - "enabled": true, - "protocol": "imap", - "host": "mail.yourdomain.com", - "port": 143, - "user": "admin@yourdomain.com", - "password": "mail-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Archive/*", "Projects/*"], - "exclude": ["*/Drafts", "Trash"] - }, - "messageFilter": { - "since": "2023-06-01", - "subjectKeywords": ["invoice", "receipt", "statement"] - } - }, - { - "name": "Legacy Account", - "enabled": false, - "protocol": "imap", - "host": "legacy.mailserver.com", - "port": 993, - "user": "old@account.com", - "password": "legacy-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX"], - "exclude": [] - }, - "messageFilter": {} - } - ] -} -``` - -### Configuration Options Reference - -#### CouchDB Configuration -- `url`: CouchDB server URL with protocol and port -- `user`: CouchDB username with database access -- `password`: CouchDB password - -#### Mail Source Configuration -- `name`: Descriptive name (used for database naming) -- `enabled`: Boolean to enable/disable this source -- `protocol`: Only `"imap"` currently supported -- `host`: IMAP server hostname -- `port`: IMAP port (993 for TLS, 143 for plain, 3143 for testing) -- `user`: Email account username -- `password`: Email account password (use app passwords for Gmail/Outlook) -- `mode`: `"sync"` (mirror server) or `"archive"` (preserve all messages) - -#### Folder Filter Configuration -- `include`: Array of folder patterns to process (empty = all folders) -- `exclude`: Array of folder patterns to skip - -#### Message Filter Configuration -- `since`: Date string (YYYY-MM-DD) to process messages from -- `subjectKeywords`: Array of keywords that must appear in subject line -- `senderKeywords`: Array of keywords that must appear in sender addresses -- `recipientKeywords`: Array of keywords that must appear in recipient addresses - -## Production Deployment - -### Security Considerations -- Use app passwords instead of account passwords -- Store configuration files with restricted permissions (600) -- Use HTTPS for CouchDB connections in production -- Consider encrypting sensitive configuration data - -### Monitoring and Maintenance -- Review sync metadata documents for sync health -- Monitor CouchDB database sizes and compaction -- Set up log rotation for application output -- Schedule regular backups of CouchDB databases - -### Performance Tuning -- Use `--max-messages` to limit processing load -- Run during off-peak hours for large initial syncs -- Monitor IMAP server rate limits and connection limits -- Consider running multiple instances for different accounts - -## Troubleshooting - -### Common Issues - -**Connection Errors**: -- Verify IMAP server settings and credentials -- Check firewall and network connectivity -- Ensure correct ports (993 for TLS, 143 for plain) - -**Authentication Failures**: -- Use app passwords for Gmail, Outlook, and other providers -- Enable "Less Secure Apps" if required by provider -- Verify account permissions and 2FA settings - -**Sync Issues**: -- Check CouchDB connectivity and permissions -- Review sync metadata documents for error states -- Verify folder names and patterns match server structure - -**Performance Problems**: -- Use date filtering (`since`) for large mailboxes -- Implement `--max-messages` limits for initial syncs -- Monitor server-side rate limiting - -For detailed troubleshooting, see the [test environment documentation](test/README.md). - -## Contributing - -This project welcomes contributions! Please see [CLAUDE.md](CLAUDE.md) for development setup and architecture details. - -## License - -[License information to be added] +At least two implementations will be available, on in Rust and one in Go. diff --git a/TODO.md b/TODO.md deleted file mode 100644 index e004c00..0000000 --- a/TODO.md +++ /dev/null @@ -1,47 +0,0 @@ -# mail2couch TODO and Feature Requests - -## Planned Features - -### Keyword Filtering for Messages - -Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration. - -**Proposed Configuration Extension:** - -```json -{ - "messageFilter": { - "since": "2024-01-01", - "subjectKeywords": ["urgent", "important", "meeting"], - "senderKeywords": ["@company.com", "notifications"], - "recipientKeywords": ["team@company.com", "all@"] - } -} -``` - -**Implementation Details:** - -- `subjectKeywords`: Array of keywords to match in email subject lines -- `senderKeywords`: Array of keywords to match in sender email addresses or names -- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names -- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns -- Case-insensitive matching by default -- Support for simple wildcards or regex patterns - -**Use Cases:** - -1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords -2. **Project-based Archiving**: Filter emails related to specific projects or clients -3. **Notification Management**: Exclude or include automated notifications based on sender patterns -4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders - -**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails. - -## Other Planned Improvements - -1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content -2. **Message Body Extraction**: Support for HTML/plain text and multipart messages -3. **Attachment Handling**: Optional support for email attachments -4. **Batch Operations**: Improve CouchDB insertion performance -5. **Error Recovery**: Retry logic and partial sync recovery -6. **Testing**: Comprehensive unit test coverage \ No newline at end of file diff --git a/config-advanced.json b/config-advanced.json deleted file mode 100644 index a54f350..0000000 --- a/config-advanced.json +++ /dev/null @@ -1,78 +0,0 @@ -{ - "couchDb": { - "url": "https://your-couchdb.example.com:5984", - "user": "backup_user", - "password": "secure_password" - }, - "mailSources": [ - { - "name": "Work Email", - "enabled": true, - "protocol": "imap", - "host": "outlook.office365.com", - "port": 993, - "user": "you@company.com", - "password": "app-password", - "mode": "sync", - "folderFilter": { - "include": ["*"], - "exclude": ["Deleted Items", "Junk Email", "Drafts"] - }, - "messageFilter": { - "since": "2023-01-01", - "subjectKeywords": ["project", "meeting", "urgent"], - "senderKeywords": ["@company.com", "@client.com"] - } - }, - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "personal@gmail.com", - "password": "gmail-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Important", "Work/*", "Personal/*"], - "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"] - }, - "messageFilter": { - "recipientKeywords": ["family@", "personal@"] - } - }, - { - "name": "Self-Hosted Mail", - "enabled": true, - "protocol": "imap", - "host": "mail.yourdomain.com", - "port": 143, - "user": "admin@yourdomain.com", - "password": "mail-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Archive/*", "Projects/*"], - "exclude": ["*/Drafts", "Trash"] - }, - "messageFilter": { - "since": "2023-06-01", - "subjectKeywords": ["invoice", "receipt", "statement"] - } - }, - { - "name": "Legacy Account", - "enabled": false, - "protocol": "imap", - "host": "legacy.mailserver.com", - "port": 993, - "user": "old@account.com", - "password": "legacy-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX"], - "exclude": [] - }, - "messageFilter": {} - } - ] -} \ No newline at end of file diff --git a/config-providers.json b/config-providers.json deleted file mode 100644 index 577294c..0000000 --- a/config-providers.json +++ /dev/null @@ -1,90 +0,0 @@ -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Gmail Account", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "your-email@gmail.com", - "password": "your-16-character-app-password", - "mode": "archive", - "folderFilter": { - "include": ["*"], - "exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "[Gmail]/Drafts"] - }, - "messageFilter": { - "since": "2024-01-01" - } - }, - { - "name": "Outlook 365", - "enabled": true, - "protocol": "imap", - "host": "outlook.office365.com", - "port": 993, - "user": "you@outlook.com", - "password": "your-app-password", - "mode": "sync", - "folderFilter": { - "include": ["INBOX", "Sent Items", "Archive"], - "exclude": ["Deleted Items", "Junk Email"] - }, - "messageFilter": { - "since": "2023-06-01" - } - }, - { - "name": "Yahoo Mail", - "enabled": false, - "protocol": "imap", - "host": "imap.mail.yahoo.com", - "port": 993, - "user": "your-email@yahoo.com", - "password": "your-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": ["Trash", "Spam"] - }, - "messageFilter": {} - }, - { - "name": "iCloud Mail", - "enabled": false, - "protocol": "imap", - "host": "imap.mail.me.com", - "port": 993, - "user": "your-email@icloud.com", - "password": "your-app-specific-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent Messages"], - "exclude": ["Deleted Messages", "Junk"] - }, - "messageFilter": {} - }, - { - "name": "Custom IMAP Server", - "enabled": false, - "protocol": "imap", - "host": "mail.example.com", - "port": 993, - "user": "username@example.com", - "password": "password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": ["Trash"] - }, - "messageFilter": { - "since": "2024-01-01" - } - } - ] -} \ No newline at end of file diff --git a/config-simple.json b/config-simple.json deleted file mode 100644 index 4c9cb50..0000000 --- a/config-simple.json +++ /dev/null @@ -1,26 +0,0 @@ -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "your-email@gmail.com", - "password": "your-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": [] - }, - "messageFilter": { - "since": "2024-01-01" - } - } - ] -} \ No newline at end of file diff --git a/config.json b/config.json deleted file mode 100644 index 39f4af0..0000000 --- a/config.json +++ /dev/null @@ -1,41 +0,0 @@ -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Personal Gmail", - "enabled": true, - "protocol": "imap", - "host": "imap.gmail.com", - "port": 993, - "user": "your-email@gmail.com", - "password": "your-app-password", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": ["Spam", "Trash"] - }, - "messageFilter": { - "since": "2024-01-01" - } - }, - { - "name": "Work Account", - "enabled": true, - "protocol": "imap", - "host": "imap.work.com", - "port": 993, - "user": "user@work.com", - "password": "password", - "mode": "sync", - "folderFilter": { - "include": [], - "exclude": [] - }, - "messageFilter": {} - } - ] -} diff --git a/go/config/config.go b/go/config/config.go deleted file mode 100644 index 5581094..0000000 --- a/go/config/config.go +++ /dev/null @@ -1,153 +0,0 @@ -package config - -import ( - "encoding/json" - "flag" - "fmt" - "os" - "path/filepath" -) - -type Config struct { - CouchDb CouchDbConfig `json:"couchDb"` - MailSources []MailSource `json:"mailSources"` -} - -type CouchDbConfig struct { - URL string `json:"url"` - User string `json:"user"` - Password string `json:"password"` -} - -type MailSource struct { - Name string `json:"name"` - Enabled bool `json:"enabled"` - Protocol string `json:"protocol"` - Host string `json:"host"` - Port int `json:"port"` - User string `json:"user"` - Password string `json:"password"` - Mode string `json:"mode"` // "sync" or "archive" - FolderFilter FolderFilter `json:"folderFilter"` - MessageFilter MessageFilter `json:"messageFilter"` -} - -type FolderFilter struct { - Include []string `json:"include"` - Exclude []string `json:"exclude"` -} - -type MessageFilter struct { - Since string `json:"since,omitempty"` - SubjectKeywords []string `json:"subjectKeywords,omitempty"` // Filter by keywords in subject - SenderKeywords []string `json:"senderKeywords,omitempty"` // Filter by keywords in sender addresses - RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses -} - -func LoadConfig(path string) (*Config, error) { - configFile, err := os.Open(path) - if err != nil { - return nil, err - } - defer configFile.Close() - - var config Config - jsonParser := json.NewDecoder(configFile) - if err = jsonParser.Decode(&config); err != nil { - return nil, err - } - - // Validate and set defaults for mail sources - for i := range config.MailSources { - source := &config.MailSources[i] - if source.Mode == "" { - source.Mode = "archive" // Default to archive mode - } - if source.Mode != "sync" && source.Mode != "archive" { - return nil, fmt.Errorf("invalid mode '%s' for mail source '%s': must be 'sync' or 'archive'", source.Mode, source.Name) - } - } - - return &config, nil -} - -// IsSyncMode returns true if the mail source is in sync mode -func (ms *MailSource) IsSyncMode() bool { - return ms.Mode == "sync" -} - -// IsArchiveMode returns true if the mail source is in archive mode -func (ms *MailSource) IsArchiveMode() bool { - return ms.Mode == "archive" || ms.Mode == "" // Default to archive -} - -// CommandLineArgs holds parsed command line arguments -type CommandLineArgs struct { - ConfigPath string - MaxMessages int -} - -// ParseCommandLine parses command line arguments -func ParseCommandLine() *CommandLineArgs { - configFlag := flag.String("config", "", "Path to configuration file") - maxMessagesFlag := flag.Int("max-messages", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)") - flag.Parse() - - return &CommandLineArgs{ - ConfigPath: *configFlag, - MaxMessages: *maxMessagesFlag, - } -} - -// FindConfigFile searches for config.json in the following order: -// 1. Path specified by -config flag -// 2. ./config.json (current directory) -// 3. ~/.config/mail2couch/config.json (user config directory) -// 4. ~/.mail2couch.json (user home directory) -func FindConfigFile(args *CommandLineArgs) (string, error) { - if args.ConfigPath != "" { - if _, err := os.Stat(args.ConfigPath); err == nil { - return args.ConfigPath, nil - } - return "", fmt.Errorf("specified config file not found: %s", args.ConfigPath) - } - - // List of possible config file locations in order of preference - candidates := []string{ - "config.json", // Current directory - "config/config.json", // Config subdirectory - } - - // Add user directory paths - if homeDir, err := os.UserHomeDir(); err == nil { - candidates = append(candidates, - filepath.Join(homeDir, ".config", "mail2couch", "config.json"), - filepath.Join(homeDir, ".mail2couch.json"), - ) - } - - // Try each candidate location - for _, candidate := range candidates { - if _, err := os.Stat(candidate); err == nil { - return candidate, nil - } - } - - return "", fmt.Errorf("no configuration file found. Searched locations: %v", candidates) -} - -// LoadConfigWithDiscovery loads configuration using automatic file discovery -func LoadConfigWithDiscovery(args *CommandLineArgs) (*Config, error) { - configPath, err := FindConfigFile(args) - if err != nil { - return nil, err - } - - fmt.Printf("Using configuration file: %s\n", configPath) - if args.MaxMessages > 0 { - fmt.Printf("Maximum messages per mailbox: %d\n", args.MaxMessages) - } else { - fmt.Printf("Maximum messages per mailbox: unlimited\n") - } - return LoadConfig(configPath) -} diff --git a/go/couch/couch.go b/go/couch/couch.go deleted file mode 100644 index 7a3c5ab..0000000 --- a/go/couch/couch.go +++ /dev/null @@ -1,374 +0,0 @@ -package couch - -import ( - "context" - "fmt" - "io" - "net/url" - "regexp" - "strings" - "time" - - "github.com/go-kivik/kivik/v4" - _ "github.com/go-kivik/kivik/v4/couchdb" // The CouchDB driver - "mail2couch/config" - "mail2couch/mail" -) - -// Client wraps the Kivik client -type Client struct { - *kivik.Client -} - -// MailDocument represents an email message stored in CouchDB -type MailDocument struct { - ID string `json:"_id,omitempty"` - Rev string `json:"_rev,omitempty"` - Attachments map[string]AttachmentStub `json:"_attachments,omitempty"` // CouchDB attachments - SourceUID string `json:"sourceUid"` // Unique ID from the mail source (e.g., IMAP UID) - Mailbox string `json:"mailbox"` // Source mailbox name - From []string `json:"from"` - To []string `json:"to"` - Subject string `json:"subject"` - Date time.Time `json:"date"` - Body string `json:"body"` - Headers map[string][]string `json:"headers"` - StoredAt time.Time `json:"storedAt"` // When the document was stored - DocType string `json:"docType"` // Always "mail" - HasAttachments bool `json:"hasAttachments"` // Indicates if message has attachments -} - -// AttachmentStub represents metadata for a CouchDB attachment -type AttachmentStub struct { - ContentType string `json:"content_type"` - Length int64 `json:"length,omitempty"` - Stub bool `json:"stub,omitempty"` -} - -// SyncMetadata represents sync state information stored in CouchDB -type SyncMetadata struct { - ID string `json:"_id,omitempty"` - Rev string `json:"_rev,omitempty"` - DocType string `json:"docType"` // Always "sync_metadata" - Mailbox string `json:"mailbox"` // Mailbox name - LastSyncTime time.Time `json:"lastSyncTime"` // When this mailbox was last synced - LastMessageUID uint32 `json:"lastMessageUID"` // Highest UID processed in last sync - MessageCount int `json:"messageCount"` // Number of messages processed in last sync - UpdatedAt time.Time `json:"updatedAt"` // When this metadata was last updated -} - -// NewClient creates a new CouchDB client from the configuration -func NewClient(cfg *config.CouchDbConfig) (*Client, error) { - parsedURL, err := url.Parse(cfg.URL) - if err != nil { - return nil, fmt.Errorf("invalid couchdb url: %w", err) - } - - parsedURL.User = url.UserPassword(cfg.User, cfg.Password) - dsn := parsedURL.String() - - client, err := kivik.New("couch", dsn) - if err != nil { - return nil, err - } - - return &Client{client}, nil -} - -// EnsureDB ensures that the configured database exists. -func (c *Client) EnsureDB(ctx context.Context, dbName string) error { - exists, err := c.DBExists(ctx, dbName) - if err != nil { - return err - } - if !exists { - return c.CreateDB(ctx, dbName) - } - return nil -} - -// GenerateAccountDBName creates a CouchDB-compatible database name from account info -func GenerateAccountDBName(accountName, userEmail string) string { - // Use account name if available, otherwise fall back to email - name := accountName - if name == "" { - name = userEmail - } - - // Convert to lowercase and replace invalid characters with underscores - name = strings.ToLower(name) - // CouchDB database names must match: ^[a-z][a-z0-9_$()+/-]*$ - validName := regexp.MustCompile(`[^a-z0-9_$()+/-]`).ReplaceAllString(name, "_") - - // Ensure it starts with a letter and add m2c prefix - if len(validName) > 0 && (validName[0] < 'a' || validName[0] > 'z') { - validName = "m2c_mail_" + validName - } else { - validName = "m2c_" + validName - } - - return validName -} - -// ConvertMessage converts an IMAP message to a MailDocument -func ConvertMessage(msg *mail.Message, mailbox string) *MailDocument { - docID := fmt.Sprintf("%s_%d", mailbox, msg.UID) - - doc := &MailDocument{ - ID: docID, - SourceUID: fmt.Sprintf("%d", msg.UID), - Mailbox: mailbox, - From: msg.From, - To: msg.To, - Subject: msg.Subject, - Date: msg.Date, - Body: msg.Body, - Headers: msg.Headers, - StoredAt: time.Now(), - DocType: "mail", - HasAttachments: len(msg.Attachments) > 0, - } - - // Don't add attachment metadata here - CouchDB will handle this when we store attachments - // We'll add the attachment metadata after successful document creation - - return doc -} - -// StoreMessage stores a mail message in CouchDB with attachments -func (c *Client) StoreMessage(ctx context.Context, dbName string, doc *MailDocument, msg *mail.Message) error { - db := c.DB(dbName) - if db.Err() != nil { - return db.Err() - } - - // Check if document already exists - exists, err := c.DocumentExists(ctx, dbName, doc.ID) - if err != nil { - return fmt.Errorf("failed to check if document exists: %w", err) - } - - if exists { - return nil // Document already exists, skip - } - - // Store the document first (without attachments) - rev, err := db.Put(ctx, doc.ID, doc) - if err != nil { - return fmt.Errorf("failed to store document: %w", err) - } - - // If there are attachments, store them as CouchDB attachments - if msg != nil && len(msg.Attachments) > 0 { - currentRev := rev - for _, att := range msg.Attachments { - newRev, err := c.StoreAttachment(ctx, dbName, doc.ID, currentRev, att.Filename, att.ContentType, att.Content) - if err != nil { - return fmt.Errorf("failed to store attachment %s: %w", att.Filename, err) - } - currentRev = newRev // Update revision for next attachment - } - } - - return nil -} - -// StoreAttachment stores an attachment to an existing CouchDB document -func (c *Client) StoreAttachment(ctx context.Context, dbName, docID, rev, filename, contentType string, content []byte) (string, error) { - db := c.DB(dbName) - if db.Err() != nil { - return "", db.Err() - } - - att := &kivik.Attachment{ - Filename: filename, - ContentType: contentType, - Content: io.NopCloser(strings.NewReader(string(content))), - } - - newRev, err := db.PutAttachment(ctx, docID, att, kivik.Rev(rev)) - if err != nil { - return "", fmt.Errorf("failed to store attachment: %w", err) - } - - return newRev, nil -} - -// StoreMessages stores multiple mail messages in CouchDB with their corresponding attachments -func (c *Client) StoreMessages(ctx context.Context, dbName string, docs []*MailDocument, messages []*mail.Message) error { - for i, doc := range docs { - var msg *mail.Message - if i < len(messages) { - msg = messages[i] - } - if err := c.StoreMessage(ctx, dbName, doc, msg); err != nil { - return err - } - } - return nil -} - -// DocumentExists checks if a document with the given ID already exists. -func (c *Client) DocumentExists(ctx context.Context, dbName, docID string) (bool, error) { - db := c.DB(dbName) - if db.Err() != nil { - return false, db.Err() - } - - row := db.Get(ctx, docID) - return row.Err() == nil, nil -} - -// GetAllMailDocumentIDs returns all mail document IDs from a database for a specific mailbox -func (c *Client) GetAllMailDocumentIDs(ctx context.Context, dbName, mailbox string) (map[string]bool, error) { - db := c.DB(dbName) - if db.Err() != nil { - return nil, db.Err() - } - - // Create a view query to get all document IDs for the specified mailbox - rows := db.AllDocs(ctx) - - docIDs := make(map[string]bool) - for rows.Next() { - docID, err := rows.ID() - if err != nil { - continue - } - // Filter by mailbox prefix (documents are named like "INBOX_123") - if mailbox == "" || strings.HasPrefix(docID, mailbox+"_") { - docIDs[docID] = true - } - } - - if rows.Err() != nil { - return nil, rows.Err() - } - - return docIDs, nil -} - -// DeleteDocument removes a document from CouchDB -func (c *Client) DeleteDocument(ctx context.Context, dbName, docID string) error { - db := c.DB(dbName) - if db.Err() != nil { - return db.Err() - } - - // Get the current revision - row := db.Get(ctx, docID) - if row.Err() != nil { - return row.Err() // Document doesn't exist or other error - } - - var doc struct { - Rev string `json:"_rev"` - } - if err := row.ScanDoc(&doc); err != nil { - return err - } - - // Delete the document - _, err := db.Delete(ctx, docID, doc.Rev) - return err -} - -// SyncMailbox synchronizes a mailbox between mail server and CouchDB -// In sync mode: removes documents from CouchDB that are no longer in the mail account -// In archive mode: keeps all documents (no removal) -func (c *Client) SyncMailbox(ctx context.Context, dbName, mailbox string, currentMessageUIDs map[uint32]bool, syncMode bool) error { - if !syncMode { - return nil // Archive mode - don't remove anything - } - - // Get all existing document IDs for this mailbox from CouchDB - existingDocs, err := c.GetAllMailDocumentIDs(ctx, dbName, mailbox) - if err != nil { - return fmt.Errorf("failed to get existing documents: %w", err) - } - - // Find documents that should be removed (exist in CouchDB but not in mail account) - var toDelete []string - for docID := range existingDocs { - // Extract UID from document ID (format: "mailbox_uid") - parts := strings.Split(docID, "_") - if len(parts) < 2 { - continue - } - - uidStr := parts[len(parts)-1] - uid := uint32(0) - if _, err := fmt.Sscanf(uidStr, "%d", &uid); err != nil { - continue - } - - // If this UID is not in the current mail account, mark for deletion - if !currentMessageUIDs[uid] { - toDelete = append(toDelete, docID) - } - } - - // Delete documents that are no longer in the mail account - for _, docID := range toDelete { - if err := c.DeleteDocument(ctx, dbName, docID); err != nil { - return fmt.Errorf("failed to delete document %s: %w", docID, err) - } - } - - if len(toDelete) > 0 { - fmt.Printf(" Sync mode: Removed %d documents no longer in mail account\n", len(toDelete)) - } - - return nil -} - -// GetSyncMetadata retrieves the sync metadata for a specific mailbox -func (c *Client) GetSyncMetadata(ctx context.Context, dbName, mailbox string) (*SyncMetadata, error) { - db := c.DB(dbName) - if db.Err() != nil { - return nil, db.Err() - } - - metadataID := fmt.Sprintf("sync_metadata_%s", mailbox) - row := db.Get(ctx, metadataID) - if row.Err() != nil { - // If metadata doesn't exist, return nil (not an error for first sync) - return nil, nil - } - - var metadata SyncMetadata - if err := row.ScanDoc(&metadata); err != nil { - return nil, fmt.Errorf("failed to scan sync metadata: %w", err) - } - - return &metadata, nil -} - -// StoreSyncMetadata stores or updates sync metadata for a mailbox -func (c *Client) StoreSyncMetadata(ctx context.Context, dbName string, metadata *SyncMetadata) error { - db := c.DB(dbName) - if db.Err() != nil { - return db.Err() - } - - metadata.ID = fmt.Sprintf("sync_metadata_%s", metadata.Mailbox) - metadata.DocType = "sync_metadata" - metadata.UpdatedAt = time.Now() - - // Check if metadata already exists to get current revision - existing, err := c.GetSyncMetadata(ctx, dbName, metadata.Mailbox) - if err != nil { - return fmt.Errorf("failed to check existing sync metadata: %w", err) - } - - if existing != nil { - metadata.Rev = existing.Rev - } - - _, err = db.Put(ctx, metadata.ID, metadata) - if err != nil { - return fmt.Errorf("failed to store sync metadata: %w", err) - } - - return nil -} diff --git a/go/go.mod b/go/go.mod deleted file mode 100644 index 377160a..0000000 --- a/go/go.mod +++ /dev/null @@ -1,16 +0,0 @@ -module mail2couch - -go 1.24.4 - -require ( - github.com/emersion/go-imap/v2 v2.0.0-beta.5 - github.com/go-kivik/kivik/v4 v4.4.0 -) - -require ( - github.com/emersion/go-message v0.18.1 // indirect - github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 // indirect - github.com/google/uuid v1.6.0 // indirect - golang.org/x/net v0.25.0 // indirect - golang.org/x/sync v0.10.0 // indirect -) diff --git a/go/go.sum b/go/go.sum deleted file mode 100644 index 7851761..0000000 --- a/go/go.sum +++ /dev/null @@ -1,65 +0,0 @@ -github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0= -github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ= -github.com/cenkalti/backoff/v4 v4.1.3 h1:cFAlzYUlVYDysBEH2T5hyJZMh3+5+WCBvSnK6Q8UtC4= -github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw= -github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM= -github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= -github.com/emersion/go-imap/v2 v2.0.0-beta.5 h1:H3858DNmBuXyMK1++YrQIRdpKE1MwBc+ywBtg3n+0wA= -github.com/emersion/go-imap/v2 v2.0.0-beta.5/go.mod h1:BZTFHsS1hmgBkFlHqbxGLXk2hnRqTItUgwjSSCsYNAk= -github.com/emersion/go-message v0.18.1 h1:tfTxIoXFSFRwWaZsgnqS1DSZuGpYGzSmCZD8SK3QA2E= -github.com/emersion/go-message v0.18.1/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA= -github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 h1:hH4PQfOndHDlpzYfLAAfl63E8Le6F2+EL/cdhlkyRJY= -github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ= -github.com/go-kivik/kivik/v4 v4.4.0 h1:1YMqNvRMIIC+CJUtyldD7c4Czl6SqdUcnbusCoFOTfk= -github.com/go-kivik/kivik/v4 v4.4.0/go.mod h1:DnPzIEO7CcLOqJNuqxuo7EMZeK4bPsEbUSSmAfi+tL4= -github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI= -github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY= -github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= -github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= -github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g= -github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k= -github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0 h1:nHoRIX8iXob3Y2kdt9KsjyIb7iApSvb3vgsd93xb5Ow= -github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0/go.mod h1:c1tRKs5Tx7E2+uHGSyyncziFjvGpgv4H2HrqXeUQ/Uk= -github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4= -github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0= -github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U= -github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= -github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY= -gitlab.com/flimzy/testy v0.14.0 h1:2nZV4Wa1OSJb3rOKHh0GJqvvhtE03zT+sKnPCI0owfQ= -gitlab.com/flimzy/testy v0.14.0/go.mod h1:m3aGuwdXc+N3QgnH+2Ar2zf1yg0UxNdIaXKvC5SlfMk= -golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w= -golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc= -golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4= -golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs= -golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s= -golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg= -golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c= -golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs= -golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac= -golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM= -golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM= -golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ= -golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk= -golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY= -golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= -golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= -golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo= -golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8= -golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k= -golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ= -golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ= -golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ= -golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8= -golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU= -golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ= -golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo= -golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc= -golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU= -golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0= -gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= -gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/go/mail/imap.go b/go/mail/imap.go deleted file mode 100644 index 63c7d7e..0000000 --- a/go/mail/imap.go +++ /dev/null @@ -1,481 +0,0 @@ -package mail - -import ( - "bytes" - "fmt" - "io" - "log" - "mime" - "path/filepath" - "strings" - "time" - - "github.com/emersion/go-imap/v2" - "github.com/emersion/go-imap/v2/imapclient" - "github.com/emersion/go-message" - "mail2couch/config" -) - -// ImapClient wraps the IMAP client -type ImapClient struct { - *imapclient.Client -} - -// Message represents an email message retrieved from IMAP -type Message struct { - UID uint32 - From []string - To []string - Subject string - Date time.Time - Body string - Headers map[string][]string - Attachments []Attachment -} - -// Attachment represents an email attachment -type Attachment struct { - Filename string - ContentType string - Content []byte -} - -// NewImapClient creates a new IMAP client from the configuration -func NewImapClient(source *config.MailSource) (*ImapClient, error) { - addr := fmt.Sprintf("%s:%d", source.Host, source.Port) - - var client *imapclient.Client - var err error - - // Try TLS first for standard IMAPS ports (993, 465) - if source.Port == 993 || source.Port == 465 { - client, err = imapclient.DialTLS(addr, nil) - } else { - // Use insecure connection for other ports (143, 3143, etc.) - client, err = imapclient.DialInsecure(addr, nil) - } - - if err != nil { - return nil, fmt.Errorf("failed to dial IMAP server: %w", err) - } - - if err := client.Login(source.User, source.Password).Wait(); err != nil { - return nil, fmt.Errorf("failed to login: %w", err) - } - - return &ImapClient{client}, nil -} - -// ListMailboxes lists all available mailboxes -func (c *ImapClient) ListMailboxes() ([]string, error) { - var mailboxes []string - cmd := c.List("", "*", nil) - - infos, err := cmd.Collect() - if err != nil { - return nil, err - } - - for _, info := range infos { - mailboxes = append(mailboxes, info.Mailbox) - } - - return mailboxes, nil -} - -// GetMessages retrieves messages from a specific mailbox with filtering support -// Returns messages and a map of all current UIDs in the mailbox -// maxMessages: 0 means no limit, > 0 limits the number of messages to fetch -// since: if provided, only fetch messages newer than this date (for incremental sync) -func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages int, messageFilter *config.MessageFilter) ([]*Message, map[uint32]bool, error) { - // Select the mailbox - mbox, err := c.Select(mailbox, nil).Wait() - if err != nil { - return nil, nil, fmt.Errorf("failed to select mailbox %s: %w", mailbox, err) - } - - if mbox.NumMessages == 0 { - return []*Message{}, make(map[uint32]bool), nil - } - - var messages []*Message - currentUIDs := make(map[uint32]bool) - - // First, get all current UIDs in the mailbox for sync purposes - allUIDsSet := imap.SeqSet{} - allUIDsSet.AddRange(1, mbox.NumMessages) - - // Fetch UIDs for all messages to track current state - uidCmd := c.Fetch(allUIDsSet, &imap.FetchOptions{UID: true}) - for { - msg := uidCmd.Next() - if msg == nil { - break - } - - data, err := msg.Collect() - if err != nil { - continue - } - - if data.UID != 0 { - currentUIDs[uint32(data.UID)] = true - } - } - uidCmd.Close() - - // Determine which messages to fetch based on since date - var seqSet imap.SeqSet - - if since != nil { - // Use IMAP SEARCH to find messages since the specified date - searchCriteria := &imap.SearchCriteria{ - Since: *since, - } - - searchCmd := c.Search(searchCriteria, nil) - searchResults, err := searchCmd.Wait() - if err != nil { - log.Printf("IMAP SEARCH failed, falling back to fetch all: %v", err) - // Fall back to fetching all messages - numToFetch := mbox.NumMessages - if maxMessages > 0 && int(numToFetch) > maxMessages { - numToFetch = uint32(maxMessages) - } - seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) - } else { - // Convert search results to sequence set - searchSeqNums := searchResults.AllSeqNums() - if len(searchSeqNums) == 0 { - return []*Message{}, currentUIDs, nil - } - - // Limit results if maxMessages is specified - if maxMessages > 0 && len(searchSeqNums) > maxMessages { - searchSeqNums = searchSeqNums[len(searchSeqNums)-maxMessages:] - } - - for _, seqNum := range searchSeqNums { - seqSet.AddNum(seqNum) - } - } - } else { - // No since date specified, fetch recent messages up to maxMessages - numToFetch := mbox.NumMessages - if maxMessages > 0 && int(numToFetch) > maxMessages { - numToFetch = uint32(maxMessages) - } - - if numToFetch == 0 { - return []*Message{}, currentUIDs, nil - } - - // Fetch the most recent messages - seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) - } - - // Fetch message data - get envelope and full message body - options := &imap.FetchOptions{ - Envelope: true, - UID: true, - BodySection: []*imap.FetchItemBodySection{ - {}, // Empty section gets the entire message - }, - } - - fetchCmd := c.Fetch(seqSet, options) - - for { - msg := fetchCmd.Next() - if msg == nil { - break - } - - parsedMsg, err := c.parseMessage(msg) - if err != nil { - log.Printf("Failed to parse message: %v", err) - continue - } - - // Apply message-level keyword filtering - if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) { - continue // Skip this message due to keyword filter - } - - messages = append(messages, parsedMsg) - } - - if err := fetchCmd.Close(); err != nil { - return nil, nil, fmt.Errorf("failed to fetch messages: %w", err) - } - - return messages, currentUIDs, nil -} - -// parseMessage parses an IMAP fetch response into our Message struct -func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) { - msg := &Message{ - UID: fetchMsg.SeqNum, // Using sequence number for now - Headers: make(map[string][]string), - Attachments: []Attachment{}, - } - - // Collect all fetch data first - buffer, err := fetchMsg.Collect() - if err != nil { - return nil, fmt.Errorf("failed to collect fetch data: %w", err) - } - - // Parse envelope for basic headers - if buffer.Envelope != nil { - env := buffer.Envelope - msg.Subject = env.Subject - msg.Date = env.Date - - // Parse From addresses - for _, addr := range env.From { - if addr.Mailbox != "" { - fullAddr := addr.Mailbox - if addr.Host != "" { - fullAddr = addr.Mailbox + "@" + addr.Host - } - msg.From = append(msg.From, fullAddr) - } - } - - // Parse To addresses - for _, addr := range env.To { - if addr.Mailbox != "" { - fullAddr := addr.Mailbox - if addr.Host != "" { - fullAddr = addr.Mailbox + "@" + addr.Host - } - msg.To = append(msg.To, fullAddr) - } - } - } - - // Get UID if available - if buffer.UID != 0 { - msg.UID = uint32(buffer.UID) - } - - // Parse full message content - if len(buffer.BodySection) > 0 { - bodyBuffer := buffer.BodySection[0] - reader := bytes.NewReader(bodyBuffer.Bytes) - - // Parse the message using go-message - entity, err := message.Read(reader) - if err != nil { - return nil, fmt.Errorf("failed to parse message: %w", err) - } - - // Extract headers - header := entity.Header - for field := header.Fields(); field.Next(); { - key := field.Key() - value, _ := field.Text() - msg.Headers[key] = append(msg.Headers[key], value) - } - - // Parse message body and attachments - if err := c.parseMessageBody(entity, msg); err != nil { - return nil, fmt.Errorf("failed to parse message body: %w", err) - } - } - - return msg, nil -} - -// parseMessageBody extracts the body and attachments from a message entity -func (c *ImapClient) parseMessageBody(entity *message.Entity, msg *Message) error { - mediaType, _, err := entity.Header.ContentType() - if err != nil { - // Default to text/plain if no content type - mediaType = "text/plain" - } - - if strings.HasPrefix(mediaType, "multipart/") { - // Handle multipart message - mr := entity.MultipartReader() - if mr == nil { - return fmt.Errorf("failed to create multipart reader") - } - - for { - part, err := mr.NextPart() - if err == io.EOF { - break - } - if err != nil { - return fmt.Errorf("failed to read multipart: %w", err) - } - - if err := c.parseMessagePart(part, msg); err != nil { - log.Printf("Failed to parse message part: %v", err) - // Continue processing other parts - } - } - } else { - // Handle single part message - if err := c.parseMessagePart(entity, msg); err != nil { - return err - } - } - - return nil -} - -// parseMessagePart processes a single message part (body or attachment) -func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) error { - mediaType, params, err := entity.Header.ContentType() - if err != nil { - mediaType = "text/plain" - } - - // Get content disposition - disposition, dispositionParams, _ := entity.Header.ContentDisposition() - - // Determine if this is an attachment - isAttachment := disposition == "attachment" || - (disposition == "inline" && dispositionParams["filename"] != "") || - params["name"] != "" - - if isAttachment { - // Handle attachment - filename := dispositionParams["filename"] - if filename == "" { - filename = params["name"] - } - if filename == "" { - filename = "unnamed_attachment" - } - - // Decode filename if needed - decoder := &mime.WordDecoder{} - filename, _ = decoder.DecodeHeader(filename) - - // Read attachment content - content, err := io.ReadAll(entity.Body) - if err != nil { - return fmt.Errorf("failed to read attachment content: %w", err) - } - - attachment := Attachment{ - Filename: filename, - ContentType: mediaType, - Content: content, - } - - msg.Attachments = append(msg.Attachments, attachment) - } else if strings.HasPrefix(mediaType, "text/") && msg.Body == "" { - // Handle text body (only take the first text part as body) - bodyBytes, err := io.ReadAll(entity.Body) - if err != nil { - return fmt.Errorf("failed to read message body: %w", err) - } - msg.Body = string(bodyBytes) - } - - return nil -} - -// ShouldProcessMailbox checks if a mailbox should be processed based on filters with wildcard support -func (c *ImapClient) ShouldProcessMailbox(mailbox string, filter *config.FolderFilter) bool { - // If include list is specified, mailbox must match at least one pattern - if len(filter.Include) > 0 { - found := false - for _, pattern := range filter.Include { - // Handle special case: "*" means include all folders - if pattern == "*" { - found = true - break - } - // Use filepath.Match for wildcard pattern matching - if matched, err := filepath.Match(pattern, mailbox); err == nil && matched { - found = true - break - } - // Also support exact string matching for backwards compatibility - if mailbox == pattern { - found = true - break - } - } - if !found { - return false - } - } - - // If exclude list is specified, mailbox must not match any exclude pattern - for _, pattern := range filter.Exclude { - // Use filepath.Match for wildcard pattern matching - if matched, err := filepath.Match(pattern, mailbox); err == nil && matched { - return false - } - // Also support exact string matching for backwards compatibility - if mailbox == pattern { - return false - } - } - - return true -} - -// ShouldProcessMessage checks if a message should be processed based on keyword filters -func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool { - // Check subject keywords - if len(filter.SubjectKeywords) > 0 { - if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) { - return false - } - } - - // Check sender keywords - if len(filter.SenderKeywords) > 0 { - senderMatch := false - for _, sender := range msg.From { - if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) { - senderMatch = true - break - } - } - if !senderMatch { - return false - } - } - - // Check recipient keywords - if len(filter.RecipientKeywords) > 0 { - recipientMatch := false - for _, recipient := range msg.To { - if c.containsAnyKeyword(strings.ToLower(recipient), filter.RecipientKeywords) { - recipientMatch = true - break - } - } - if !recipientMatch { - return false - } - } - - return true -} - -// containsAnyKeyword checks if the text contains any of the specified keywords (case-insensitive) -func (c *ImapClient) containsAnyKeyword(text string, keywords []string) bool { - for _, keyword := range keywords { - if strings.Contains(text, strings.ToLower(keyword)) { - return true - } - } - return false -} - -// Logout logs the client out -func (c *ImapClient) Logout() { - if err := c.Client.Logout(); err != nil { - log.Printf("Failed to logout: %v", err) - } -} diff --git a/go/mail2couch b/go/mail2couch deleted file mode 100755 index 2133741..0000000 Binary files a/go/mail2couch and /dev/null differ diff --git a/go/main.go b/go/main.go deleted file mode 100644 index 8d4b661..0000000 --- a/go/main.go +++ /dev/null @@ -1,205 +0,0 @@ -package main - -import ( - "context" - "fmt" - "log" - "time" - - "mail2couch/config" - "mail2couch/couch" - "mail2couch/mail" -) - -func main() { - args := config.ParseCommandLine() - - cfg, err := config.LoadConfigWithDiscovery(args) - if err != nil { - log.Fatalf("Failed to load configuration: %v", err) - } - - // Initialize CouchDB client - couchClient, err := couch.NewClient(&cfg.CouchDb) - if err != nil { - log.Fatalf("Failed to create CouchDB client: %v", err) - } - - fmt.Printf("Found %d mail source(s) to process.\n", len(cfg.MailSources)) - for _, source := range cfg.MailSources { - if !source.Enabled { - continue - } - - // Generate per-account database name - dbName := couch.GenerateAccountDBName(source.Name, source.User) - - // Ensure the account-specific database exists - ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) - err = couchClient.EnsureDB(ctx, dbName) - cancel() - - if err != nil { - log.Printf("Could not ensure CouchDB database '%s' exists (is it running?): %v", dbName, err) - continue - } else { - fmt.Printf("CouchDB database '%s' is ready for account: %s\n", dbName, source.Name) - } - - fmt.Printf(" - Processing source: %s\n", source.Name) - if source.Protocol == "imap" { - err := processImapSource(&source, couchClient, dbName, args.MaxMessages) - if err != nil { - log.Printf(" ERROR: Failed to process IMAP source %s: %v", source.Name, err) - } - } - } -} - -func processImapSource(source *config.MailSource, couchClient *couch.Client, dbName string, maxMessages int) error { - fmt.Printf(" Connecting to IMAP server: %s:%d\n", source.Host, source.Port) - imapClient, err := mail.NewImapClient(source) - if err != nil { - return fmt.Errorf("failed to connect to IMAP server: %w", err) - } - defer imapClient.Logout() - - fmt.Println(" IMAP connection successful.") - - mailboxes, err := imapClient.ListMailboxes() - if err != nil { - return fmt.Errorf("failed to list mailboxes: %w", err) - } - - fmt.Printf(" Found %d mailboxes.\n", len(mailboxes)) - - // Parse the since date from config if provided (fallback for first sync) - var configSinceDate *time.Time - if source.MessageFilter.Since != "" { - parsed, err := time.Parse("2006-01-02", source.MessageFilter.Since) - if err != nil { - log.Printf(" WARNING: Invalid since date format '%s', ignoring filter", source.MessageFilter.Since) - } else { - configSinceDate = &parsed - } - } - - totalMessages := 0 - totalStored := 0 - - // Process each mailbox - for _, mailbox := range mailboxes { - // Check if this mailbox should be processed based on filters - if !imapClient.ShouldProcessMailbox(mailbox, &source.FolderFilter) { - fmt.Printf(" Skipping mailbox: %s (filtered)\n", mailbox) - continue - } - - fmt.Printf(" Processing mailbox: %s (mode: %s)\n", mailbox, source.Mode) - - // Get sync metadata to determine incremental sync date - syncCtx, syncCancel := context.WithTimeout(context.Background(), 10*time.Second) - syncMetadata, err := couchClient.GetSyncMetadata(syncCtx, dbName, mailbox) - syncCancel() - if err != nil { - log.Printf(" ERROR: Failed to get sync metadata for %s: %v", mailbox, err) - continue - } - - // Determine the since date for incremental sync - var sinceDate *time.Time - if syncMetadata != nil { - // Use last sync time for incremental sync - sinceDate = &syncMetadata.LastSyncTime - fmt.Printf(" Incremental sync since: %s (last synced %d messages)\n", - sinceDate.Format("2006-01-02 15:04:05"), syncMetadata.MessageCount) - } else { - // First sync - use config since date if available - sinceDate = configSinceDate - if sinceDate != nil { - fmt.Printf(" First sync since: %s (from config)\n", sinceDate.Format("2006-01-02")) - } else { - fmt.Printf(" First full sync (no date filter)\n") - } - } - - // Retrieve messages from the mailbox - messages, currentUIDs, err := imapClient.GetMessages(mailbox, sinceDate, maxMessages, &source.MessageFilter) - if err != nil { - log.Printf(" ERROR: Failed to get messages from %s: %v", mailbox, err) - continue - } - - // Perform sync/archive logic - mailboxSyncCtx, mailboxSyncCancel := context.WithTimeout(context.Background(), 30*time.Second) - err = couchClient.SyncMailbox(mailboxSyncCtx, dbName, mailbox, currentUIDs, source.IsSyncMode()) - mailboxSyncCancel() - if err != nil { - log.Printf(" ERROR: Failed to sync mailbox %s: %v", mailbox, err) - continue - } - - if len(messages) == 0 { - fmt.Printf(" No new messages found in %s\n", mailbox) - continue - } - - fmt.Printf(" Found %d messages in %s\n", len(messages), mailbox) - totalMessages += len(messages) - - // Convert messages to CouchDB documents - var docs []*couch.MailDocument - for _, msg := range messages { - doc := couch.ConvertMessage(msg, mailbox) - docs = append(docs, doc) - } - - // Store messages in CouchDB with attachments - ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) - stored := 0 - for i, doc := range docs { - err := couchClient.StoreMessage(ctx, dbName, doc, messages[i]) - if err != nil { - log.Printf(" ERROR: Failed to store message %s: %v", doc.ID, err) - } else { - stored++ - } - } - cancel() - - fmt.Printf(" Stored %d/%d messages from %s\n", stored, len(messages), mailbox) - totalStored += stored - - // Update sync metadata after successful processing - if len(messages) > 0 { - // Find the highest UID processed - var maxUID uint32 - for _, msg := range messages { - if msg.UID > maxUID { - maxUID = msg.UID - } - } - - // Create/update sync metadata - newMetadata := &couch.SyncMetadata{ - Mailbox: mailbox, - LastSyncTime: time.Now(), - LastMessageUID: maxUID, - MessageCount: stored, - } - - // Store sync metadata - metadataCtx, metadataCancel := context.WithTimeout(context.Background(), 10*time.Second) - err = couchClient.StoreSyncMetadata(metadataCtx, dbName, newMetadata) - metadataCancel() - if err != nil { - log.Printf(" WARNING: Failed to store sync metadata for %s: %v", mailbox, err) - } else { - fmt.Printf(" Updated sync metadata (last UID: %d)\n", maxUID) - } - } - } - - fmt.Printf(" Summary: Processed %d messages, stored %d new messages\n", totalMessages, totalStored) - return nil -} diff --git a/test/README.md b/test/README.md deleted file mode 100644 index bd4c180..0000000 --- a/test/README.md +++ /dev/null @@ -1,250 +0,0 @@ -# mail2couch Test Environment - -This directory contains a complete test environment for mail2couch using Podman containers. - -## Overview - -The test environment provides: -- **CouchDB**: Database for storing email messages -- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages -- **Test Configuration**: Ready-to-use config for testing both sync and archive modes - -## Quick Start - -### Run Basic Integration Tests -```bash -./run-tests.sh -``` -This comprehensive test will: -1. Start all containers with cleanup -2. Populate test data -3. Build and run mail2couch -4. Verify database creation and document storage -5. Test incremental sync behavior -6. Clean up automatically - -### Run Wildcard Pattern Tests -```bash -./test-wildcard-patterns.sh -``` -This will test various wildcard folder patterns including: -- `*` (all folders) -- `Work*` (prefix patterns) -- `*/Drafts` (subfolder patterns) -- Complex include/exclude combinations - -### Run Incremental Sync Tests -```bash -./test-incremental-sync.sh -``` -This will test incremental synchronization functionality: -- First sync establishes baseline -- New messages are added to test accounts -- Second sync should only fetch new messages -- Sync metadata tracking and IMAP SEARCH with SINCE - -### Manual Testing Environment -```bash -# Start persistent test environment (for manual experimentation) -./start-test-env.sh - -# Run mail2couch manually with different configurations -cd ../go -./mail2couch -config ../test/config-test.json -./mail2couch -config ../test/config-wildcard-examples.json - -# Stop test environment when done -cd ../test -./stop-test-env.sh -``` - -## Test Scripts Overview - -### Automated Testing (Recommended) -- **`./run-tests.sh`**: Complete integration test with automatic cleanup - - Starts containers, populates data, runs mail2couch, verifies results - - Tests basic functionality, database creation, and incremental sync - - Cleans up automatically - perfect for CI/CD or quick validation - -### Specialized Feature Testing -- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing - - Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns - - Self-contained with own setup/teardown -- **`./test-incremental-sync.sh`**: Incremental synchronization testing - - Tests sync metadata tracking and IMAP SEARCH with SINCE - - Multi-step validation: baseline sync → add messages → incremental sync - - Self-contained with own setup/teardown - -### Manual Testing Environment -- **`./start-test-env.sh`**: Start persistent test containers - - Keeps environment running for manual experimentation - - Populates test data once - - Use with different configurations for development -- **`./stop-test-env.sh`**: Clean up manual test environment - - Only needed after using `start-test-env.sh` - -## Test Accounts - -The test environment includes these IMAP accounts: - -| Username | Password | Mode | Folder Pattern | Purpose | -|----------|----------|------|---------------|---------| -| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test | -| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test | -| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test | -| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test | - -Each account contains: -- 10 messages in INBOX (every 3rd has an attachment) -- 3 messages in each additional folder -- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash` -- Various message types for comprehensive wildcard testing - -## Services - -### CouchDB -- **URL**: http://localhost:5984 -- **Admin**: `admin` / `password` -- **Web UI**: http://localhost:5984/_utils - -### GreenMail Server -- **Host**: localhost -- **IMAP Port**: 3143 (plain) -- **IMAPS Port**: 3993 (SSL) -- **SMTP Port**: 3025 -- **Server**: GreenMail (Java-based test server) - -## Database Structure - -mail2couch will create separate databases for each mail source (with `m2c_` prefix): -- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode) -- `m2c_work_pattern_test` - Work Pattern Test (sync mode) -- `m2c_specific_folders_only` - Specific Folders Only (archive mode) -- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode) - -Each database contains documents with: -- `mailbox` field indicating the origin folder -- Native CouchDB attachments for email attachments -- Full message headers and body content - -## Testing Sync vs Archive Modes - -### Sync Mode (`syncuser`) -- Database exactly matches mail account -- If messages are deleted from IMAP, they're removed from CouchDB -- 1-to-1 relationship - -### Archive Mode (`archiveuser`, `testuser1`) -- Database preserves all messages ever seen -- Messages deleted from IMAP remain in CouchDB -- Archive/backup behavior - -## Wildcard Pattern Examples - -The test environment demonstrates these wildcard patterns: - -### All Folders Pattern (`*`) -```json -{ - "folderFilter": { - "include": ["*"], - "exclude": ["Drafts", "Trash"] - } -} -``` -Processes all folders except Drafts and Trash. - -### Work Pattern (`Work*`) -```json -{ - "folderFilter": { - "include": ["Work*", "Important*", "INBOX"], - "exclude": ["*Temp*"] - } -} -``` -Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp. - -### Specific Folders -```json -{ - "folderFilter": { - "include": ["INBOX", "Sent", "Personal"], - "exclude": [] - } -} -``` -Only processes the exact named folders. - -### Subfolder Pattern (`Work/*`) -```json -{ - "folderFilter": { - "include": ["Work/*", "Archive/*"], - "exclude": ["*/Drafts"] - } -} -``` -Includes all subfolders under Work and Archive, but excludes any Drafts subfolder. - -## File Structure - -``` -test/ -├── podman-compose.yml # Container orchestration (GreenMail + CouchDB) -├── config-test.json # Main test configuration with wildcard examples -├── config-wildcard-examples.json # Advanced wildcard patterns -├── run-tests.sh # Automated integration test (recommended) -├── test-wildcard-patterns.sh # Specialized wildcard pattern testing -├── test-incremental-sync.sh # Specialized incremental sync testing -├── start-test-env.sh # Start persistent test environment -├── stop-test-env.sh # Stop test environment -├── populate-greenmail.py # Create test messages across multiple folders -├── dovecot/ # Dovecot configuration (legacy, unused) -└── README.md # This file -``` - -## Prerequisites - -- Podman and podman-compose -- OpenSSL (for certificate generation) -- curl and nc (for connectivity checks) -- Go (for building mail2couch) - -## Troubleshooting - -### Containers won't start -```bash -# Check podman status -podman ps -a - -# View logs -podman logs mail2couch_test_couchdb -podman logs mail2couch_test_imap -``` - -### CouchDB connection issues -- Verify CouchDB is running: `curl http://localhost:5984` -- Check admin credentials: `admin/password` - -### IMAP connection issues -- Test IMAP connection: `nc -z localhost 143` -- Check Dovecot logs: `podman logs mail2couch_test_imap` - -### Permission issues -- Ensure scripts are executable: `chmod +x *.sh` -- Check file permissions in dovecot directory - -## Advanced Usage - -### Add custom test messages -Edit `populate-test-messages.sh` to create additional test scenarios. - -### Modify IMAP configuration -Edit `dovecot/dovecot.conf` and restart containers. - -### Test with SSL -Update `config-test.json` to use port 993 and enable SSL. - -### Custom test scenarios -Create additional configuration files for specific test cases. \ No newline at end of file diff --git a/test/config-test.json b/test/config-test.json deleted file mode 100644 index e618c2e..0000000 --- a/test/config-test.json +++ /dev/null @@ -1,74 +0,0 @@ -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "Wildcard All Folders Test", - "enabled": true, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "testuser1", - "password": "password123", - "mode": "archive", - "folderFilter": { - "include": ["*"], - "exclude": ["Drafts", "Trash"] - }, - "messageFilter": { - "subjectKeywords": ["meeting", "important"], - "senderKeywords": ["@company.com"] - } - }, - { - "name": "Work Pattern Test", - "enabled": true, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "syncuser", - "password": "syncpass", - "mode": "sync", - "folderFilter": { - "include": ["Work*", "Important*", "INBOX"], - "exclude": ["*Temp*"] - }, - "messageFilter": { - "recipientKeywords": ["support@", "team@"] - } - }, - { - "name": "Specific Folders Only", - "enabled": true, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "archiveuser", - "password": "archivepass", - "mode": "archive", - "folderFilter": { - "include": ["INBOX", "Sent", "Personal"], - "exclude": [] - }, - "messageFilter": {} - }, - { - "name": "Subfolder Pattern Test", - "enabled": false, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "testuser2", - "password": "password456", - "mode": "archive", - "folderFilter": { - "include": ["Work/*", "Archive/*"], - "exclude": ["*/Drafts"] - }, - "messageFilter": {} - } - ] -} \ No newline at end of file diff --git a/test/config-wildcard-examples.json b/test/config-wildcard-examples.json deleted file mode 100644 index 9db4df1..0000000 --- a/test/config-wildcard-examples.json +++ /dev/null @@ -1,71 +0,0 @@ -{ - "couchDb": { - "url": "http://localhost:5984", - "user": "admin", - "password": "password" - }, - "mailSources": [ - { - "name": "All Folders Example", - "enabled": true, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "testuser1", - "password": "password123", - "mode": "archive", - "folderFilter": { - "include": ["*"], - "exclude": ["Drafts", "Trash", "Spam"] - }, - "messageFilter": {} - }, - { - "name": "Inbox and Sent Only", - "enabled": false, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "testuser2", - "password": "password456", - "mode": "sync", - "folderFilter": { - "include": ["INBOX", "Sent"], - "exclude": [] - }, - "messageFilter": {} - }, - { - "name": "Work Folders Pattern", - "enabled": false, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "workuser", - "password": "workpass", - "mode": "archive", - "folderFilter": { - "include": ["Work*", "Projects*", "INBOX"], - "exclude": ["*Temp*", "*Draft*"] - }, - "messageFilter": { - "senderKeywords": ["@company.com"] - } - }, - { - "name": "Archive Pattern Example", - "enabled": false, - "protocol": "imap", - "host": "localhost", - "port": 3143, - "user": "archiveuser", - "password": "archivepass", - "mode": "archive", - "folderFilter": { - "include": ["Archive*", "*Important*", "INBOX"], - "exclude": ["*Temp"] - }, - "messageFilter": {} - } - ] -} \ No newline at end of file diff --git a/test/dovecot/dovecot.conf b/test/dovecot/dovecot.conf deleted file mode 100644 index af20de0..0000000 --- a/test/dovecot/dovecot.conf +++ /dev/null @@ -1,79 +0,0 @@ -# Dovecot configuration for testing mail2couch - -# Basic settings -protocols = imap -listen = * - -# SSL/TLS settings - make optional for easier testing -ssl = optional -ssl_cert = /dev/null 2>&1; then - addgroup -g 97 dovecot - adduser -D -u 97 -G dovecot -s /sbin/nologin dovecot -fi - -# Set proper ownership -chown -R dovecot:dovecot /var/mail -chown -R dovecot:dovecot /var/run/dovecot -chown -R root:dovecot /etc/dovecot -chmod -R 0640 /etc/dovecot -chmod 0644 /etc/dovecot/dovecot.conf - -# Generate SSL certificates if they don't exist -if [ ! -f /etc/dovecot/ssl/server.crt ] || [ ! -f /etc/dovecot/ssl/server.key ]; then - echo "Generating SSL certificates..." - mkdir -p /etc/dovecot/ssl - - # Generate DH parameters (small for testing) - openssl dhparam -out /etc/dovecot/ssl/dh.pem 1024 - - # Generate private key - openssl genrsa -out /etc/dovecot/ssl/server.key 2048 - - # Generate certificate - openssl req -new -key /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.csr -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost" - openssl x509 -req -days 365 -in /etc/dovecot/ssl/server.csr -signkey /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.crt - rm /etc/dovecot/ssl/server.csr -fi - -# Ensure SSL directory permissions -chown -R dovecot:dovecot /etc/dovecot/ssl -chmod 600 /etc/dovecot/ssl/server.key -chmod 644 /etc/dovecot/ssl/server.crt - -echo "Starting Dovecot..." -exec dovecot -F \ No newline at end of file diff --git a/test/dovecot/passwd b/test/dovecot/passwd deleted file mode 100644 index 51e3cc3..0000000 --- a/test/dovecot/passwd +++ /dev/null @@ -1,8 +0,0 @@ -# Password database for Dovecot testing -# Format: username:password - -# Test accounts with simple passwords for testing -testuser1:password123 -testuser2:password456 -syncuser:syncpass -archiveuser:archivepass \ No newline at end of file diff --git a/test/dovecot/ssl/dh.pem b/test/dovecot/ssl/dh.pem deleted file mode 100644 index 513492c..0000000 --- a/test/dovecot/ssl/dh.pem +++ /dev/null @@ -1,8 +0,0 @@ ------BEGIN DH PARAMETERS----- -MIIBDAKCAQEAjcUSAHFs60qgDRg/cT7byhuhF3vwZQhmm1QToCFgG4VWu/EOVXq2 -kHxjxmo3hBuJCqUZqTAyF91Tum7A2QuQhXFrxOpRF8EiyVSgBabjN/WcEHIow1uh -Vtb4JOcDl/Q9IJfFT6zyXdQQiHPBOWnpOBKXeQQQIx5plgsrmK0cTO2ZxtyrmHHp -wxtE3INKYuBlGH3Y0zghc+Hoezpf/hbIHZibGQ0l79EtBDQjqmqoDJCIiv5gsTt8 -9VpkR6FFvjWTNOb5qY10W/PRhLGjioX29bp1B6qW5PNJcd//cqrBLebKlkAoXnyx -x0uTUy6pmmIt5vdYxx0symrMXZEjrL7uzwIBAgICAOE= ------END DH PARAMETERS----- diff --git a/test/dovecot/ssl/server.crt b/test/dovecot/ssl/server.crt deleted file mode 100644 index 5641479..0000000 --- a/test/dovecot/ssl/server.crt +++ /dev/null @@ -1,20 +0,0 @@ ------BEGIN CERTIFICATE----- -MIIDVzCCAj+gAwIBAgIUYvkZSKbVH08s/3B70AW8IEpTB/kwDQYJKoZIhvcNAQEL -BQAwVDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx -EzARBgNVBAoMCk1haWwyQ291Y2gxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yNTA4 -MDExNDIyMzVaFw0yNjA4MDExNDIyMzVaMFQxCzAJBgNVBAYTAlVTMQ0wCwYDVQQI -DARUZXN0MQ0wCwYDVQQHDARUZXN0MRMwEQYDVQQKDApNYWlsMkNvdWNoMRIwEAYD -VQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs -leQcVLlKx7/bJGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vx -Td1mVCm9Lsd9bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDW -knco7/MEatakliVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331 -HEFdIqqccpimnIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwo -eN8zBzGFJe9xzqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5 -LGN5xSzXvFIzgP20y5s5AgMBAAGjITAfMB0GA1UdDgQWBBTkgYZGp2s74D+1ltyl -rudF/o7jODANBgkqhkiG9w0BAQsFAAOCAQEATc6ekhuk32meLuxhalz6lNwBxfDg -EG3gGUxNwehwgiNCcKIKQFtCwjJde6drOobkRDANtb7g3gSlAxlUCPsO6xnL1c6E -HhehFn++7HOpXvmEy/mnoqBL6PLzRZRMRlDynlPVV9Y82zsdrQiQEhGyNTfgP5dk -u9RMIMQl1hIK381V738b5MXfdpYhmRiTGEd6hCxCnzkx0OakCLM9lnJASr0dYPuh -LYKoClxhr3sV/JsgAmx91BuHGpzaPYQ2zFvCJSqD2ihM7zIl9K2bLIUR87/CznyH -JuPRbgt6/cxzwdqflP73j+TTZdlI4gckEA3H0WhNN4nB2SEjTgS+kDctMA== ------END CERTIFICATE----- diff --git a/test/dovecot/ssl/server.key b/test/dovecot/ssl/server.key deleted file mode 100644 index b7dd757..0000000 --- a/test/dovecot/ssl/server.key +++ /dev/null @@ -1,28 +0,0 @@ ------BEGIN PRIVATE KEY----- -MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsleQcVLlKx7/b -JGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vxTd1mVCm9Lsd9 -bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDWknco7/MEatak -liVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331HEFdIqqccpim -nIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwoeN8zBzGFJe9x -zqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5LGN5xSzXvFIz -gP20y5s5AgMBAAECggEAOfBaM76ns/iqKpYlamnjfIs7svotEjhrHcsub6fWvEsE -XLzRmSqHeWP+t55oo2XeL2zOCofvuUDGnQ+rXE2mHwzhP3FJzsOibm2qmtCJvZwe -ozRj4xTMLILGDnGRhHAJ21cxZM9lPNLnOzri0868DeYimib49xAdroLBLyKRgDGN -O7OAwA4KWAebZRBU8cowEF87WGAI2hOLJA5WIKX7X9SiRDx5sNnDpTpqJvyvuleR -D/wKGHzkQiZx8WNJx5A571dilfKEp8S49o4sJdz8DQ/4ruVov2Vi+nSdMEzHnp5m -M7ZlZ7IcJRJDKPrDYasmxvtM8EiKyJf87/DPANfYFwKBgQDa+ANr2pGg2oMJVJIq -r/mj8wtRsLRYFgfs96BUwp7e9Vo+Rx+E0uZxGGMul6hn4b2/TQljLfiX/CP8ZTyw -MStlNlnAXaF425JNuQIFQ2wGGiGdsx2I1WNw7co4UPVVjH11nr3o30g+NDJcp7Tq -rvpKvGbeoZtHzOj0bF7fA/B7qwKBgQDJxcUUEH5A50n3oUP99aK/DzSA7kcte2Aw -tjv9hbPnbOmcM24Fm+KU7bsRYt9QPa0PU2lV3O1KrHj4q+QRcPFl2P2mzZC+Hzmx -H8dEjMmH8YdrjGqethMoUHJCguNfskNwjgWFlxTSBLY+NffghXNzZgiF9d6WqF48 -iqwH+HsAqwKBgQCb/B2D0Xn4WnEKToKpgh6WGmcv1G9EaL1Qo75FYzcFoUaeItBj -MFIUssjEwiinh/pBssFDM9Zpfqar//pRkVVWjnc1P/3tOI1qbKbx1Ou5FRhpXNVn -SovCQMLTh2idfq1JAsJKh/TQyyItOxL4M5n9b2Tgp8MUTPaOWDzlJctEbQKBgEVu -oNq+sjNzY6iq/dKubEqC2PZlCGlGQ1t/2jTrhXTlrZ3qtLmJYvcMt4rMEzxxfNQB -SAYb+CvyHc60l87Ipsj9WovDwUMrS5b/8HpOWCtHmeoQb8Adt4nv5OGuWL/dgAeD -V7MYwjljFbNiruG8CnZzbgtrCCWf2o3KylgT0X/xAoGAUhSdBge5Vpg0JcT1VDgm -q5rgc6dD1LJtXfBaq3w4kHYK/iLFcPOLUKcIJXNbhMwWza/JwVYK6hsCIw3/b4va -NhJ8ABpC3fZqkl28glEF8bnrPAkE1akn2GiBaaEbTCQRMrhZ2SW3JCyjX6yCvvvz -m7b2ZpDMJEMIBmgrK70E3Oo= ------END PRIVATE KEY----- diff --git a/test/dovecot/users b/test/dovecot/users deleted file mode 100644 index 322dfc1..0000000 --- a/test/dovecot/users +++ /dev/null @@ -1,8 +0,0 @@ -# User database for Dovecot testing -# Format: username:uid:gid:home:shell - -# Test user accounts -testuser1::1001:1001:/var/mail/testuser1:: -testuser2::1002:1002:/var/mail/testuser2:: -syncuser::1003:1003:/var/mail/syncuser:: -archiveuser::1004:1004:/var/mail/archiveuser:: \ No newline at end of file diff --git a/test/generate-ssl.sh b/test/generate-ssl.sh deleted file mode 100755 index cfa0c0c..0000000 --- a/test/generate-ssl.sh +++ /dev/null @@ -1,28 +0,0 @@ -#!/bin/bash - -# Generate SSL certificates for Dovecot testing - -set -e - -CERT_DIR="dovecot/ssl" -mkdir -p "$CERT_DIR" - -# Generate DH parameters -echo "Generating DH parameters..." -openssl dhparam -out "$CERT_DIR/dh.pem" 2048 - -# Generate private key -echo "Generating private key..." -openssl genrsa -out "$CERT_DIR/server.key" 2048 - -# Generate certificate signing request -echo "Generating certificate..." -openssl req -new -key "$CERT_DIR/server.key" -out "$CERT_DIR/server.csr" -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost" - -# Generate self-signed certificate -openssl x509 -req -days 365 -in "$CERT_DIR/server.csr" -signkey "$CERT_DIR/server.key" -out "$CERT_DIR/server.crt" - -# Clean up CSR -rm "$CERT_DIR/server.csr" - -echo "SSL certificates generated successfully in $CERT_DIR/" \ No newline at end of file diff --git a/test/podman-compose.yml b/test/podman-compose.yml deleted file mode 100644 index ba24506..0000000 --- a/test/podman-compose.yml +++ /dev/null @@ -1,38 +0,0 @@ -version: '3.8' - -services: - # CouchDB for testing - couchdb: - image: docker.io/couchdb:3.3 - container_name: mail2couch_test_couchdb - environment: - - COUCHDB_USER=admin - - COUCHDB_PASSWORD=password - ports: - - "5984:5984" - volumes: - - couchdb_data:/opt/couchdb/data - networks: - - mail2couch_test - - # GreenMail IMAP server for testing - greenmail: - image: docker.io/greenmail/standalone:2.0.1 - container_name: mail2couch_test_imap - ports: - - "3143:3143" # IMAP - - "3993:3993" # IMAPS - - "3025:3025" # SMTP - environment: - - GREENMAIL_OPTS=-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.users=testuser1:password123@localhost,testuser2:password456@localhost,syncuser:syncpass@localhost,archiveuser:archivepass@localhost - networks: - - mail2couch_test - depends_on: - - couchdb - -volumes: - couchdb_data: - -networks: - mail2couch_test: - driver: bridge \ No newline at end of file diff --git a/test/populate-greenmail.py b/test/populate-greenmail.py deleted file mode 100755 index c79b593..0000000 --- a/test/populate-greenmail.py +++ /dev/null @@ -1,179 +0,0 @@ -#!/usr/bin/env python3 - -import imaplib -import email -from email.mime.text import MIMEText -from email.mime.multipart import MIMEMultipart -from email.mime.base import MIMEBase -from email import encoders -import time -import sys - -def create_simple_message(subject, body, from_addr="test-sender@example.com", to_addr="user@example.com"): - """Create a simple text message""" - msg = MIMEText(body) - msg['Subject'] = subject - msg['From'] = from_addr - msg['To'] = to_addr - msg['Date'] = email.utils.formatdate(localtime=True) - return msg.as_string() - -def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com", to_addr="user@example.com"): - """Create a message with an attachment""" - msg = MIMEMultipart() - msg['Subject'] = subject - msg['From'] = from_addr - msg['To'] = to_addr - msg['Date'] = email.utils.formatdate(localtime=True) - - # Add body - msg.attach(MIMEText(body, 'plain')) - - # Add attachment - part = MIMEBase('text', 'plain') - part.set_payload(attachment_content) - encoders.encode_base64(part) - part.add_header('Content-Disposition', 'attachment; filename="attachment.txt"') - msg.attach(part) - - return msg.as_string() - -def populate_user_mailbox(username, password, host='localhost', port=3143): - """Populate a user's mailbox with test messages""" - print(f"Connecting to {username}@{host}:{port}") - - try: - # Connect to IMAP server - imap = imaplib.IMAP4(host, port) - imap.login(username, password) - - # Create additional folders for testing wildcards - # These folders are designed to test various wildcard patterns - test_folders = [ - 'Sent', # Exact match - 'Work/Projects', # Work/* pattern - 'Work/Archive', # Work/* pattern - 'Work/Temp', # Work/* but excluded by *Temp* - 'Personal', # Exact match - 'Important/Urgent', # Important/* pattern - 'Important/Meetings', # Important/* pattern - 'Archive/2024', # Archive/* pattern - 'Archive/Projects', # Archive/* pattern - 'Archive/Drafts', # Archive/* but excluded by */Drafts - 'Drafts', # Should be excluded - 'Trash' # Should be excluded - ] - created_folders = ['INBOX'] # INBOX always exists - - for folder in test_folders: - try: - imap.create(folder) - created_folders.append(folder) - print(f" Created folder: {folder}") - except Exception as e: - print(f" Folder {folder} may already exist or creation failed: {e}") - # Try to select it to see if it exists - try: - imap.select(folder) - created_folders.append(folder) - print(f" Folder {folder} already exists") - except: - pass - - print(f"Available folders for {username}: {created_folders}") - - # Populate each folder with messages - for folder in created_folders[:3]: # Limit to first 3 folders to avoid too many messages - imap.select(folder) - print(f"Creating messages in {folder} for {username}...") - - # Create fewer messages per folder for testing - message_count = 3 if folder != 'INBOX' else 10 - - for i in range(1, message_count + 1): - if i % 3 == 0 and folder == 'INBOX': - # Every 3rd message has attachment (only in INBOX) - msg = create_message_with_attachment( - f"[{folder}] Test Message {i} with Attachment", - f"This is test message {i} in {folder} for {username} with an attachment.", - f"Sample attachment content for message {i} in {folder}", - "test-sender@example.com", - f"{username}@example.com" - ) - print(f" Created message {i} (with attachment)") - else: - # Simple message - msg = create_simple_message( - f"[{folder}] Test Message {i}", - f"This is test message {i} in {folder} for {username}.", - "test-sender@example.com", - f"{username}@example.com" - ) - print(f" Created message {i}") - - # Append message to current folder - imap.append(folder, None, None, msg.encode('utf-8')) - time.sleep(0.1) # Small delay to avoid overwhelming - - # Add special messages only to INBOX for keyword filtering tests - if folder == 'INBOX': - special_messages = [ - ("Important Meeting Reminder", "This is an important meeting message for testing keyword filters.", "manager@company.com", "team@company.com"), - ("Urgent: System Maintenance", "Important notification about system maintenance.", "admin@company.com", f"{username}@example.com"), - ("Regular Newsletter", "This is a regular newsletter message.", "newsletter@external.com", f"{username}@example.com"), - ("Team Meeting Notes", "Meeting notes from the team.", "secretary@company.com", "support@company.com"), - ("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀", "test-sender@example.com", f"{username}@example.com") - ] - - for subject, body, sender, recipient in special_messages: - msg = create_simple_message(subject, body, sender, recipient) - imap.append(folder, None, None, msg.encode('utf-8')) - print(f" Created special message: {subject} from {sender} to {recipient}") - time.sleep(0.1) - - imap.logout() - print(f"✅ Successfully created messages across {len(created_folders[:3])} folders for {username}") - return True - - except Exception as e: - print(f"❌ Error populating {username}: {e}") - return False - -def main(): - print("🚀 Populating GreenMail with test messages using IMAP...") - - # Test accounts - accounts = [ - ("testuser1", "password123"), - ("testuser2", "password456"), - ("syncuser", "syncpass"), - ("archiveuser", "archivepass") - ] - - # Wait for GreenMail to be ready - print("Waiting for GreenMail to be ready...") - time.sleep(5) - - success_count = 0 - for username, password in accounts: - if populate_user_mailbox(username, password): - success_count += 1 - time.sleep(1) # Brief pause between accounts - - print(f"\n🎉 Successfully populated {success_count}/{len(accounts)} accounts!") - - if success_count == len(accounts): - print("\n✅ All test accounts ready:") - for username, password in accounts: - print(f" - {username}:{password}@localhost") - print(f"\nGreenMail Services:") - print(f" - IMAP: localhost:3143") - print(f" - IMAPS: localhost:3993") - print(f" - SMTP: localhost:3025") - return 0 - else: - print(f"\n❌ Failed to populate {len(accounts) - success_count} accounts") - return 1 - -if __name__ == "__main__": - sys.exit(main()) \ No newline at end of file diff --git a/test/run-tests.sh b/test/run-tests.sh deleted file mode 100755 index 79d5a75..0000000 --- a/test/run-tests.sh +++ /dev/null @@ -1,128 +0,0 @@ -#!/bin/bash - -# Run basic integration tests with test containers -# This is a comprehensive test that handles its own setup and teardown - -set -e - -cd "$(dirname "$0")" - -echo "🚀 Running basic integration tests..." - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -NC='\033[0m' # No Color - -# Function to print colored output -print_status() { - echo -e "${GREEN}[INFO]${NC} $1" -} - -print_warning() { - echo -e "${YELLOW}[WARN]${NC} $1" -} - -print_error() { - echo -e "${RED}[ERROR]${NC} $1" -} - -# Cleanup function -cleanup() { - print_status "Cleaning up test containers..." - podman-compose -f podman-compose.yml down -v 2>/dev/null || true -} - -# Set up cleanup trap -trap cleanup EXIT - -# Start containers -print_status "Starting test containers..." -podman-compose -f podman-compose.yml up -d - -# Wait for containers to be ready -print_status "Waiting for containers to be ready..." -sleep 10 - -# Check if CouchDB is ready -print_status "Checking CouchDB connectivity..." -timeout=30 -while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do - timeout=$((timeout - 1)) - if [ $timeout -le 0 ]; then - print_error "CouchDB failed to start within 30 seconds" - exit 1 - fi - sleep 1 -done -print_status "CouchDB is ready!" - -# Check if IMAP server is ready -print_status "Checking IMAP server connectivity..." -timeout=30 -while ! nc -z localhost 3143 > /dev/null 2>&1; do - timeout=$((timeout - 1)) - if [ $timeout -le 0 ]; then - print_error "IMAP server failed to start within 30 seconds" - exit 1 - fi - sleep 1 -done -print_status "IMAP server is ready!" - -# Populate test messages -print_status "Populating test messages..." -python3 ./populate-greenmail.py - -# Build mail2couch -print_status "Building mail2couch..." -cd ../go -go build -o mail2couch . -cd ../test - -# Run mail2couch with test configuration -print_status "Running mail2couch with test configuration..." -../go/mail2couch -config config-test.json -max-messages 3 - -# Verify results -print_status "Verifying test results..." - -# Check CouchDB databases were created (using correct database names with m2c prefix) -EXPECTED_DBS=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only") - -for db in "${EXPECTED_DBS[@]}"; do - if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then - print_status "✅ Database '$db' created successfully" - else - print_error "❌ Database '$db' was not created" - exit 1 - fi -done - -# Check document counts -for db in "${EXPECTED_DBS[@]}"; do - doc_count=$(curl -s "http://admin:password@localhost:5984/$db" | grep -o '"doc_count":[0-9]*' | cut -d':' -f2) - if [ "$doc_count" -gt 0 ]; then - print_status "✅ Database '$db' contains $doc_count documents" - else - print_warning "⚠️ Database '$db' contains no documents" - fi -done - -# Test sync mode by running again (should show incremental behavior) -print_status "Running mail2couch again to test incremental sync..." -../go/mail2couch -config config-test.json -max-messages 3 - -print_status "🎉 Basic integration tests completed successfully!" - -# Show summary -print_status "Test Summary:" -echo " - IMAP Server: localhost:3143" -echo " - CouchDB: http://localhost:5984" -echo " - Test accounts: testuser1, syncuser, archiveuser" -echo " - Databases created: ${EXPECTED_DBS[*]}" -echo "" -echo "For more comprehensive tests, run:" -echo " - ./test-wildcard-patterns.sh (test folder pattern matching)" -echo " - ./test-incremental-sync.sh (test incremental synchronization)" \ No newline at end of file diff --git a/test/start-test-env.sh b/test/start-test-env.sh deleted file mode 100755 index 2e6a82c..0000000 --- a/test/start-test-env.sh +++ /dev/null @@ -1,67 +0,0 @@ -#!/bin/bash - -# Start test environment for manual testing - -cd "$(dirname "$0")" - -echo "🚀 Starting mail2couch test environment..." - -# Start containers -echo "Starting containers..." -podman-compose -f podman-compose.yml up -d - -# Wait for services -echo "Waiting for services to be ready..." -sleep 10 - -# Check CouchDB -echo "Checking CouchDB..." -timeout=30 -while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do - timeout=$((timeout - 1)) - if [ $timeout -le 0 ]; then - echo "❌ CouchDB failed to start" - exit 1 - fi - sleep 1 -done -echo "✅ CouchDB is ready at http://localhost:5984" - -# Check IMAP -echo "Checking IMAP server..." -timeout=30 -while ! nc -z localhost 3143 > /dev/null 2>&1; do - timeout=$((timeout - 1)) - if [ $timeout -le 0 ]; then - echo "❌ IMAP server failed to start" - exit 1 - fi - sleep 1 -done -echo "✅ IMAP server is ready at localhost:3143" - -# Populate test data -echo "Populating test messages..." -python3 ./populate-greenmail.py - -echo "" -echo "🎉 Test environment is ready!" -echo "" -echo "Services:" -echo " - CouchDB: http://localhost:5984 (admin/password)" -echo " - CouchDB Web UI: http://localhost:5984/_utils" -echo " - IMAP Server: localhost:3143" -echo " - IMAPS Server: localhost:3993" -echo " - SMTP Server: localhost:3025" -echo "" -echo "Test accounts:" -echo " - testuser1:password123" -echo " - testuser2:password456" -echo " - syncuser:syncpass" -echo " - archiveuser:archivepass" -echo "" -echo "To run mail2couch:" -echo " cd ../go && ./mail2couch -config ../test/config-test.json" -echo "" -echo "To stop the environment:" -echo " ./stop-test-env.sh" \ No newline at end of file diff --git a/test/stop-test-env.sh b/test/stop-test-env.sh deleted file mode 100755 index 93c08b5..0000000 --- a/test/stop-test-env.sh +++ /dev/null @@ -1,12 +0,0 @@ -#!/bin/bash - -# Stop test environment - -cd "$(dirname "$0")" - -echo "🛑 Stopping mail2couch test environment..." - -# Stop and remove containers -podman-compose -f podman-compose.yml down -v - -echo "✅ Test environment stopped and cleaned up!" \ No newline at end of file diff --git a/test/test-incremental-sync.sh b/test/test-incremental-sync.sh deleted file mode 100755 index bc35016..0000000 --- a/test/test-incremental-sync.sh +++ /dev/null @@ -1,242 +0,0 @@ -#!/bin/bash - -# Test script to validate incremental sync functionality -# This script tests that mail2couch properly implements incremental synchronization - -set -e - -echo "🔄 Testing Incremental Sync Functionality" -echo "==========================================" - -# Make sure we're in the right directory -cd "$(dirname "$0")/.." - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Function to check if containers are running -check_containers() { - echo "🔍 Checking if test containers are running..." - - if ! podman ps | grep -q "greenmail"; then - echo -e "${RED}❌ GreenMail container not running${NC}" - echo "Please run: cd test && ./start-test-env.sh" - exit 1 - fi - - if ! podman ps | grep -q "couchdb"; then - echo -e "${RED}❌ CouchDB container not running${NC}" - echo "Please run: cd test && ./start-test-env.sh" - exit 1 - fi - - echo -e "${GREEN}✅ Test containers are running${NC}" -} - -# Function to populate initial test data -populate_initial_data() { - echo "📧 Populating initial test data..." - cd test - if python3 populate-greenmail.py; then - echo -e "${GREEN}✅ Initial test data populated${NC}" - else - echo -e "${RED}❌ Failed to populate initial test data${NC}" - exit 1 - fi - cd .. -} - -# Function to build the application -build_app() { - echo "🔨 Building mail2couch..." - cd go - if go build -o mail2couch .; then - echo -e "${GREEN}✅ Build successful${NC}" - else - echo -e "${RED}❌ Build failed${NC}" - exit 1 - fi - cd .. -} - -# Function to run first sync -run_first_sync() { - echo -e "\n${BLUE}Running first sync...${NC}" - cd go - ./mail2couch -config ../test/config-test.json -max-messages 5 - cd .. -} - -# Function to add new messages to test incremental sync -add_new_messages() { - echo -e "\n${YELLOW}Adding new messages for incremental sync test...${NC}" - - # Create a simple Python script to add messages directly to GreenMail - cat > test/add_incremental_messages.py << 'EOF' -#!/usr/bin/env python3 - -import imaplib -import time -from test.populate_greenmail import create_simple_message - -def add_new_messages(): - """Add new messages to test incremental sync""" - accounts = [ - ("testuser1", "password123"), - ("syncuser", "syncpass"), - ("archiveuser", "archivepass") - ] - - for username, password in accounts: - try: - print(f"Adding new messages to {username}...") - imap = imaplib.IMAP4('localhost', 3143) - imap.login(username, password) - imap.select('INBOX') - - # Add 3 new messages with timestamps after the first sync - for i in range(1, 4): - subject = f"Incremental Sync Test Message {i}" - body = f"This message was added after the first sync for incremental testing. Message {i} for {username}." - - msg = create_simple_message(subject, body, f"incremental-test@example.com", f"{username}@example.com") - imap.append('INBOX', None, None, msg.encode('utf-8')) - print(f" Added: {subject}") - time.sleep(0.1) - - imap.logout() - print(f"✅ Added 3 new messages to {username}") - - except Exception as e: - print(f"❌ Error adding messages to {username}: {e}") - -if __name__ == "__main__": - add_new_messages() -EOF - - # Add the parent directory to Python path and run the script - cd test - PYTHONPATH=.. python3 add_incremental_messages.py - cd .. -} - -# Function to run second sync (incremental) -run_incremental_sync() { - echo -e "\n${BLUE}Running incremental sync...${NC}" - cd go - ./mail2couch -config ../test/config-test.json -max-messages 10 - cd .. -} - -# Function to verify incremental sync results -verify_results() { - echo -e "\n${YELLOW}Verifying incremental sync results...${NC}" - - # Check CouchDB for sync metadata documents - echo "Checking for sync metadata in CouchDB databases..." - - # List of expected databases based on test config (with m2c prefix) - databases=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only") - - for db in "${databases[@]}"; do - echo " Checking database: $db" - - # Check if database exists - if curl -s -f "http://admin:password@localhost:5984/$db" > /dev/null; then - echo " ✅ Database exists" - - # Look for sync metadata documents - metadata_docs=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"" | grep -o '"total_rows":[0-9]*' | cut -d: -f2 || echo "0") - - if [ "$metadata_docs" -gt 0 ]; then - echo " ✅ Found sync metadata documents: $metadata_docs" - - # Get a sample sync metadata document - sample_doc=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"&include_docs=true&limit=1") - echo " Sample sync metadata:" - echo "$sample_doc" | python3 -m json.tool | grep -E "(lastSyncTime|lastMessageUID|messageCount)" | head -3 - else - echo " ⚠️ No sync metadata documents found" - fi - else - echo " ❌ Database does not exist" - fi - done -} - -# Main test execution -main() { - echo "Starting incremental sync tests..." - - # Pre-test setup - check_containers - build_app - - # Clean up any existing data - echo "🧹 Cleaning up existing test data..." - curl -s -X DELETE "http://admin:password@localhost:5984/m2c_wildcard_all_folders_test" > /dev/null || true - curl -s -X DELETE "http://admin:password@localhost:5984/m2c_work_pattern_test" > /dev/null || true - curl -s -X DELETE "http://admin:password@localhost:5984/m2c_specific_folders_only" > /dev/null || true - - # Step 1: Populate initial test data - populate_initial_data - - # Wait for data to settle - echo "⏳ Waiting for initial data to settle..." - sleep 5 - - # Step 2: Run first sync to establish baseline - echo -e "\n${YELLOW}=== STEP 1: First Sync (Baseline) ===${NC}" - run_first_sync - - # Wait between syncs - echo "⏳ Waiting between syncs..." - sleep 3 - - # Step 3: Add new messages for incremental sync test - echo -e "\n${YELLOW}=== STEP 2: Add New Messages ===${NC}" - add_new_messages - - # Wait for new messages to be ready - echo "⏳ Waiting for new messages to be ready..." - sleep 2 - - # Step 4: Run incremental sync - echo -e "\n${YELLOW}=== STEP 3: Incremental Sync ===${NC}" - run_incremental_sync - - # Step 5: Verify results - echo -e "\n${YELLOW}=== STEP 4: Verification ===${NC}" - verify_results - - echo -e "\n${GREEN}🎉 Incremental sync test completed!${NC}" - echo "" - echo "Key features tested:" - echo " ✅ Sync metadata storage and retrieval" - echo " ✅ IMAP SEARCH with SINCE for efficient incremental fetching" - echo " ✅ Last sync timestamp tracking per mailbox" - echo " ✅ Proper handling of first sync vs incremental sync" - echo "" - echo "To verify results manually:" - echo " - Check CouchDB: http://localhost:5984/_utils" - echo " - Look for 'sync_metadata_*' documents in each database" - echo " - Verify incremental messages were added after baseline sync" -} - -# Cleanup function -cleanup() { - echo "🧹 Cleaning up test artifacts..." - rm -f test/add_incremental_messages.py -} - -# Set trap to cleanup on exit -trap cleanup EXIT - -# Run main function if executed directly -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - main "$@" -fi \ No newline at end of file diff --git a/test/test-wildcard-patterns.sh b/test/test-wildcard-patterns.sh deleted file mode 100755 index e277ea3..0000000 --- a/test/test-wildcard-patterns.sh +++ /dev/null @@ -1,137 +0,0 @@ -#!/bin/bash - -# Test script to validate wildcard folder pattern functionality -# This script tests the various wildcard patterns against the test environment - -set -e - -echo "🧪 Testing Wildcard Folder Pattern Functionality" -echo "================================================" - -# Make sure we're in the right directory -cd "$(dirname "$0")/.." - -# Colors for output -RED='\033[0;31m' -GREEN='\033[0;32m' -YELLOW='\033[1;33m' -BLUE='\033[0;34m' -NC='\033[0m' # No Color - -# Function to run mail2couch with a specific config and capture output -run_test() { - local test_name="$1" - local config_file="$2" - local max_messages="$3" - - echo -e "\n${BLUE}Testing: $test_name${NC}" - echo "Config: $config_file" - echo "Max messages: $max_messages" - echo "----------------------------------------" - - # Run mail2couch and capture output - cd go - if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then - echo -e "${GREEN}✅ Test completed successfully${NC}" - else - echo -e "${RED}❌ Test failed${NC}" - return 1 - fi - cd .. -} - -# Function to check if containers are running -check_containers() { - echo "🔍 Checking if test containers are running..." - - if ! podman ps | grep -q "greenmail"; then - echo -e "${RED}❌ GreenMail container not running${NC}" - echo "Please run: cd test && ./start-test-env.sh" - exit 1 - fi - - if ! podman ps | grep -q "couchdb"; then - echo -e "${RED}❌ CouchDB container not running${NC}" - echo "Please run: cd test && ./start-test-env.sh" - exit 1 - fi - - echo -e "${GREEN}✅ Test containers are running${NC}" -} - -# Function to populate test data -populate_test_data() { - echo "📧 Populating test data..." - cd test - if python3 populate-greenmail.py; then - echo -e "${GREEN}✅ Test data populated successfully${NC}" - else - echo -e "${RED}❌ Failed to populate test data${NC}" - exit 1 - fi - cd .. -} - -# Function to build the application -build_app() { - echo "🔨 Building mail2couch..." - cd go - if go build -o mail2couch .; then - echo -e "${GREEN}✅ Build successful${NC}" - else - echo -e "${RED}❌ Build failed${NC}" - exit 1 - fi - cd .. -} - -# Main test execution -main() { - echo "Starting wildcard pattern tests..." - - # Pre-test checks - check_containers - build_app - populate_test_data - - # Wait a moment for test data to be fully ready - echo "⏳ Waiting for test data to settle..." - sleep 3 - - # Test 1: Wildcard all folders (*) - echo -e "\n${YELLOW}Test 1: Wildcard All Folders Pattern (*)${NC}" - echo "Expected: Should process all folders except Drafts and Trash" - run_test "Wildcard All Folders" "config-test.json" 3 - - # Test 2: Work pattern (Work*) - echo -e "\n${YELLOW}Test 2: Work Pattern (Work*)${NC}" - echo "Expected: Should process Work/Projects, Work/Archive but not Work/Temp (excluded by *Temp*)" - run_test "Work Pattern" "config-test.json" 3 - - # Test 3: Specific folders only - echo -e "\n${YELLOW}Test 3: Specific Folders Only${NC}" - echo "Expected: Should only process INBOX, Sent, and Personal folders" - run_test "Specific Folders" "config-test.json" 3 - - # Test 4: Advanced wildcard examples - echo -e "\n${YELLOW}Test 4: Advanced Wildcard Examples${NC}" - echo "Expected: Various complex patterns should work correctly" - run_test "Advanced Patterns" "config-wildcard-examples.json" 2 - - echo -e "\n${GREEN}🎉 All wildcard pattern tests completed!${NC}" - echo "" - echo "To verify results, check the CouchDB databases:" - echo " http://localhost:5984/_utils" - echo "" - echo "Expected databases should be created for each account:" - echo " - wildcard_all_folders_test" - echo " - work_pattern_test" - echo " - specific_folders_only" - echo "" - echo "Each database should contain documents with 'mailbox' field showing origin folder." -} - -# Run main function if executed directly -if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then - main "$@" -fi \ No newline at end of file