Compare commits

..

No commits in common. "c2ad55eaafb3392b3c52040e927981d18c67af9f" and "d0caff800a0eb70a8dd365a0179b73ab52d351cd" have entirely different histories.

33 changed files with 2 additions and 3653 deletions

161
CLAUDE.md
View file

@ -1,161 +0,0 @@
# CLAUDE.md
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
## Project Overview
mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations:
- **Go implementation**: Located in `/go/` directory (currently the active implementation)
- **Rust implementation**: Planned but not yet implemented
## Development Commands
### Go Implementation (Primary)
```bash
# Build the application
cd go && go build -o mail2couch .
# Run the application with automatic config discovery
cd go && ./mail2couch
# Run with specific config file
cd go && ./mail2couch -config /path/to/config.json
# Run with message limit (useful for large mailboxes)
cd go && ./mail2couch -max-messages 100
# Run with both config and message limit
cd go && ./mail2couch -config /path/to/config.json -max-messages 50
# Run linting/static analysis
cd go && go vet ./...
# Run integration tests with Podman containers
cd test && ./run-tests.sh
# Run specialized tests
cd test && ./test-wildcard-patterns.sh
cd test && ./test-incremental-sync.sh
# Run unit tests (none currently implemented)
cd go && go test ./...
# Check dependencies
cd go && go mod tidy
```
## Architecture
### Core Components
1. **Configuration (`config/`)**: JSON-based configuration system
- Supports multiple mail sources with filtering options
- CouchDB connection settings
- Each source can have folder and message filters
2. **Mail Handling (`mail/`)**: IMAP client implementation
- Uses `github.com/emersion/go-imap/v2` for IMAP operations
- Supports TLS connections
- Currently only lists mailboxes (backup functionality not yet implemented)
3. **CouchDB Integration (`couch/`)**: Database operations
- Uses `github.com/go-kivik/kivik/v4` as CouchDB driver
- Handles database creation and document management
- Defines `MailDocument` structure for email storage
### Configuration Structure
The application uses `config.json` for configuration with the following structure:
- `couchDb`: Database connection settings (URL, credentials)
- `mailSources`: Array of mail sources with individual settings:
- Protocol support (currently only IMAP)
- Connection details (host, port, credentials)
- `mode`: Either "sync" or "archive" (defaults to "archive" if not specified)
- **sync**: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB)
- **archive**: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents)
- Filtering options for folders and messages with wildcard support
- Enable/disable per source
### Configuration File Discovery
The application automatically searches for configuration files in the following order:
1. Path specified by `-config` command line flag
2. `./config.json` (current working directory)
3. `./config/config.json` (config subdirectory)
4. `~/.config/mail2couch/config.json` (user XDG config directory)
5. `~/.mail2couch.json` (user home directory)
This design ensures the same `config.json` format will work for both Go and Rust implementations.
### Current Implementation Status
- ✅ Configuration loading with automatic file discovery
- ✅ Command line flag support for config file path
- ✅ Per-account CouchDB database creation and management
- ✅ IMAP connection and mailbox listing
- ✅ Build error fixes
- ✅ Email message retrieval framework (with placeholder data)
- ✅ Email storage to CouchDB framework with native attachments
- ✅ Folder filtering logic with wildcard support (`*`, `?`, `[abc]` patterns)
- ✅ Date filtering support
- ✅ Keyword filtering support (subject, sender, recipient keywords)
- ✅ Duplicate detection and prevention
- ✅ Sync vs Archive mode implementation
- ✅ CouchDB attachment storage for email attachments
- ✅ Real IMAP message parsing with go-message library
- ✅ Full message body and attachment handling with MIME multipart support
- ✅ Command line argument support (--max-messages flag)
- ✅ Per-account CouchDB databases for better organization
- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking
- ❌ Rust implementation
### Key Dependencies
- `github.com/emersion/go-imap/v2`: IMAP client library
- `github.com/go-kivik/kivik/v4`: CouchDB client library
### Incremental Sync Implementation
The application implements intelligent incremental synchronization to avoid re-processing messages:
- **Sync Metadata Storage**: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed
- **IMAP SEARCH Integration**: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages
- **Per-Mailbox Tracking**: Sync state is tracked independently for each mailbox in each account
- **Fallback Behavior**: Gracefully falls back to fetching recent messages if IMAP SEARCH fails
- **First Sync Handling**: Initial sync can use config `since` date or perform full sync
Sync metadata documents are stored in CouchDB with ID format: `sync_metadata_{mailbox}` and include:
- `lastSyncTime`: When this mailbox was last successfully synced
- `lastMessageUID`: Highest UID processed in the last sync
- `messageCount`: Number of messages processed in the last sync
### Development Notes
- The main entry point is `main.go` which orchestrates the configuration loading, CouchDB setup, and mail source processing
- Each mail source gets its own CouchDB database named using `GenerateAccountDBName()` function with `m2c_` prefix
- Each mail source is processed sequentially with proper error handling
- The application uses real IMAP message parsing with go-message library for full email processing
- Message filtering by folder (wildcard patterns), date (since), and keywords is implemented
- Duplicate detection prevents re-storing existing messages
- Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account
- Email attachments are stored as native CouchDB attachments linked to the email document
- Comprehensive test environment with Podman containers and automated test scripts
- The application uses automatic config file discovery as documented above
### Next Steps
The following enhancements could further improve the implementation:
1. **Error Recovery**: Add retry logic for network failures and partial sync recovery
2. **Performance Optimization**: Add batch operations for better CouchDB insertion performance
3. **Unit Testing**: Add comprehensive unit tests for all major components
4. **Advanced Filtering**: Add support for more complex filter expressions and regex patterns
5. **Monitoring**: Add metrics and logging for production deployment
6. **Configuration Validation**: Enhanced validation for configuration files
7. **Multi-threading**: Parallel processing of multiple mailboxes or accounts
## Development Guidelines
### Code Quality and Standards
- All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly

View file

@ -1,102 +0,0 @@
# Folder Pattern Matching in mail2couch
mail2couch supports powerful wildcard patterns for selecting which folders to process. This allows flexible configuration for different mail backup scenarios.
## Pattern Syntax
The folder filtering uses Go's `filepath.Match` syntax, which supports:
- `*` matches any sequence of characters (including none)
- `?` matches any single character
- `[abc]` matches any character within the brackets
- `[a-z]` matches any character in the range
- `\` escapes special characters
## Special Cases
- `"*"` in the include list means **ALL available folders** will be processed
- Empty include list with exclude patterns will process all folders except excluded ones
- Exact string matching is supported for backwards compatibility
## Examples
### Include All Folders
```json
{
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash", "Spam"]
}
}
```
This processes all folders except Drafts, Trash, and Spam.
### Work-Related Folders Only
```json
{
"folderFilter": {
"include": ["Work*", "Projects*", "INBOX"],
"exclude": ["*Temp*", "*Draft*"]
}
}
```
This includes folders starting with "Work" or "Projects", plus INBOX, but excludes any folder containing "Temp" or "Draft".
### Archive Patterns
```json
{
"folderFilter": {
"include": ["Archive*", "*Important*", "INBOX"],
"exclude": ["*Temp"]
}
}
```
This includes folders starting with "Archive", any folder containing "Important", and INBOX, excluding temporary folders.
### Specific Folders Only
```json
{
"folderFilter": {
"include": ["INBOX", "Sent", "Important"],
"exclude": []
}
}
```
This processes only the exact folders: INBOX, Sent, and Important.
### Subfolder Patterns
```json
{
"folderFilter": {
"include": ["Work/*", "Personal/*"],
"exclude": ["*/Drafts"]
}
}
```
This includes all subfolders under Work and Personal, but excludes any Drafts subfolder.
## Folder Hierarchy
Different IMAP servers use different separators for folder hierarchies:
- Most servers use `/` (e.g., `Work/Projects`, `Archive/2024`)
- Some use `.` (e.g., `Work.Projects`, `Archive.2024`)
The patterns work with whatever separator your IMAP server uses.
## Common Use Cases
1. **Corporate Email**: `["*"]` with exclude `["Drafts", "Trash", "Spam"]` for complete backup
2. **Selective Backup**: `["INBOX", "Sent", "Important"]` for essential folders only
3. **Project-based**: `["Project*", "Client*"]` to backup work-related folders
4. **Archive Mode**: `["Archive*", "*Important*"]` for long-term storage
5. **Sync Mode**: `["INBOX"]` for real-time synchronization
## Message Origin Tracking
All messages stored in CouchDB include a `mailbox` field that records the original folder name. This ensures you can always identify which folder a message came from, regardless of how it was selected by the folder filter.
## Performance Considerations
- Using `"*"` processes all folders, which may be slow for accounts with many folders
- Specific folder names are faster than wildcard patterns
- Consider using exclude patterns to filter out large, unimportant folders like Trash or Spam

388
README.md
View file

@ -1,389 +1,5 @@
# mail2couch
A powerful email backup utility that synchronizes mail from IMAP accounts to CouchDB databases with intelligent incremental sync, comprehensive filtering, and native attachment support.
A utility to back up mail from various sources to couchdb
## Features
### Core Functionality
- **IMAP Email Backup**: Connect to any IMAP server (Gmail, Outlook, self-hosted)
- **CouchDB Storage**: Store emails as JSON documents with native CouchDB attachments
- **Incremental Sync**: Efficiently sync only new messages using IMAP SEARCH with timestamp tracking
- **Per-Account Databases**: Each mail source gets its own CouchDB database for better organization
- **Duplicate Prevention**: Automatic detection and prevention of duplicate message storage
### Sync Modes
- **Archive Mode**: Preserve all messages ever seen, even if deleted from mail server (default)
- **Sync Mode**: Maintain 1-to-1 relationship with mail server (removes deleted messages from CouchDB)
### Advanced Filtering
- **Wildcard Folder Patterns**: Use `*`, `?`, `[abc]` patterns for flexible folder selection
- **Keyword Filtering**: Filter messages by keywords in subjects, senders, or recipients
- **Date Filtering**: Process only messages since a specific date
- **Include/Exclude Logic**: Combine multiple filter types for precise control
### Message Processing
- **Full MIME Support**: Parse multipart messages, HTML/plain text, and embedded content
- **Native Attachments**: Store email attachments as CouchDB native attachments with compression
- **Complete Headers**: Preserve all email headers and metadata
- **UTF-8 Support**: Handle international characters and special content
### Operational Features
- **Automatic Config Discovery**: Finds configuration files in standard locations
- **Command Line Control**: Override settings with `--max-messages` and `--config` flags
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
- **Error Resilience**: Graceful handling of network issues and server problems
## Quick Start
### Installation
1. **Install dependencies**:
```bash
# Go 1.21+ required
go version
```
2. **Clone and build**:
```bash
git clone <repository-url>
cd mail2couch/go
go build -o mail2couch .
```
### Basic Usage
1. **Create configuration file** (`config.json`):
```json
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "your-email@gmail.com",
"password": "your-app-password",
"mode": "archive",
"folderFilter": {
"include": ["*"],
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam"]
}
}
]
}
```
2. **Run mail2couch**:
```bash
./mail2couch
```
The application will:
- Create a CouchDB database named `m2c_personal_gmail`
- Sync all folders except Trash and Spam
- Store messages with native attachments
- Track sync state for efficient incremental updates
## Configuration
### Configuration File Discovery
mail2couch automatically searches for configuration files in this order:
1. Path specified by `--config` flag
2. `./config.json` (current directory)
3. `./config/config.json` (config subdirectory)
4. `~/.config/mail2couch/config.json` (user config directory)
5. `~/.mail2couch.json` (user home directory)
### Command Line Options
```bash
./mail2couch [options]
Options:
--config PATH Specify configuration file path
--max-messages N Limit messages processed per mailbox per run (0 = unlimited)
```
### Folder Pattern Examples
| Pattern | Description | Matches |
|---------|-------------|---------|
| `"*"` | All folders | `INBOX`, `Sent`, `Work/Projects`, etc. |
| `"INBOX"` | Exact match | `INBOX` only |
| `"Work*"` | Prefix match | `Work`, `Work/Projects`, `WorkStuff` |
| `"*/Archive"` | Suffix match | `Personal/Archive`, `Work/Archive` |
| `"Work/*"` | Subfolder match | `Work/Projects`, `Work/Clients` |
### Keyword Filtering Examples
```json
{
"messageFilter": {
"subjectKeywords": ["urgent", "meeting", "invoice"],
"senderKeywords": ["@company.com", "noreply@"],
"recipientKeywords": ["team@", "support@"]
}
}
```
## Advanced Configuration Examples
See the [example configurations](#example-configurations) section below for detailed configuration scenarios.
## Testing
A comprehensive test environment is included with Podman containers:
```bash
cd test
# Quick automated testing (recommended)
./run-tests.sh # Complete integration test with automatic cleanup
# Specialized feature testing
./test-wildcard-patterns.sh # Test folder pattern matching
./test-incremental-sync.sh # Test incremental synchronization
# Manual testing environment
./start-test-env.sh # Start persistent test environment
# ... manual testing with various configurations ...
./stop-test-env.sh # Clean up when done
```
## Architecture
### Database Structure
- **Per-Account Databases**: Each mail source creates its own CouchDB database with `m2c_` prefix
- **Message Documents**: Each email becomes a CouchDB document with metadata
- **Native Attachments**: Email attachments stored as CouchDB attachments (compressed)
- **Sync Metadata**: Tracks incremental sync state per mailbox
### Document Structure
```json
{
"_id": "INBOX_12345",
"sourceUid": "12345",
"mailbox": "INBOX",
"from": ["sender@example.com"],
"to": ["recipient@example.com"],
"subject": "Sample Email",
"date": "2024-01-15T10:30:00Z",
"body": "Email content...",
"headers": {"Content-Type": ["text/plain"]},
"storedAt": "2024-01-15T10:35:00Z",
"docType": "mail",
"hasAttachments": true,
"_attachments": {
"document.pdf": {
"content_type": "application/pdf",
"length": 54321
}
}
}
```
## Example Configurations
### Simple Configuration
Basic setup for a single Gmail account:
```json
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "your-email@gmail.com",
"password": "your-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": []
},
"messageFilter": {
"since": "2024-01-01"
}
}
]
}
```
### Advanced Multi-Account Configuration
Complex setup with multiple accounts, filtering, and different sync modes:
```json
{
"couchDb": {
"url": "https://your-couchdb.example.com:5984",
"user": "backup_user",
"password": "secure_password"
},
"mailSources": [
{
"name": "Work Email",
"enabled": true,
"protocol": "imap",
"host": "outlook.office365.com",
"port": 993,
"user": "you@company.com",
"password": "app-password",
"mode": "sync",
"folderFilter": {
"include": ["*"],
"exclude": ["Deleted Items", "Junk Email", "Drafts"]
},
"messageFilter": {
"since": "2023-01-01",
"subjectKeywords": ["project", "meeting", "urgent"],
"senderKeywords": ["@company.com", "@client.com"]
}
},
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "personal@gmail.com",
"password": "gmail-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Important", "Work/*", "Personal/*"],
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"]
},
"messageFilter": {
"recipientKeywords": ["family@", "personal@"]
}
},
{
"name": "Self-Hosted Mail",
"enabled": true,
"protocol": "imap",
"host": "mail.yourdomain.com",
"port": 143,
"user": "admin@yourdomain.com",
"password": "mail-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Archive/*", "Projects/*"],
"exclude": ["*/Drafts", "Trash"]
},
"messageFilter": {
"since": "2023-06-01",
"subjectKeywords": ["invoice", "receipt", "statement"]
}
},
{
"name": "Legacy Account",
"enabled": false,
"protocol": "imap",
"host": "legacy.mailserver.com",
"port": 993,
"user": "old@account.com",
"password": "legacy-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX"],
"exclude": []
},
"messageFilter": {}
}
]
}
```
### Configuration Options Reference
#### CouchDB Configuration
- `url`: CouchDB server URL with protocol and port
- `user`: CouchDB username with database access
- `password`: CouchDB password
#### Mail Source Configuration
- `name`: Descriptive name (used for database naming)
- `enabled`: Boolean to enable/disable this source
- `protocol`: Only `"imap"` currently supported
- `host`: IMAP server hostname
- `port`: IMAP port (993 for TLS, 143 for plain, 3143 for testing)
- `user`: Email account username
- `password`: Email account password (use app passwords for Gmail/Outlook)
- `mode`: `"sync"` (mirror server) or `"archive"` (preserve all messages)
#### Folder Filter Configuration
- `include`: Array of folder patterns to process (empty = all folders)
- `exclude`: Array of folder patterns to skip
#### Message Filter Configuration
- `since`: Date string (YYYY-MM-DD) to process messages from
- `subjectKeywords`: Array of keywords that must appear in subject line
- `senderKeywords`: Array of keywords that must appear in sender addresses
- `recipientKeywords`: Array of keywords that must appear in recipient addresses
## Production Deployment
### Security Considerations
- Use app passwords instead of account passwords
- Store configuration files with restricted permissions (600)
- Use HTTPS for CouchDB connections in production
- Consider encrypting sensitive configuration data
### Monitoring and Maintenance
- Review sync metadata documents for sync health
- Monitor CouchDB database sizes and compaction
- Set up log rotation for application output
- Schedule regular backups of CouchDB databases
### Performance Tuning
- Use `--max-messages` to limit processing load
- Run during off-peak hours for large initial syncs
- Monitor IMAP server rate limits and connection limits
- Consider running multiple instances for different accounts
## Troubleshooting
### Common Issues
**Connection Errors**:
- Verify IMAP server settings and credentials
- Check firewall and network connectivity
- Ensure correct ports (993 for TLS, 143 for plain)
**Authentication Failures**:
- Use app passwords for Gmail, Outlook, and other providers
- Enable "Less Secure Apps" if required by provider
- Verify account permissions and 2FA settings
**Sync Issues**:
- Check CouchDB connectivity and permissions
- Review sync metadata documents for error states
- Verify folder names and patterns match server structure
**Performance Problems**:
- Use date filtering (`since`) for large mailboxes
- Implement `--max-messages` limits for initial syncs
- Monitor server-side rate limiting
For detailed troubleshooting, see the [test environment documentation](test/README.md).
## Contributing
This project welcomes contributions! Please see [CLAUDE.md](CLAUDE.md) for development setup and architecture details.
## License
[License information to be added]
At least two implementations will be available, on in Rust and one in Go.

47
TODO.md
View file

@ -1,47 +0,0 @@
# mail2couch TODO and Feature Requests
## Planned Features
### Keyword Filtering for Messages
Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration.
**Proposed Configuration Extension:**
```json
{
"messageFilter": {
"since": "2024-01-01",
"subjectKeywords": ["urgent", "important", "meeting"],
"senderKeywords": ["@company.com", "notifications"],
"recipientKeywords": ["team@company.com", "all@"]
}
}
```
**Implementation Details:**
- `subjectKeywords`: Array of keywords to match in email subject lines
- `senderKeywords`: Array of keywords to match in sender email addresses or names
- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names
- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns
- Case-insensitive matching by default
- Support for simple wildcards or regex patterns
**Use Cases:**
1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords
2. **Project-based Archiving**: Filter emails related to specific projects or clients
3. **Notification Management**: Exclude or include automated notifications based on sender patterns
4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders
**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails.
## Other Planned Improvements
1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content
2. **Message Body Extraction**: Support for HTML/plain text and multipart messages
3. **Attachment Handling**: Optional support for email attachments
4. **Batch Operations**: Improve CouchDB insertion performance
5. **Error Recovery**: Retry logic and partial sync recovery
6. **Testing**: Comprehensive unit test coverage

View file

@ -1,78 +0,0 @@
{
"couchDb": {
"url": "https://your-couchdb.example.com:5984",
"user": "backup_user",
"password": "secure_password"
},
"mailSources": [
{
"name": "Work Email",
"enabled": true,
"protocol": "imap",
"host": "outlook.office365.com",
"port": 993,
"user": "you@company.com",
"password": "app-password",
"mode": "sync",
"folderFilter": {
"include": ["*"],
"exclude": ["Deleted Items", "Junk Email", "Drafts"]
},
"messageFilter": {
"since": "2023-01-01",
"subjectKeywords": ["project", "meeting", "urgent"],
"senderKeywords": ["@company.com", "@client.com"]
}
},
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "personal@gmail.com",
"password": "gmail-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Important", "Work/*", "Personal/*"],
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"]
},
"messageFilter": {
"recipientKeywords": ["family@", "personal@"]
}
},
{
"name": "Self-Hosted Mail",
"enabled": true,
"protocol": "imap",
"host": "mail.yourdomain.com",
"port": 143,
"user": "admin@yourdomain.com",
"password": "mail-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Archive/*", "Projects/*"],
"exclude": ["*/Drafts", "Trash"]
},
"messageFilter": {
"since": "2023-06-01",
"subjectKeywords": ["invoice", "receipt", "statement"]
}
},
{
"name": "Legacy Account",
"enabled": false,
"protocol": "imap",
"host": "legacy.mailserver.com",
"port": 993,
"user": "old@account.com",
"password": "legacy-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX"],
"exclude": []
},
"messageFilter": {}
}
]
}

View file

@ -1,90 +0,0 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Gmail Account",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "your-email@gmail.com",
"password": "your-16-character-app-password",
"mode": "archive",
"folderFilter": {
"include": ["*"],
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "[Gmail]/Drafts"]
},
"messageFilter": {
"since": "2024-01-01"
}
},
{
"name": "Outlook 365",
"enabled": true,
"protocol": "imap",
"host": "outlook.office365.com",
"port": 993,
"user": "you@outlook.com",
"password": "your-app-password",
"mode": "sync",
"folderFilter": {
"include": ["INBOX", "Sent Items", "Archive"],
"exclude": ["Deleted Items", "Junk Email"]
},
"messageFilter": {
"since": "2023-06-01"
}
},
{
"name": "Yahoo Mail",
"enabled": false,
"protocol": "imap",
"host": "imap.mail.yahoo.com",
"port": 993,
"user": "your-email@yahoo.com",
"password": "your-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": ["Trash", "Spam"]
},
"messageFilter": {}
},
{
"name": "iCloud Mail",
"enabled": false,
"protocol": "imap",
"host": "imap.mail.me.com",
"port": 993,
"user": "your-email@icloud.com",
"password": "your-app-specific-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent Messages"],
"exclude": ["Deleted Messages", "Junk"]
},
"messageFilter": {}
},
{
"name": "Custom IMAP Server",
"enabled": false,
"protocol": "imap",
"host": "mail.example.com",
"port": 993,
"user": "username@example.com",
"password": "password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": ["Trash"]
},
"messageFilter": {
"since": "2024-01-01"
}
}
]
}

View file

@ -1,26 +0,0 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "your-email@gmail.com",
"password": "your-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": []
},
"messageFilter": {
"since": "2024-01-01"
}
}
]
}

View file

@ -1,41 +0,0 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Personal Gmail",
"enabled": true,
"protocol": "imap",
"host": "imap.gmail.com",
"port": 993,
"user": "your-email@gmail.com",
"password": "your-app-password",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": ["Spam", "Trash"]
},
"messageFilter": {
"since": "2024-01-01"
}
},
{
"name": "Work Account",
"enabled": true,
"protocol": "imap",
"host": "imap.work.com",
"port": 993,
"user": "user@work.com",
"password": "password",
"mode": "sync",
"folderFilter": {
"include": [],
"exclude": []
},
"messageFilter": {}
}
]
}

View file

@ -1,153 +0,0 @@
package config
import (
"encoding/json"
"flag"
"fmt"
"os"
"path/filepath"
)
type Config struct {
CouchDb CouchDbConfig `json:"couchDb"`
MailSources []MailSource `json:"mailSources"`
}
type CouchDbConfig struct {
URL string `json:"url"`
User string `json:"user"`
Password string `json:"password"`
}
type MailSource struct {
Name string `json:"name"`
Enabled bool `json:"enabled"`
Protocol string `json:"protocol"`
Host string `json:"host"`
Port int `json:"port"`
User string `json:"user"`
Password string `json:"password"`
Mode string `json:"mode"` // "sync" or "archive"
FolderFilter FolderFilter `json:"folderFilter"`
MessageFilter MessageFilter `json:"messageFilter"`
}
type FolderFilter struct {
Include []string `json:"include"`
Exclude []string `json:"exclude"`
}
type MessageFilter struct {
Since string `json:"since,omitempty"`
SubjectKeywords []string `json:"subjectKeywords,omitempty"` // Filter by keywords in subject
SenderKeywords []string `json:"senderKeywords,omitempty"` // Filter by keywords in sender addresses
RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses
}
func LoadConfig(path string) (*Config, error) {
configFile, err := os.Open(path)
if err != nil {
return nil, err
}
defer configFile.Close()
var config Config
jsonParser := json.NewDecoder(configFile)
if err = jsonParser.Decode(&config); err != nil {
return nil, err
}
// Validate and set defaults for mail sources
for i := range config.MailSources {
source := &config.MailSources[i]
if source.Mode == "" {
source.Mode = "archive" // Default to archive mode
}
if source.Mode != "sync" && source.Mode != "archive" {
return nil, fmt.Errorf("invalid mode '%s' for mail source '%s': must be 'sync' or 'archive'", source.Mode, source.Name)
}
}
return &config, nil
}
// IsSyncMode returns true if the mail source is in sync mode
func (ms *MailSource) IsSyncMode() bool {
return ms.Mode == "sync"
}
// IsArchiveMode returns true if the mail source is in archive mode
func (ms *MailSource) IsArchiveMode() bool {
return ms.Mode == "archive" || ms.Mode == "" // Default to archive
}
// CommandLineArgs holds parsed command line arguments
type CommandLineArgs struct {
ConfigPath string
MaxMessages int
}
// ParseCommandLine parses command line arguments
func ParseCommandLine() *CommandLineArgs {
configFlag := flag.String("config", "", "Path to configuration file")
maxMessagesFlag := flag.Int("max-messages", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)")
flag.Parse()
return &CommandLineArgs{
ConfigPath: *configFlag,
MaxMessages: *maxMessagesFlag,
}
}
// FindConfigFile searches for config.json in the following order:
// 1. Path specified by -config flag
// 2. ./config.json (current directory)
// 3. ~/.config/mail2couch/config.json (user config directory)
// 4. ~/.mail2couch.json (user home directory)
func FindConfigFile(args *CommandLineArgs) (string, error) {
if args.ConfigPath != "" {
if _, err := os.Stat(args.ConfigPath); err == nil {
return args.ConfigPath, nil
}
return "", fmt.Errorf("specified config file not found: %s", args.ConfigPath)
}
// List of possible config file locations in order of preference
candidates := []string{
"config.json", // Current directory
"config/config.json", // Config subdirectory
}
// Add user directory paths
if homeDir, err := os.UserHomeDir(); err == nil {
candidates = append(candidates,
filepath.Join(homeDir, ".config", "mail2couch", "config.json"),
filepath.Join(homeDir, ".mail2couch.json"),
)
}
// Try each candidate location
for _, candidate := range candidates {
if _, err := os.Stat(candidate); err == nil {
return candidate, nil
}
}
return "", fmt.Errorf("no configuration file found. Searched locations: %v", candidates)
}
// LoadConfigWithDiscovery loads configuration using automatic file discovery
func LoadConfigWithDiscovery(args *CommandLineArgs) (*Config, error) {
configPath, err := FindConfigFile(args)
if err != nil {
return nil, err
}
fmt.Printf("Using configuration file: %s\n", configPath)
if args.MaxMessages > 0 {
fmt.Printf("Maximum messages per mailbox: %d\n", args.MaxMessages)
} else {
fmt.Printf("Maximum messages per mailbox: unlimited\n")
}
return LoadConfig(configPath)
}

View file

@ -1,374 +0,0 @@
package couch
import (
"context"
"fmt"
"io"
"net/url"
"regexp"
"strings"
"time"
"github.com/go-kivik/kivik/v4"
_ "github.com/go-kivik/kivik/v4/couchdb" // The CouchDB driver
"mail2couch/config"
"mail2couch/mail"
)
// Client wraps the Kivik client
type Client struct {
*kivik.Client
}
// MailDocument represents an email message stored in CouchDB
type MailDocument struct {
ID string `json:"_id,omitempty"`
Rev string `json:"_rev,omitempty"`
Attachments map[string]AttachmentStub `json:"_attachments,omitempty"` // CouchDB attachments
SourceUID string `json:"sourceUid"` // Unique ID from the mail source (e.g., IMAP UID)
Mailbox string `json:"mailbox"` // Source mailbox name
From []string `json:"from"`
To []string `json:"to"`
Subject string `json:"subject"`
Date time.Time `json:"date"`
Body string `json:"body"`
Headers map[string][]string `json:"headers"`
StoredAt time.Time `json:"storedAt"` // When the document was stored
DocType string `json:"docType"` // Always "mail"
HasAttachments bool `json:"hasAttachments"` // Indicates if message has attachments
}
// AttachmentStub represents metadata for a CouchDB attachment
type AttachmentStub struct {
ContentType string `json:"content_type"`
Length int64 `json:"length,omitempty"`
Stub bool `json:"stub,omitempty"`
}
// SyncMetadata represents sync state information stored in CouchDB
type SyncMetadata struct {
ID string `json:"_id,omitempty"`
Rev string `json:"_rev,omitempty"`
DocType string `json:"docType"` // Always "sync_metadata"
Mailbox string `json:"mailbox"` // Mailbox name
LastSyncTime time.Time `json:"lastSyncTime"` // When this mailbox was last synced
LastMessageUID uint32 `json:"lastMessageUID"` // Highest UID processed in last sync
MessageCount int `json:"messageCount"` // Number of messages processed in last sync
UpdatedAt time.Time `json:"updatedAt"` // When this metadata was last updated
}
// NewClient creates a new CouchDB client from the configuration
func NewClient(cfg *config.CouchDbConfig) (*Client, error) {
parsedURL, err := url.Parse(cfg.URL)
if err != nil {
return nil, fmt.Errorf("invalid couchdb url: %w", err)
}
parsedURL.User = url.UserPassword(cfg.User, cfg.Password)
dsn := parsedURL.String()
client, err := kivik.New("couch", dsn)
if err != nil {
return nil, err
}
return &Client{client}, nil
}
// EnsureDB ensures that the configured database exists.
func (c *Client) EnsureDB(ctx context.Context, dbName string) error {
exists, err := c.DBExists(ctx, dbName)
if err != nil {
return err
}
if !exists {
return c.CreateDB(ctx, dbName)
}
return nil
}
// GenerateAccountDBName creates a CouchDB-compatible database name from account info
func GenerateAccountDBName(accountName, userEmail string) string {
// Use account name if available, otherwise fall back to email
name := accountName
if name == "" {
name = userEmail
}
// Convert to lowercase and replace invalid characters with underscores
name = strings.ToLower(name)
// CouchDB database names must match: ^[a-z][a-z0-9_$()+/-]*$
validName := regexp.MustCompile(`[^a-z0-9_$()+/-]`).ReplaceAllString(name, "_")
// Ensure it starts with a letter and add m2c prefix
if len(validName) > 0 && (validName[0] < 'a' || validName[0] > 'z') {
validName = "m2c_mail_" + validName
} else {
validName = "m2c_" + validName
}
return validName
}
// ConvertMessage converts an IMAP message to a MailDocument
func ConvertMessage(msg *mail.Message, mailbox string) *MailDocument {
docID := fmt.Sprintf("%s_%d", mailbox, msg.UID)
doc := &MailDocument{
ID: docID,
SourceUID: fmt.Sprintf("%d", msg.UID),
Mailbox: mailbox,
From: msg.From,
To: msg.To,
Subject: msg.Subject,
Date: msg.Date,
Body: msg.Body,
Headers: msg.Headers,
StoredAt: time.Now(),
DocType: "mail",
HasAttachments: len(msg.Attachments) > 0,
}
// Don't add attachment metadata here - CouchDB will handle this when we store attachments
// We'll add the attachment metadata after successful document creation
return doc
}
// StoreMessage stores a mail message in CouchDB with attachments
func (c *Client) StoreMessage(ctx context.Context, dbName string, doc *MailDocument, msg *mail.Message) error {
db := c.DB(dbName)
if db.Err() != nil {
return db.Err()
}
// Check if document already exists
exists, err := c.DocumentExists(ctx, dbName, doc.ID)
if err != nil {
return fmt.Errorf("failed to check if document exists: %w", err)
}
if exists {
return nil // Document already exists, skip
}
// Store the document first (without attachments)
rev, err := db.Put(ctx, doc.ID, doc)
if err != nil {
return fmt.Errorf("failed to store document: %w", err)
}
// If there are attachments, store them as CouchDB attachments
if msg != nil && len(msg.Attachments) > 0 {
currentRev := rev
for _, att := range msg.Attachments {
newRev, err := c.StoreAttachment(ctx, dbName, doc.ID, currentRev, att.Filename, att.ContentType, att.Content)
if err != nil {
return fmt.Errorf("failed to store attachment %s: %w", att.Filename, err)
}
currentRev = newRev // Update revision for next attachment
}
}
return nil
}
// StoreAttachment stores an attachment to an existing CouchDB document
func (c *Client) StoreAttachment(ctx context.Context, dbName, docID, rev, filename, contentType string, content []byte) (string, error) {
db := c.DB(dbName)
if db.Err() != nil {
return "", db.Err()
}
att := &kivik.Attachment{
Filename: filename,
ContentType: contentType,
Content: io.NopCloser(strings.NewReader(string(content))),
}
newRev, err := db.PutAttachment(ctx, docID, att, kivik.Rev(rev))
if err != nil {
return "", fmt.Errorf("failed to store attachment: %w", err)
}
return newRev, nil
}
// StoreMessages stores multiple mail messages in CouchDB with their corresponding attachments
func (c *Client) StoreMessages(ctx context.Context, dbName string, docs []*MailDocument, messages []*mail.Message) error {
for i, doc := range docs {
var msg *mail.Message
if i < len(messages) {
msg = messages[i]
}
if err := c.StoreMessage(ctx, dbName, doc, msg); err != nil {
return err
}
}
return nil
}
// DocumentExists checks if a document with the given ID already exists.
func (c *Client) DocumentExists(ctx context.Context, dbName, docID string) (bool, error) {
db := c.DB(dbName)
if db.Err() != nil {
return false, db.Err()
}
row := db.Get(ctx, docID)
return row.Err() == nil, nil
}
// GetAllMailDocumentIDs returns all mail document IDs from a database for a specific mailbox
func (c *Client) GetAllMailDocumentIDs(ctx context.Context, dbName, mailbox string) (map[string]bool, error) {
db := c.DB(dbName)
if db.Err() != nil {
return nil, db.Err()
}
// Create a view query to get all document IDs for the specified mailbox
rows := db.AllDocs(ctx)
docIDs := make(map[string]bool)
for rows.Next() {
docID, err := rows.ID()
if err != nil {
continue
}
// Filter by mailbox prefix (documents are named like "INBOX_123")
if mailbox == "" || strings.HasPrefix(docID, mailbox+"_") {
docIDs[docID] = true
}
}
if rows.Err() != nil {
return nil, rows.Err()
}
return docIDs, nil
}
// DeleteDocument removes a document from CouchDB
func (c *Client) DeleteDocument(ctx context.Context, dbName, docID string) error {
db := c.DB(dbName)
if db.Err() != nil {
return db.Err()
}
// Get the current revision
row := db.Get(ctx, docID)
if row.Err() != nil {
return row.Err() // Document doesn't exist or other error
}
var doc struct {
Rev string `json:"_rev"`
}
if err := row.ScanDoc(&doc); err != nil {
return err
}
// Delete the document
_, err := db.Delete(ctx, docID, doc.Rev)
return err
}
// SyncMailbox synchronizes a mailbox between mail server and CouchDB
// In sync mode: removes documents from CouchDB that are no longer in the mail account
// In archive mode: keeps all documents (no removal)
func (c *Client) SyncMailbox(ctx context.Context, dbName, mailbox string, currentMessageUIDs map[uint32]bool, syncMode bool) error {
if !syncMode {
return nil // Archive mode - don't remove anything
}
// Get all existing document IDs for this mailbox from CouchDB
existingDocs, err := c.GetAllMailDocumentIDs(ctx, dbName, mailbox)
if err != nil {
return fmt.Errorf("failed to get existing documents: %w", err)
}
// Find documents that should be removed (exist in CouchDB but not in mail account)
var toDelete []string
for docID := range existingDocs {
// Extract UID from document ID (format: "mailbox_uid")
parts := strings.Split(docID, "_")
if len(parts) < 2 {
continue
}
uidStr := parts[len(parts)-1]
uid := uint32(0)
if _, err := fmt.Sscanf(uidStr, "%d", &uid); err != nil {
continue
}
// If this UID is not in the current mail account, mark for deletion
if !currentMessageUIDs[uid] {
toDelete = append(toDelete, docID)
}
}
// Delete documents that are no longer in the mail account
for _, docID := range toDelete {
if err := c.DeleteDocument(ctx, dbName, docID); err != nil {
return fmt.Errorf("failed to delete document %s: %w", docID, err)
}
}
if len(toDelete) > 0 {
fmt.Printf(" Sync mode: Removed %d documents no longer in mail account\n", len(toDelete))
}
return nil
}
// GetSyncMetadata retrieves the sync metadata for a specific mailbox
func (c *Client) GetSyncMetadata(ctx context.Context, dbName, mailbox string) (*SyncMetadata, error) {
db := c.DB(dbName)
if db.Err() != nil {
return nil, db.Err()
}
metadataID := fmt.Sprintf("sync_metadata_%s", mailbox)
row := db.Get(ctx, metadataID)
if row.Err() != nil {
// If metadata doesn't exist, return nil (not an error for first sync)
return nil, nil
}
var metadata SyncMetadata
if err := row.ScanDoc(&metadata); err != nil {
return nil, fmt.Errorf("failed to scan sync metadata: %w", err)
}
return &metadata, nil
}
// StoreSyncMetadata stores or updates sync metadata for a mailbox
func (c *Client) StoreSyncMetadata(ctx context.Context, dbName string, metadata *SyncMetadata) error {
db := c.DB(dbName)
if db.Err() != nil {
return db.Err()
}
metadata.ID = fmt.Sprintf("sync_metadata_%s", metadata.Mailbox)
metadata.DocType = "sync_metadata"
metadata.UpdatedAt = time.Now()
// Check if metadata already exists to get current revision
existing, err := c.GetSyncMetadata(ctx, dbName, metadata.Mailbox)
if err != nil {
return fmt.Errorf("failed to check existing sync metadata: %w", err)
}
if existing != nil {
metadata.Rev = existing.Rev
}
_, err = db.Put(ctx, metadata.ID, metadata)
if err != nil {
return fmt.Errorf("failed to store sync metadata: %w", err)
}
return nil
}

View file

@ -1,16 +0,0 @@
module mail2couch
go 1.24.4
require (
github.com/emersion/go-imap/v2 v2.0.0-beta.5
github.com/go-kivik/kivik/v4 v4.4.0
)
require (
github.com/emersion/go-message v0.18.1 // indirect
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 // indirect
github.com/google/uuid v1.6.0 // indirect
golang.org/x/net v0.25.0 // indirect
golang.org/x/sync v0.10.0 // indirect
)

View file

@ -1,65 +0,0 @@
github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=
github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=
github.com/cenkalti/backoff/v4 v4.1.3 h1:cFAlzYUlVYDysBEH2T5hyJZMh3+5+WCBvSnK6Q8UtC4=
github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/emersion/go-imap/v2 v2.0.0-beta.5 h1:H3858DNmBuXyMK1++YrQIRdpKE1MwBc+ywBtg3n+0wA=
github.com/emersion/go-imap/v2 v2.0.0-beta.5/go.mod h1:BZTFHsS1hmgBkFlHqbxGLXk2hnRqTItUgwjSSCsYNAk=
github.com/emersion/go-message v0.18.1 h1:tfTxIoXFSFRwWaZsgnqS1DSZuGpYGzSmCZD8SK3QA2E=
github.com/emersion/go-message v0.18.1/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA=
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 h1:hH4PQfOndHDlpzYfLAAfl63E8Le6F2+EL/cdhlkyRJY=
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
github.com/go-kivik/kivik/v4 v4.4.0 h1:1YMqNvRMIIC+CJUtyldD7c4Czl6SqdUcnbusCoFOTfk=
github.com/go-kivik/kivik/v4 v4.4.0/go.mod h1:DnPzIEO7CcLOqJNuqxuo7EMZeK4bPsEbUSSmAfi+tL4=
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0 h1:nHoRIX8iXob3Y2kdt9KsjyIb7iApSvb3vgsd93xb5Ow=
github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0/go.mod h1:c1tRKs5Tx7E2+uHGSyyncziFjvGpgv4H2HrqXeUQ/Uk=
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
gitlab.com/flimzy/testy v0.14.0 h1:2nZV4Wa1OSJb3rOKHh0GJqvvhtE03zT+sKnPCI0owfQ=
gitlab.com/flimzy/testy v0.14.0/go.mod h1:m3aGuwdXc+N3QgnH+2Ar2zf1yg0UxNdIaXKvC5SlfMk=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=

View file

@ -1,481 +0,0 @@
package mail
import (
"bytes"
"fmt"
"io"
"log"
"mime"
"path/filepath"
"strings"
"time"
"github.com/emersion/go-imap/v2"
"github.com/emersion/go-imap/v2/imapclient"
"github.com/emersion/go-message"
"mail2couch/config"
)
// ImapClient wraps the IMAP client
type ImapClient struct {
*imapclient.Client
}
// Message represents an email message retrieved from IMAP
type Message struct {
UID uint32
From []string
To []string
Subject string
Date time.Time
Body string
Headers map[string][]string
Attachments []Attachment
}
// Attachment represents an email attachment
type Attachment struct {
Filename string
ContentType string
Content []byte
}
// NewImapClient creates a new IMAP client from the configuration
func NewImapClient(source *config.MailSource) (*ImapClient, error) {
addr := fmt.Sprintf("%s:%d", source.Host, source.Port)
var client *imapclient.Client
var err error
// Try TLS first for standard IMAPS ports (993, 465)
if source.Port == 993 || source.Port == 465 {
client, err = imapclient.DialTLS(addr, nil)
} else {
// Use insecure connection for other ports (143, 3143, etc.)
client, err = imapclient.DialInsecure(addr, nil)
}
if err != nil {
return nil, fmt.Errorf("failed to dial IMAP server: %w", err)
}
if err := client.Login(source.User, source.Password).Wait(); err != nil {
return nil, fmt.Errorf("failed to login: %w", err)
}
return &ImapClient{client}, nil
}
// ListMailboxes lists all available mailboxes
func (c *ImapClient) ListMailboxes() ([]string, error) {
var mailboxes []string
cmd := c.List("", "*", nil)
infos, err := cmd.Collect()
if err != nil {
return nil, err
}
for _, info := range infos {
mailboxes = append(mailboxes, info.Mailbox)
}
return mailboxes, nil
}
// GetMessages retrieves messages from a specific mailbox with filtering support
// Returns messages and a map of all current UIDs in the mailbox
// maxMessages: 0 means no limit, > 0 limits the number of messages to fetch
// since: if provided, only fetch messages newer than this date (for incremental sync)
func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages int, messageFilter *config.MessageFilter) ([]*Message, map[uint32]bool, error) {
// Select the mailbox
mbox, err := c.Select(mailbox, nil).Wait()
if err != nil {
return nil, nil, fmt.Errorf("failed to select mailbox %s: %w", mailbox, err)
}
if mbox.NumMessages == 0 {
return []*Message{}, make(map[uint32]bool), nil
}
var messages []*Message
currentUIDs := make(map[uint32]bool)
// First, get all current UIDs in the mailbox for sync purposes
allUIDsSet := imap.SeqSet{}
allUIDsSet.AddRange(1, mbox.NumMessages)
// Fetch UIDs for all messages to track current state
uidCmd := c.Fetch(allUIDsSet, &imap.FetchOptions{UID: true})
for {
msg := uidCmd.Next()
if msg == nil {
break
}
data, err := msg.Collect()
if err != nil {
continue
}
if data.UID != 0 {
currentUIDs[uint32(data.UID)] = true
}
}
uidCmd.Close()
// Determine which messages to fetch based on since date
var seqSet imap.SeqSet
if since != nil {
// Use IMAP SEARCH to find messages since the specified date
searchCriteria := &imap.SearchCriteria{
Since: *since,
}
searchCmd := c.Search(searchCriteria, nil)
searchResults, err := searchCmd.Wait()
if err != nil {
log.Printf("IMAP SEARCH failed, falling back to fetch all: %v", err)
// Fall back to fetching all messages
numToFetch := mbox.NumMessages
if maxMessages > 0 && int(numToFetch) > maxMessages {
numToFetch = uint32(maxMessages)
}
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
} else {
// Convert search results to sequence set
searchSeqNums := searchResults.AllSeqNums()
if len(searchSeqNums) == 0 {
return []*Message{}, currentUIDs, nil
}
// Limit results if maxMessages is specified
if maxMessages > 0 && len(searchSeqNums) > maxMessages {
searchSeqNums = searchSeqNums[len(searchSeqNums)-maxMessages:]
}
for _, seqNum := range searchSeqNums {
seqSet.AddNum(seqNum)
}
}
} else {
// No since date specified, fetch recent messages up to maxMessages
numToFetch := mbox.NumMessages
if maxMessages > 0 && int(numToFetch) > maxMessages {
numToFetch = uint32(maxMessages)
}
if numToFetch == 0 {
return []*Message{}, currentUIDs, nil
}
// Fetch the most recent messages
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
}
// Fetch message data - get envelope and full message body
options := &imap.FetchOptions{
Envelope: true,
UID: true,
BodySection: []*imap.FetchItemBodySection{
{}, // Empty section gets the entire message
},
}
fetchCmd := c.Fetch(seqSet, options)
for {
msg := fetchCmd.Next()
if msg == nil {
break
}
parsedMsg, err := c.parseMessage(msg)
if err != nil {
log.Printf("Failed to parse message: %v", err)
continue
}
// Apply message-level keyword filtering
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) {
continue // Skip this message due to keyword filter
}
messages = append(messages, parsedMsg)
}
if err := fetchCmd.Close(); err != nil {
return nil, nil, fmt.Errorf("failed to fetch messages: %w", err)
}
return messages, currentUIDs, nil
}
// parseMessage parses an IMAP fetch response into our Message struct
func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) {
msg := &Message{
UID: fetchMsg.SeqNum, // Using sequence number for now
Headers: make(map[string][]string),
Attachments: []Attachment{},
}
// Collect all fetch data first
buffer, err := fetchMsg.Collect()
if err != nil {
return nil, fmt.Errorf("failed to collect fetch data: %w", err)
}
// Parse envelope for basic headers
if buffer.Envelope != nil {
env := buffer.Envelope
msg.Subject = env.Subject
msg.Date = env.Date
// Parse From addresses
for _, addr := range env.From {
if addr.Mailbox != "" {
fullAddr := addr.Mailbox
if addr.Host != "" {
fullAddr = addr.Mailbox + "@" + addr.Host
}
msg.From = append(msg.From, fullAddr)
}
}
// Parse To addresses
for _, addr := range env.To {
if addr.Mailbox != "" {
fullAddr := addr.Mailbox
if addr.Host != "" {
fullAddr = addr.Mailbox + "@" + addr.Host
}
msg.To = append(msg.To, fullAddr)
}
}
}
// Get UID if available
if buffer.UID != 0 {
msg.UID = uint32(buffer.UID)
}
// Parse full message content
if len(buffer.BodySection) > 0 {
bodyBuffer := buffer.BodySection[0]
reader := bytes.NewReader(bodyBuffer.Bytes)
// Parse the message using go-message
entity, err := message.Read(reader)
if err != nil {
return nil, fmt.Errorf("failed to parse message: %w", err)
}
// Extract headers
header := entity.Header
for field := header.Fields(); field.Next(); {
key := field.Key()
value, _ := field.Text()
msg.Headers[key] = append(msg.Headers[key], value)
}
// Parse message body and attachments
if err := c.parseMessageBody(entity, msg); err != nil {
return nil, fmt.Errorf("failed to parse message body: %w", err)
}
}
return msg, nil
}
// parseMessageBody extracts the body and attachments from a message entity
func (c *ImapClient) parseMessageBody(entity *message.Entity, msg *Message) error {
mediaType, _, err := entity.Header.ContentType()
if err != nil {
// Default to text/plain if no content type
mediaType = "text/plain"
}
if strings.HasPrefix(mediaType, "multipart/") {
// Handle multipart message
mr := entity.MultipartReader()
if mr == nil {
return fmt.Errorf("failed to create multipart reader")
}
for {
part, err := mr.NextPart()
if err == io.EOF {
break
}
if err != nil {
return fmt.Errorf("failed to read multipart: %w", err)
}
if err := c.parseMessagePart(part, msg); err != nil {
log.Printf("Failed to parse message part: %v", err)
// Continue processing other parts
}
}
} else {
// Handle single part message
if err := c.parseMessagePart(entity, msg); err != nil {
return err
}
}
return nil
}
// parseMessagePart processes a single message part (body or attachment)
func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) error {
mediaType, params, err := entity.Header.ContentType()
if err != nil {
mediaType = "text/plain"
}
// Get content disposition
disposition, dispositionParams, _ := entity.Header.ContentDisposition()
// Determine if this is an attachment
isAttachment := disposition == "attachment" ||
(disposition == "inline" && dispositionParams["filename"] != "") ||
params["name"] != ""
if isAttachment {
// Handle attachment
filename := dispositionParams["filename"]
if filename == "" {
filename = params["name"]
}
if filename == "" {
filename = "unnamed_attachment"
}
// Decode filename if needed
decoder := &mime.WordDecoder{}
filename, _ = decoder.DecodeHeader(filename)
// Read attachment content
content, err := io.ReadAll(entity.Body)
if err != nil {
return fmt.Errorf("failed to read attachment content: %w", err)
}
attachment := Attachment{
Filename: filename,
ContentType: mediaType,
Content: content,
}
msg.Attachments = append(msg.Attachments, attachment)
} else if strings.HasPrefix(mediaType, "text/") && msg.Body == "" {
// Handle text body (only take the first text part as body)
bodyBytes, err := io.ReadAll(entity.Body)
if err != nil {
return fmt.Errorf("failed to read message body: %w", err)
}
msg.Body = string(bodyBytes)
}
return nil
}
// ShouldProcessMailbox checks if a mailbox should be processed based on filters with wildcard support
func (c *ImapClient) ShouldProcessMailbox(mailbox string, filter *config.FolderFilter) bool {
// If include list is specified, mailbox must match at least one pattern
if len(filter.Include) > 0 {
found := false
for _, pattern := range filter.Include {
// Handle special case: "*" means include all folders
if pattern == "*" {
found = true
break
}
// Use filepath.Match for wildcard pattern matching
if matched, err := filepath.Match(pattern, mailbox); err == nil && matched {
found = true
break
}
// Also support exact string matching for backwards compatibility
if mailbox == pattern {
found = true
break
}
}
if !found {
return false
}
}
// If exclude list is specified, mailbox must not match any exclude pattern
for _, pattern := range filter.Exclude {
// Use filepath.Match for wildcard pattern matching
if matched, err := filepath.Match(pattern, mailbox); err == nil && matched {
return false
}
// Also support exact string matching for backwards compatibility
if mailbox == pattern {
return false
}
}
return true
}
// ShouldProcessMessage checks if a message should be processed based on keyword filters
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool {
// Check subject keywords
if len(filter.SubjectKeywords) > 0 {
if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) {
return false
}
}
// Check sender keywords
if len(filter.SenderKeywords) > 0 {
senderMatch := false
for _, sender := range msg.From {
if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) {
senderMatch = true
break
}
}
if !senderMatch {
return false
}
}
// Check recipient keywords
if len(filter.RecipientKeywords) > 0 {
recipientMatch := false
for _, recipient := range msg.To {
if c.containsAnyKeyword(strings.ToLower(recipient), filter.RecipientKeywords) {
recipientMatch = true
break
}
}
if !recipientMatch {
return false
}
}
return true
}
// containsAnyKeyword checks if the text contains any of the specified keywords (case-insensitive)
func (c *ImapClient) containsAnyKeyword(text string, keywords []string) bool {
for _, keyword := range keywords {
if strings.Contains(text, strings.ToLower(keyword)) {
return true
}
}
return false
}
// Logout logs the client out
func (c *ImapClient) Logout() {
if err := c.Client.Logout(); err != nil {
log.Printf("Failed to logout: %v", err)
}
}

Binary file not shown.

View file

@ -1,205 +0,0 @@
package main
import (
"context"
"fmt"
"log"
"time"
"mail2couch/config"
"mail2couch/couch"
"mail2couch/mail"
)
func main() {
args := config.ParseCommandLine()
cfg, err := config.LoadConfigWithDiscovery(args)
if err != nil {
log.Fatalf("Failed to load configuration: %v", err)
}
// Initialize CouchDB client
couchClient, err := couch.NewClient(&cfg.CouchDb)
if err != nil {
log.Fatalf("Failed to create CouchDB client: %v", err)
}
fmt.Printf("Found %d mail source(s) to process.\n", len(cfg.MailSources))
for _, source := range cfg.MailSources {
if !source.Enabled {
continue
}
// Generate per-account database name
dbName := couch.GenerateAccountDBName(source.Name, source.User)
// Ensure the account-specific database exists
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
err = couchClient.EnsureDB(ctx, dbName)
cancel()
if err != nil {
log.Printf("Could not ensure CouchDB database '%s' exists (is it running?): %v", dbName, err)
continue
} else {
fmt.Printf("CouchDB database '%s' is ready for account: %s\n", dbName, source.Name)
}
fmt.Printf(" - Processing source: %s\n", source.Name)
if source.Protocol == "imap" {
err := processImapSource(&source, couchClient, dbName, args.MaxMessages)
if err != nil {
log.Printf(" ERROR: Failed to process IMAP source %s: %v", source.Name, err)
}
}
}
}
func processImapSource(source *config.MailSource, couchClient *couch.Client, dbName string, maxMessages int) error {
fmt.Printf(" Connecting to IMAP server: %s:%d\n", source.Host, source.Port)
imapClient, err := mail.NewImapClient(source)
if err != nil {
return fmt.Errorf("failed to connect to IMAP server: %w", err)
}
defer imapClient.Logout()
fmt.Println(" IMAP connection successful.")
mailboxes, err := imapClient.ListMailboxes()
if err != nil {
return fmt.Errorf("failed to list mailboxes: %w", err)
}
fmt.Printf(" Found %d mailboxes.\n", len(mailboxes))
// Parse the since date from config if provided (fallback for first sync)
var configSinceDate *time.Time
if source.MessageFilter.Since != "" {
parsed, err := time.Parse("2006-01-02", source.MessageFilter.Since)
if err != nil {
log.Printf(" WARNING: Invalid since date format '%s', ignoring filter", source.MessageFilter.Since)
} else {
configSinceDate = &parsed
}
}
totalMessages := 0
totalStored := 0
// Process each mailbox
for _, mailbox := range mailboxes {
// Check if this mailbox should be processed based on filters
if !imapClient.ShouldProcessMailbox(mailbox, &source.FolderFilter) {
fmt.Printf(" Skipping mailbox: %s (filtered)\n", mailbox)
continue
}
fmt.Printf(" Processing mailbox: %s (mode: %s)\n", mailbox, source.Mode)
// Get sync metadata to determine incremental sync date
syncCtx, syncCancel := context.WithTimeout(context.Background(), 10*time.Second)
syncMetadata, err := couchClient.GetSyncMetadata(syncCtx, dbName, mailbox)
syncCancel()
if err != nil {
log.Printf(" ERROR: Failed to get sync metadata for %s: %v", mailbox, err)
continue
}
// Determine the since date for incremental sync
var sinceDate *time.Time
if syncMetadata != nil {
// Use last sync time for incremental sync
sinceDate = &syncMetadata.LastSyncTime
fmt.Printf(" Incremental sync since: %s (last synced %d messages)\n",
sinceDate.Format("2006-01-02 15:04:05"), syncMetadata.MessageCount)
} else {
// First sync - use config since date if available
sinceDate = configSinceDate
if sinceDate != nil {
fmt.Printf(" First sync since: %s (from config)\n", sinceDate.Format("2006-01-02"))
} else {
fmt.Printf(" First full sync (no date filter)\n")
}
}
// Retrieve messages from the mailbox
messages, currentUIDs, err := imapClient.GetMessages(mailbox, sinceDate, maxMessages, &source.MessageFilter)
if err != nil {
log.Printf(" ERROR: Failed to get messages from %s: %v", mailbox, err)
continue
}
// Perform sync/archive logic
mailboxSyncCtx, mailboxSyncCancel := context.WithTimeout(context.Background(), 30*time.Second)
err = couchClient.SyncMailbox(mailboxSyncCtx, dbName, mailbox, currentUIDs, source.IsSyncMode())
mailboxSyncCancel()
if err != nil {
log.Printf(" ERROR: Failed to sync mailbox %s: %v", mailbox, err)
continue
}
if len(messages) == 0 {
fmt.Printf(" No new messages found in %s\n", mailbox)
continue
}
fmt.Printf(" Found %d messages in %s\n", len(messages), mailbox)
totalMessages += len(messages)
// Convert messages to CouchDB documents
var docs []*couch.MailDocument
for _, msg := range messages {
doc := couch.ConvertMessage(msg, mailbox)
docs = append(docs, doc)
}
// Store messages in CouchDB with attachments
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
stored := 0
for i, doc := range docs {
err := couchClient.StoreMessage(ctx, dbName, doc, messages[i])
if err != nil {
log.Printf(" ERROR: Failed to store message %s: %v", doc.ID, err)
} else {
stored++
}
}
cancel()
fmt.Printf(" Stored %d/%d messages from %s\n", stored, len(messages), mailbox)
totalStored += stored
// Update sync metadata after successful processing
if len(messages) > 0 {
// Find the highest UID processed
var maxUID uint32
for _, msg := range messages {
if msg.UID > maxUID {
maxUID = msg.UID
}
}
// Create/update sync metadata
newMetadata := &couch.SyncMetadata{
Mailbox: mailbox,
LastSyncTime: time.Now(),
LastMessageUID: maxUID,
MessageCount: stored,
}
// Store sync metadata
metadataCtx, metadataCancel := context.WithTimeout(context.Background(), 10*time.Second)
err = couchClient.StoreSyncMetadata(metadataCtx, dbName, newMetadata)
metadataCancel()
if err != nil {
log.Printf(" WARNING: Failed to store sync metadata for %s: %v", mailbox, err)
} else {
fmt.Printf(" Updated sync metadata (last UID: %d)\n", maxUID)
}
}
}
fmt.Printf(" Summary: Processed %d messages, stored %d new messages\n", totalMessages, totalStored)
return nil
}

View file

@ -1,250 +0,0 @@
# mail2couch Test Environment
This directory contains a complete test environment for mail2couch using Podman containers.
## Overview
The test environment provides:
- **CouchDB**: Database for storing email messages
- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages
- **Test Configuration**: Ready-to-use config for testing both sync and archive modes
## Quick Start
### Run Basic Integration Tests
```bash
./run-tests.sh
```
This comprehensive test will:
1. Start all containers with cleanup
2. Populate test data
3. Build and run mail2couch
4. Verify database creation and document storage
5. Test incremental sync behavior
6. Clean up automatically
### Run Wildcard Pattern Tests
```bash
./test-wildcard-patterns.sh
```
This will test various wildcard folder patterns including:
- `*` (all folders)
- `Work*` (prefix patterns)
- `*/Drafts` (subfolder patterns)
- Complex include/exclude combinations
### Run Incremental Sync Tests
```bash
./test-incremental-sync.sh
```
This will test incremental synchronization functionality:
- First sync establishes baseline
- New messages are added to test accounts
- Second sync should only fetch new messages
- Sync metadata tracking and IMAP SEARCH with SINCE
### Manual Testing Environment
```bash
# Start persistent test environment (for manual experimentation)
./start-test-env.sh
# Run mail2couch manually with different configurations
cd ../go
./mail2couch -config ../test/config-test.json
./mail2couch -config ../test/config-wildcard-examples.json
# Stop test environment when done
cd ../test
./stop-test-env.sh
```
## Test Scripts Overview
### Automated Testing (Recommended)
- **`./run-tests.sh`**: Complete integration test with automatic cleanup
- Starts containers, populates data, runs mail2couch, verifies results
- Tests basic functionality, database creation, and incremental sync
- Cleans up automatically - perfect for CI/CD or quick validation
### Specialized Feature Testing
- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing
- Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns
- Self-contained with own setup/teardown
- **`./test-incremental-sync.sh`**: Incremental synchronization testing
- Tests sync metadata tracking and IMAP SEARCH with SINCE
- Multi-step validation: baseline sync → add messages → incremental sync
- Self-contained with own setup/teardown
### Manual Testing Environment
- **`./start-test-env.sh`**: Start persistent test containers
- Keeps environment running for manual experimentation
- Populates test data once
- Use with different configurations for development
- **`./stop-test-env.sh`**: Clean up manual test environment
- Only needed after using `start-test-env.sh`
## Test Accounts
The test environment includes these IMAP accounts:
| Username | Password | Mode | Folder Pattern | Purpose |
|----------|----------|------|---------------|---------|
| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test |
| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test |
| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test |
| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test |
Each account contains:
- 10 messages in INBOX (every 3rd has an attachment)
- 3 messages in each additional folder
- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash`
- Various message types for comprehensive wildcard testing
## Services
### CouchDB
- **URL**: http://localhost:5984
- **Admin**: `admin` / `password`
- **Web UI**: http://localhost:5984/_utils
### GreenMail Server
- **Host**: localhost
- **IMAP Port**: 3143 (plain)
- **IMAPS Port**: 3993 (SSL)
- **SMTP Port**: 3025
- **Server**: GreenMail (Java-based test server)
## Database Structure
mail2couch will create separate databases for each mail source (with `m2c_` prefix):
- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
- `m2c_work_pattern_test` - Work Pattern Test (sync mode)
- `m2c_specific_folders_only` - Specific Folders Only (archive mode)
- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode)
Each database contains documents with:
- `mailbox` field indicating the origin folder
- Native CouchDB attachments for email attachments
- Full message headers and body content
## Testing Sync vs Archive Modes
### Sync Mode (`syncuser`)
- Database exactly matches mail account
- If messages are deleted from IMAP, they're removed from CouchDB
- 1-to-1 relationship
### Archive Mode (`archiveuser`, `testuser1`)
- Database preserves all messages ever seen
- Messages deleted from IMAP remain in CouchDB
- Archive/backup behavior
## Wildcard Pattern Examples
The test environment demonstrates these wildcard patterns:
### All Folders Pattern (`*`)
```json
{
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash"]
}
}
```
Processes all folders except Drafts and Trash.
### Work Pattern (`Work*`)
```json
{
"folderFilter": {
"include": ["Work*", "Important*", "INBOX"],
"exclude": ["*Temp*"]
}
}
```
Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp.
### Specific Folders
```json
{
"folderFilter": {
"include": ["INBOX", "Sent", "Personal"],
"exclude": []
}
}
```
Only processes the exact named folders.
### Subfolder Pattern (`Work/*`)
```json
{
"folderFilter": {
"include": ["Work/*", "Archive/*"],
"exclude": ["*/Drafts"]
}
}
```
Includes all subfolders under Work and Archive, but excludes any Drafts subfolder.
## File Structure
```
test/
├── podman-compose.yml # Container orchestration (GreenMail + CouchDB)
├── config-test.json # Main test configuration with wildcard examples
├── config-wildcard-examples.json # Advanced wildcard patterns
├── run-tests.sh # Automated integration test (recommended)
├── test-wildcard-patterns.sh # Specialized wildcard pattern testing
├── test-incremental-sync.sh # Specialized incremental sync testing
├── start-test-env.sh # Start persistent test environment
├── stop-test-env.sh # Stop test environment
├── populate-greenmail.py # Create test messages across multiple folders
├── dovecot/ # Dovecot configuration (legacy, unused)
└── README.md # This file
```
## Prerequisites
- Podman and podman-compose
- OpenSSL (for certificate generation)
- curl and nc (for connectivity checks)
- Go (for building mail2couch)
## Troubleshooting
### Containers won't start
```bash
# Check podman status
podman ps -a
# View logs
podman logs mail2couch_test_couchdb
podman logs mail2couch_test_imap
```
### CouchDB connection issues
- Verify CouchDB is running: `curl http://localhost:5984`
- Check admin credentials: `admin/password`
### IMAP connection issues
- Test IMAP connection: `nc -z localhost 143`
- Check Dovecot logs: `podman logs mail2couch_test_imap`
### Permission issues
- Ensure scripts are executable: `chmod +x *.sh`
- Check file permissions in dovecot directory
## Advanced Usage
### Add custom test messages
Edit `populate-test-messages.sh` to create additional test scenarios.
### Modify IMAP configuration
Edit `dovecot/dovecot.conf` and restart containers.
### Test with SSL
Update `config-test.json` to use port 993 and enable SSL.
### Custom test scenarios
Create additional configuration files for specific test cases.

View file

@ -1,74 +0,0 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "Wildcard All Folders Test",
"enabled": true,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser1",
"password": "password123",
"mode": "archive",
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash"]
},
"messageFilter": {
"subjectKeywords": ["meeting", "important"],
"senderKeywords": ["@company.com"]
}
},
{
"name": "Work Pattern Test",
"enabled": true,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "syncuser",
"password": "syncpass",
"mode": "sync",
"folderFilter": {
"include": ["Work*", "Important*", "INBOX"],
"exclude": ["*Temp*"]
},
"messageFilter": {
"recipientKeywords": ["support@", "team@"]
}
},
{
"name": "Specific Folders Only",
"enabled": true,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "archiveuser",
"password": "archivepass",
"mode": "archive",
"folderFilter": {
"include": ["INBOX", "Sent", "Personal"],
"exclude": []
},
"messageFilter": {}
},
{
"name": "Subfolder Pattern Test",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser2",
"password": "password456",
"mode": "archive",
"folderFilter": {
"include": ["Work/*", "Archive/*"],
"exclude": ["*/Drafts"]
},
"messageFilter": {}
}
]
}

View file

@ -1,71 +0,0 @@
{
"couchDb": {
"url": "http://localhost:5984",
"user": "admin",
"password": "password"
},
"mailSources": [
{
"name": "All Folders Example",
"enabled": true,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser1",
"password": "password123",
"mode": "archive",
"folderFilter": {
"include": ["*"],
"exclude": ["Drafts", "Trash", "Spam"]
},
"messageFilter": {}
},
{
"name": "Inbox and Sent Only",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "testuser2",
"password": "password456",
"mode": "sync",
"folderFilter": {
"include": ["INBOX", "Sent"],
"exclude": []
},
"messageFilter": {}
},
{
"name": "Work Folders Pattern",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "workuser",
"password": "workpass",
"mode": "archive",
"folderFilter": {
"include": ["Work*", "Projects*", "INBOX"],
"exclude": ["*Temp*", "*Draft*"]
},
"messageFilter": {
"senderKeywords": ["@company.com"]
}
},
{
"name": "Archive Pattern Example",
"enabled": false,
"protocol": "imap",
"host": "localhost",
"port": 3143,
"user": "archiveuser",
"password": "archivepass",
"mode": "archive",
"folderFilter": {
"include": ["Archive*", "*Important*", "INBOX"],
"exclude": ["*Temp"]
},
"messageFilter": {}
}
]
}

View file

@ -1,79 +0,0 @@
# Dovecot configuration for testing mail2couch
# Basic settings
protocols = imap
listen = *
# SSL/TLS settings - make optional for easier testing
ssl = optional
ssl_cert = </etc/dovecot/ssl/server.crt
ssl_key = </etc/dovecot/ssl/server.key
# Authentication
auth_mechanisms = plain login
disable_plaintext_auth = no
# User database
passdb {
driver = passwd-file
args = scheme=plain username_format=%u /etc/dovecot/passwd
}
userdb {
driver = passwd-file
args = username_format=%u /etc/dovecot/users
}
# Mail location
mail_location = maildir:/var/mail/%u
# Mailbox settings
namespace inbox {
type = private
separator = /
prefix =
location =
inbox = yes
hidden = no
list = yes
subscriptions = yes
mailbox Drafts {
special_use = \Drafts
}
mailbox Junk {
special_use = \Junk
}
mailbox Trash {
special_use = \Trash
}
mailbox Sent {
special_use = \Sent
}
}
# Services
service imap-login {
inet_listener imap {
port = 143
}
inet_listener imaps {
port = 993
ssl = yes
}
}
service auth {
unix_listener auth-userdb {
mode = 0666
}
}
# Logging
log_path = /dev/stdout
info_log_path = /dev/stdout
debug_log_path = /dev/stdout
# Process limits
default_process_limit = 10
default_client_limit = 100

View file

@ -1,51 +0,0 @@
#!/bin/sh
# Entrypoint script for Dovecot test container
set -e
echo "Installing Dovecot..."
apk add --no-cache dovecot dovecot-lmtpd openssl
echo "Setting up directories..."
mkdir -p /var/mail
mkdir -p /var/run/dovecot
mkdir -p /var/log/dovecot
# Create dovecot user if it doesn't exist
if ! getent passwd dovecot > /dev/null 2>&1; then
addgroup -g 97 dovecot
adduser -D -u 97 -G dovecot -s /sbin/nologin dovecot
fi
# Set proper ownership
chown -R dovecot:dovecot /var/mail
chown -R dovecot:dovecot /var/run/dovecot
chown -R root:dovecot /etc/dovecot
chmod -R 0640 /etc/dovecot
chmod 0644 /etc/dovecot/dovecot.conf
# Generate SSL certificates if they don't exist
if [ ! -f /etc/dovecot/ssl/server.crt ] || [ ! -f /etc/dovecot/ssl/server.key ]; then
echo "Generating SSL certificates..."
mkdir -p /etc/dovecot/ssl
# Generate DH parameters (small for testing)
openssl dhparam -out /etc/dovecot/ssl/dh.pem 1024
# Generate private key
openssl genrsa -out /etc/dovecot/ssl/server.key 2048
# Generate certificate
openssl req -new -key /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.csr -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
openssl x509 -req -days 365 -in /etc/dovecot/ssl/server.csr -signkey /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.crt
rm /etc/dovecot/ssl/server.csr
fi
# Ensure SSL directory permissions
chown -R dovecot:dovecot /etc/dovecot/ssl
chmod 600 /etc/dovecot/ssl/server.key
chmod 644 /etc/dovecot/ssl/server.crt
echo "Starting Dovecot..."
exec dovecot -F

View file

@ -1,8 +0,0 @@
# Password database for Dovecot testing
# Format: username:password
# Test accounts with simple passwords for testing
testuser1:password123
testuser2:password456
syncuser:syncpass
archiveuser:archivepass

View file

@ -1,8 +0,0 @@
-----BEGIN DH PARAMETERS-----
MIIBDAKCAQEAjcUSAHFs60qgDRg/cT7byhuhF3vwZQhmm1QToCFgG4VWu/EOVXq2
kHxjxmo3hBuJCqUZqTAyF91Tum7A2QuQhXFrxOpRF8EiyVSgBabjN/WcEHIow1uh
Vtb4JOcDl/Q9IJfFT6zyXdQQiHPBOWnpOBKXeQQQIx5plgsrmK0cTO2ZxtyrmHHp
wxtE3INKYuBlGH3Y0zghc+Hoezpf/hbIHZibGQ0l79EtBDQjqmqoDJCIiv5gsTt8
9VpkR6FFvjWTNOb5qY10W/PRhLGjioX29bp1B6qW5PNJcd//cqrBLebKlkAoXnyx
x0uTUy6pmmIt5vdYxx0symrMXZEjrL7uzwIBAgICAOE=
-----END DH PARAMETERS-----

View file

@ -1,20 +0,0 @@
-----BEGIN CERTIFICATE-----
MIIDVzCCAj+gAwIBAgIUYvkZSKbVH08s/3B70AW8IEpTB/kwDQYJKoZIhvcNAQEL
BQAwVDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx
EzARBgNVBAoMCk1haWwyQ291Y2gxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yNTA4
MDExNDIyMzVaFw0yNjA4MDExNDIyMzVaMFQxCzAJBgNVBAYTAlVTMQ0wCwYDVQQI
DARUZXN0MQ0wCwYDVQQHDARUZXN0MRMwEQYDVQQKDApNYWlsMkNvdWNoMRIwEAYD
VQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs
leQcVLlKx7/bJGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vx
Td1mVCm9Lsd9bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDW
knco7/MEatakliVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331
HEFdIqqccpimnIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwo
eN8zBzGFJe9xzqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5
LGN5xSzXvFIzgP20y5s5AgMBAAGjITAfMB0GA1UdDgQWBBTkgYZGp2s74D+1ltyl
rudF/o7jODANBgkqhkiG9w0BAQsFAAOCAQEATc6ekhuk32meLuxhalz6lNwBxfDg
EG3gGUxNwehwgiNCcKIKQFtCwjJde6drOobkRDANtb7g3gSlAxlUCPsO6xnL1c6E
HhehFn++7HOpXvmEy/mnoqBL6PLzRZRMRlDynlPVV9Y82zsdrQiQEhGyNTfgP5dk
u9RMIMQl1hIK381V738b5MXfdpYhmRiTGEd6hCxCnzkx0OakCLM9lnJASr0dYPuh
LYKoClxhr3sV/JsgAmx91BuHGpzaPYQ2zFvCJSqD2ihM7zIl9K2bLIUR87/CznyH
JuPRbgt6/cxzwdqflP73j+TTZdlI4gckEA3H0WhNN4nB2SEjTgS+kDctMA==
-----END CERTIFICATE-----

View file

@ -1,28 +0,0 @@
-----BEGIN PRIVATE KEY-----
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsleQcVLlKx7/b
JGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vxTd1mVCm9Lsd9
bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDWknco7/MEatak
liVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331HEFdIqqccpim
nIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwoeN8zBzGFJe9x
zqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5LGN5xSzXvFIz
gP20y5s5AgMBAAECggEAOfBaM76ns/iqKpYlamnjfIs7svotEjhrHcsub6fWvEsE
XLzRmSqHeWP+t55oo2XeL2zOCofvuUDGnQ+rXE2mHwzhP3FJzsOibm2qmtCJvZwe
ozRj4xTMLILGDnGRhHAJ21cxZM9lPNLnOzri0868DeYimib49xAdroLBLyKRgDGN
O7OAwA4KWAebZRBU8cowEF87WGAI2hOLJA5WIKX7X9SiRDx5sNnDpTpqJvyvuleR
D/wKGHzkQiZx8WNJx5A571dilfKEp8S49o4sJdz8DQ/4ruVov2Vi+nSdMEzHnp5m
M7ZlZ7IcJRJDKPrDYasmxvtM8EiKyJf87/DPANfYFwKBgQDa+ANr2pGg2oMJVJIq
r/mj8wtRsLRYFgfs96BUwp7e9Vo+Rx+E0uZxGGMul6hn4b2/TQljLfiX/CP8ZTyw
MStlNlnAXaF425JNuQIFQ2wGGiGdsx2I1WNw7co4UPVVjH11nr3o30g+NDJcp7Tq
rvpKvGbeoZtHzOj0bF7fA/B7qwKBgQDJxcUUEH5A50n3oUP99aK/DzSA7kcte2Aw
tjv9hbPnbOmcM24Fm+KU7bsRYt9QPa0PU2lV3O1KrHj4q+QRcPFl2P2mzZC+Hzmx
H8dEjMmH8YdrjGqethMoUHJCguNfskNwjgWFlxTSBLY+NffghXNzZgiF9d6WqF48
iqwH+HsAqwKBgQCb/B2D0Xn4WnEKToKpgh6WGmcv1G9EaL1Qo75FYzcFoUaeItBj
MFIUssjEwiinh/pBssFDM9Zpfqar//pRkVVWjnc1P/3tOI1qbKbx1Ou5FRhpXNVn
SovCQMLTh2idfq1JAsJKh/TQyyItOxL4M5n9b2Tgp8MUTPaOWDzlJctEbQKBgEVu
oNq+sjNzY6iq/dKubEqC2PZlCGlGQ1t/2jTrhXTlrZ3qtLmJYvcMt4rMEzxxfNQB
SAYb+CvyHc60l87Ipsj9WovDwUMrS5b/8HpOWCtHmeoQb8Adt4nv5OGuWL/dgAeD
V7MYwjljFbNiruG8CnZzbgtrCCWf2o3KylgT0X/xAoGAUhSdBge5Vpg0JcT1VDgm
q5rgc6dD1LJtXfBaq3w4kHYK/iLFcPOLUKcIJXNbhMwWza/JwVYK6hsCIw3/b4va
NhJ8ABpC3fZqkl28glEF8bnrPAkE1akn2GiBaaEbTCQRMrhZ2SW3JCyjX6yCvvvz
m7b2ZpDMJEMIBmgrK70E3Oo=
-----END PRIVATE KEY-----

View file

@ -1,8 +0,0 @@
# User database for Dovecot testing
# Format: username:uid:gid:home:shell
# Test user accounts
testuser1::1001:1001:/var/mail/testuser1::
testuser2::1002:1002:/var/mail/testuser2::
syncuser::1003:1003:/var/mail/syncuser::
archiveuser::1004:1004:/var/mail/archiveuser::

View file

@ -1,28 +0,0 @@
#!/bin/bash
# Generate SSL certificates for Dovecot testing
set -e
CERT_DIR="dovecot/ssl"
mkdir -p "$CERT_DIR"
# Generate DH parameters
echo "Generating DH parameters..."
openssl dhparam -out "$CERT_DIR/dh.pem" 2048
# Generate private key
echo "Generating private key..."
openssl genrsa -out "$CERT_DIR/server.key" 2048
# Generate certificate signing request
echo "Generating certificate..."
openssl req -new -key "$CERT_DIR/server.key" -out "$CERT_DIR/server.csr" -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
# Generate self-signed certificate
openssl x509 -req -days 365 -in "$CERT_DIR/server.csr" -signkey "$CERT_DIR/server.key" -out "$CERT_DIR/server.crt"
# Clean up CSR
rm "$CERT_DIR/server.csr"
echo "SSL certificates generated successfully in $CERT_DIR/"

View file

@ -1,38 +0,0 @@
version: '3.8'
services:
# CouchDB for testing
couchdb:
image: docker.io/couchdb:3.3
container_name: mail2couch_test_couchdb
environment:
- COUCHDB_USER=admin
- COUCHDB_PASSWORD=password
ports:
- "5984:5984"
volumes:
- couchdb_data:/opt/couchdb/data
networks:
- mail2couch_test
# GreenMail IMAP server for testing
greenmail:
image: docker.io/greenmail/standalone:2.0.1
container_name: mail2couch_test_imap
ports:
- "3143:3143" # IMAP
- "3993:3993" # IMAPS
- "3025:3025" # SMTP
environment:
- GREENMAIL_OPTS=-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.users=testuser1:password123@localhost,testuser2:password456@localhost,syncuser:syncpass@localhost,archiveuser:archivepass@localhost
networks:
- mail2couch_test
depends_on:
- couchdb
volumes:
couchdb_data:
networks:
mail2couch_test:
driver: bridge

View file

@ -1,179 +0,0 @@
#!/usr/bin/env python3
import imaplib
import email
from email.mime.text import MIMEText
from email.mime.multipart import MIMEMultipart
from email.mime.base import MIMEBase
from email import encoders
import time
import sys
def create_simple_message(subject, body, from_addr="test-sender@example.com", to_addr="user@example.com"):
"""Create a simple text message"""
msg = MIMEText(body)
msg['Subject'] = subject
msg['From'] = from_addr
msg['To'] = to_addr
msg['Date'] = email.utils.formatdate(localtime=True)
return msg.as_string()
def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com", to_addr="user@example.com"):
"""Create a message with an attachment"""
msg = MIMEMultipart()
msg['Subject'] = subject
msg['From'] = from_addr
msg['To'] = to_addr
msg['Date'] = email.utils.formatdate(localtime=True)
# Add body
msg.attach(MIMEText(body, 'plain'))
# Add attachment
part = MIMEBase('text', 'plain')
part.set_payload(attachment_content)
encoders.encode_base64(part)
part.add_header('Content-Disposition', 'attachment; filename="attachment.txt"')
msg.attach(part)
return msg.as_string()
def populate_user_mailbox(username, password, host='localhost', port=3143):
"""Populate a user's mailbox with test messages"""
print(f"Connecting to {username}@{host}:{port}")
try:
# Connect to IMAP server
imap = imaplib.IMAP4(host, port)
imap.login(username, password)
# Create additional folders for testing wildcards
# These folders are designed to test various wildcard patterns
test_folders = [
'Sent', # Exact match
'Work/Projects', # Work/* pattern
'Work/Archive', # Work/* pattern
'Work/Temp', # Work/* but excluded by *Temp*
'Personal', # Exact match
'Important/Urgent', # Important/* pattern
'Important/Meetings', # Important/* pattern
'Archive/2024', # Archive/* pattern
'Archive/Projects', # Archive/* pattern
'Archive/Drafts', # Archive/* but excluded by */Drafts
'Drafts', # Should be excluded
'Trash' # Should be excluded
]
created_folders = ['INBOX'] # INBOX always exists
for folder in test_folders:
try:
imap.create(folder)
created_folders.append(folder)
print(f" Created folder: {folder}")
except Exception as e:
print(f" Folder {folder} may already exist or creation failed: {e}")
# Try to select it to see if it exists
try:
imap.select(folder)
created_folders.append(folder)
print(f" Folder {folder} already exists")
except:
pass
print(f"Available folders for {username}: {created_folders}")
# Populate each folder with messages
for folder in created_folders[:3]: # Limit to first 3 folders to avoid too many messages
imap.select(folder)
print(f"Creating messages in {folder} for {username}...")
# Create fewer messages per folder for testing
message_count = 3 if folder != 'INBOX' else 10
for i in range(1, message_count + 1):
if i % 3 == 0 and folder == 'INBOX':
# Every 3rd message has attachment (only in INBOX)
msg = create_message_with_attachment(
f"[{folder}] Test Message {i} with Attachment",
f"This is test message {i} in {folder} for {username} with an attachment.",
f"Sample attachment content for message {i} in {folder}",
"test-sender@example.com",
f"{username}@example.com"
)
print(f" Created message {i} (with attachment)")
else:
# Simple message
msg = create_simple_message(
f"[{folder}] Test Message {i}",
f"This is test message {i} in {folder} for {username}.",
"test-sender@example.com",
f"{username}@example.com"
)
print(f" Created message {i}")
# Append message to current folder
imap.append(folder, None, None, msg.encode('utf-8'))
time.sleep(0.1) # Small delay to avoid overwhelming
# Add special messages only to INBOX for keyword filtering tests
if folder == 'INBOX':
special_messages = [
("Important Meeting Reminder", "This is an important meeting message for testing keyword filters.", "manager@company.com", "team@company.com"),
("Urgent: System Maintenance", "Important notification about system maintenance.", "admin@company.com", f"{username}@example.com"),
("Regular Newsletter", "This is a regular newsletter message.", "newsletter@external.com", f"{username}@example.com"),
("Team Meeting Notes", "Meeting notes from the team.", "secretary@company.com", "support@company.com"),
("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀", "test-sender@example.com", f"{username}@example.com")
]
for subject, body, sender, recipient in special_messages:
msg = create_simple_message(subject, body, sender, recipient)
imap.append(folder, None, None, msg.encode('utf-8'))
print(f" Created special message: {subject} from {sender} to {recipient}")
time.sleep(0.1)
imap.logout()
print(f"✅ Successfully created messages across {len(created_folders[:3])} folders for {username}")
return True
except Exception as e:
print(f"❌ Error populating {username}: {e}")
return False
def main():
print("🚀 Populating GreenMail with test messages using IMAP...")
# Test accounts
accounts = [
("testuser1", "password123"),
("testuser2", "password456"),
("syncuser", "syncpass"),
("archiveuser", "archivepass")
]
# Wait for GreenMail to be ready
print("Waiting for GreenMail to be ready...")
time.sleep(5)
success_count = 0
for username, password in accounts:
if populate_user_mailbox(username, password):
success_count += 1
time.sleep(1) # Brief pause between accounts
print(f"\n🎉 Successfully populated {success_count}/{len(accounts)} accounts!")
if success_count == len(accounts):
print("\n✅ All test accounts ready:")
for username, password in accounts:
print(f" - {username}:{password}@localhost")
print(f"\nGreenMail Services:")
print(f" - IMAP: localhost:3143")
print(f" - IMAPS: localhost:3993")
print(f" - SMTP: localhost:3025")
return 0
else:
print(f"\n❌ Failed to populate {len(accounts) - success_count} accounts")
return 1
if __name__ == "__main__":
sys.exit(main())

View file

@ -1,128 +0,0 @@
#!/bin/bash
# Run basic integration tests with test containers
# This is a comprehensive test that handles its own setup and teardown
set -e
cd "$(dirname "$0")"
echo "🚀 Running basic integration tests..."
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
NC='\033[0m' # No Color
# Function to print colored output
print_status() {
echo -e "${GREEN}[INFO]${NC} $1"
}
print_warning() {
echo -e "${YELLOW}[WARN]${NC} $1"
}
print_error() {
echo -e "${RED}[ERROR]${NC} $1"
}
# Cleanup function
cleanup() {
print_status "Cleaning up test containers..."
podman-compose -f podman-compose.yml down -v 2>/dev/null || true
}
# Set up cleanup trap
trap cleanup EXIT
# Start containers
print_status "Starting test containers..."
podman-compose -f podman-compose.yml up -d
# Wait for containers to be ready
print_status "Waiting for containers to be ready..."
sleep 10
# Check if CouchDB is ready
print_status "Checking CouchDB connectivity..."
timeout=30
while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
timeout=$((timeout - 1))
if [ $timeout -le 0 ]; then
print_error "CouchDB failed to start within 30 seconds"
exit 1
fi
sleep 1
done
print_status "CouchDB is ready!"
# Check if IMAP server is ready
print_status "Checking IMAP server connectivity..."
timeout=30
while ! nc -z localhost 3143 > /dev/null 2>&1; do
timeout=$((timeout - 1))
if [ $timeout -le 0 ]; then
print_error "IMAP server failed to start within 30 seconds"
exit 1
fi
sleep 1
done
print_status "IMAP server is ready!"
# Populate test messages
print_status "Populating test messages..."
python3 ./populate-greenmail.py
# Build mail2couch
print_status "Building mail2couch..."
cd ../go
go build -o mail2couch .
cd ../test
# Run mail2couch with test configuration
print_status "Running mail2couch with test configuration..."
../go/mail2couch -config config-test.json -max-messages 3
# Verify results
print_status "Verifying test results..."
# Check CouchDB databases were created (using correct database names with m2c prefix)
EXPECTED_DBS=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
for db in "${EXPECTED_DBS[@]}"; do
if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then
print_status "✅ Database '$db' created successfully"
else
print_error "❌ Database '$db' was not created"
exit 1
fi
done
# Check document counts
for db in "${EXPECTED_DBS[@]}"; do
doc_count=$(curl -s "http://admin:password@localhost:5984/$db" | grep -o '"doc_count":[0-9]*' | cut -d':' -f2)
if [ "$doc_count" -gt 0 ]; then
print_status "✅ Database '$db' contains $doc_count documents"
else
print_warning "⚠️ Database '$db' contains no documents"
fi
done
# Test sync mode by running again (should show incremental behavior)
print_status "Running mail2couch again to test incremental sync..."
../go/mail2couch -config config-test.json -max-messages 3
print_status "🎉 Basic integration tests completed successfully!"
# Show summary
print_status "Test Summary:"
echo " - IMAP Server: localhost:3143"
echo " - CouchDB: http://localhost:5984"
echo " - Test accounts: testuser1, syncuser, archiveuser"
echo " - Databases created: ${EXPECTED_DBS[*]}"
echo ""
echo "For more comprehensive tests, run:"
echo " - ./test-wildcard-patterns.sh (test folder pattern matching)"
echo " - ./test-incremental-sync.sh (test incremental synchronization)"

View file

@ -1,67 +0,0 @@
#!/bin/bash
# Start test environment for manual testing
cd "$(dirname "$0")"
echo "🚀 Starting mail2couch test environment..."
# Start containers
echo "Starting containers..."
podman-compose -f podman-compose.yml up -d
# Wait for services
echo "Waiting for services to be ready..."
sleep 10
# Check CouchDB
echo "Checking CouchDB..."
timeout=30
while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
timeout=$((timeout - 1))
if [ $timeout -le 0 ]; then
echo "❌ CouchDB failed to start"
exit 1
fi
sleep 1
done
echo "✅ CouchDB is ready at http://localhost:5984"
# Check IMAP
echo "Checking IMAP server..."
timeout=30
while ! nc -z localhost 3143 > /dev/null 2>&1; do
timeout=$((timeout - 1))
if [ $timeout -le 0 ]; then
echo "❌ IMAP server failed to start"
exit 1
fi
sleep 1
done
echo "✅ IMAP server is ready at localhost:3143"
# Populate test data
echo "Populating test messages..."
python3 ./populate-greenmail.py
echo ""
echo "🎉 Test environment is ready!"
echo ""
echo "Services:"
echo " - CouchDB: http://localhost:5984 (admin/password)"
echo " - CouchDB Web UI: http://localhost:5984/_utils"
echo " - IMAP Server: localhost:3143"
echo " - IMAPS Server: localhost:3993"
echo " - SMTP Server: localhost:3025"
echo ""
echo "Test accounts:"
echo " - testuser1:password123"
echo " - testuser2:password456"
echo " - syncuser:syncpass"
echo " - archiveuser:archivepass"
echo ""
echo "To run mail2couch:"
echo " cd ../go && ./mail2couch -config ../test/config-test.json"
echo ""
echo "To stop the environment:"
echo " ./stop-test-env.sh"

View file

@ -1,12 +0,0 @@
#!/bin/bash
# Stop test environment
cd "$(dirname "$0")"
echo "🛑 Stopping mail2couch test environment..."
# Stop and remove containers
podman-compose -f podman-compose.yml down -v
echo "✅ Test environment stopped and cleaned up!"

View file

@ -1,242 +0,0 @@
#!/bin/bash
# Test script to validate incremental sync functionality
# This script tests that mail2couch properly implements incremental synchronization
set -e
echo "🔄 Testing Incremental Sync Functionality"
echo "=========================================="
# Make sure we're in the right directory
cd "$(dirname "$0")/.."
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to check if containers are running
check_containers() {
echo "🔍 Checking if test containers are running..."
if ! podman ps | grep -q "greenmail"; then
echo -e "${RED}❌ GreenMail container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
if ! podman ps | grep -q "couchdb"; then
echo -e "${RED}❌ CouchDB container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
echo -e "${GREEN}✅ Test containers are running${NC}"
}
# Function to populate initial test data
populate_initial_data() {
echo "📧 Populating initial test data..."
cd test
if python3 populate-greenmail.py; then
echo -e "${GREEN}✅ Initial test data populated${NC}"
else
echo -e "${RED}❌ Failed to populate initial test data${NC}"
exit 1
fi
cd ..
}
# Function to build the application
build_app() {
echo "🔨 Building mail2couch..."
cd go
if go build -o mail2couch .; then
echo -e "${GREEN}✅ Build successful${NC}"
else
echo -e "${RED}❌ Build failed${NC}"
exit 1
fi
cd ..
}
# Function to run first sync
run_first_sync() {
echo -e "\n${BLUE}Running first sync...${NC}"
cd go
./mail2couch -config ../test/config-test.json -max-messages 5
cd ..
}
# Function to add new messages to test incremental sync
add_new_messages() {
echo -e "\n${YELLOW}Adding new messages for incremental sync test...${NC}"
# Create a simple Python script to add messages directly to GreenMail
cat > test/add_incremental_messages.py << 'EOF'
#!/usr/bin/env python3
import imaplib
import time
from test.populate_greenmail import create_simple_message
def add_new_messages():
"""Add new messages to test incremental sync"""
accounts = [
("testuser1", "password123"),
("syncuser", "syncpass"),
("archiveuser", "archivepass")
]
for username, password in accounts:
try:
print(f"Adding new messages to {username}...")
imap = imaplib.IMAP4('localhost', 3143)
imap.login(username, password)
imap.select('INBOX')
# Add 3 new messages with timestamps after the first sync
for i in range(1, 4):
subject = f"Incremental Sync Test Message {i}"
body = f"This message was added after the first sync for incremental testing. Message {i} for {username}."
msg = create_simple_message(subject, body, f"incremental-test@example.com", f"{username}@example.com")
imap.append('INBOX', None, None, msg.encode('utf-8'))
print(f" Added: {subject}")
time.sleep(0.1)
imap.logout()
print(f"✅ Added 3 new messages to {username}")
except Exception as e:
print(f"❌ Error adding messages to {username}: {e}")
if __name__ == "__main__":
add_new_messages()
EOF
# Add the parent directory to Python path and run the script
cd test
PYTHONPATH=.. python3 add_incremental_messages.py
cd ..
}
# Function to run second sync (incremental)
run_incremental_sync() {
echo -e "\n${BLUE}Running incremental sync...${NC}"
cd go
./mail2couch -config ../test/config-test.json -max-messages 10
cd ..
}
# Function to verify incremental sync results
verify_results() {
echo -e "\n${YELLOW}Verifying incremental sync results...${NC}"
# Check CouchDB for sync metadata documents
echo "Checking for sync metadata in CouchDB databases..."
# List of expected databases based on test config (with m2c prefix)
databases=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
for db in "${databases[@]}"; do
echo " Checking database: $db"
# Check if database exists
if curl -s -f "http://admin:password@localhost:5984/$db" > /dev/null; then
echo " ✅ Database exists"
# Look for sync metadata documents
metadata_docs=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"" | grep -o '"total_rows":[0-9]*' | cut -d: -f2 || echo "0")
if [ "$metadata_docs" -gt 0 ]; then
echo " ✅ Found sync metadata documents: $metadata_docs"
# Get a sample sync metadata document
sample_doc=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"&include_docs=true&limit=1")
echo " Sample sync metadata:"
echo "$sample_doc" | python3 -m json.tool | grep -E "(lastSyncTime|lastMessageUID|messageCount)" | head -3
else
echo " ⚠️ No sync metadata documents found"
fi
else
echo " ❌ Database does not exist"
fi
done
}
# Main test execution
main() {
echo "Starting incremental sync tests..."
# Pre-test setup
check_containers
build_app
# Clean up any existing data
echo "🧹 Cleaning up existing test data..."
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_wildcard_all_folders_test" > /dev/null || true
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_work_pattern_test" > /dev/null || true
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_specific_folders_only" > /dev/null || true
# Step 1: Populate initial test data
populate_initial_data
# Wait for data to settle
echo "⏳ Waiting for initial data to settle..."
sleep 5
# Step 2: Run first sync to establish baseline
echo -e "\n${YELLOW}=== STEP 1: First Sync (Baseline) ===${NC}"
run_first_sync
# Wait between syncs
echo "⏳ Waiting between syncs..."
sleep 3
# Step 3: Add new messages for incremental sync test
echo -e "\n${YELLOW}=== STEP 2: Add New Messages ===${NC}"
add_new_messages
# Wait for new messages to be ready
echo "⏳ Waiting for new messages to be ready..."
sleep 2
# Step 4: Run incremental sync
echo -e "\n${YELLOW}=== STEP 3: Incremental Sync ===${NC}"
run_incremental_sync
# Step 5: Verify results
echo -e "\n${YELLOW}=== STEP 4: Verification ===${NC}"
verify_results
echo -e "\n${GREEN}🎉 Incremental sync test completed!${NC}"
echo ""
echo "Key features tested:"
echo " ✅ Sync metadata storage and retrieval"
echo " ✅ IMAP SEARCH with SINCE for efficient incremental fetching"
echo " ✅ Last sync timestamp tracking per mailbox"
echo " ✅ Proper handling of first sync vs incremental sync"
echo ""
echo "To verify results manually:"
echo " - Check CouchDB: http://localhost:5984/_utils"
echo " - Look for 'sync_metadata_*' documents in each database"
echo " - Verify incremental messages were added after baseline sync"
}
# Cleanup function
cleanup() {
echo "🧹 Cleaning up test artifacts..."
rm -f test/add_incremental_messages.py
}
# Set trap to cleanup on exit
trap cleanup EXIT
# Run main function if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi

View file

@ -1,137 +0,0 @@
#!/bin/bash
# Test script to validate wildcard folder pattern functionality
# This script tests the various wildcard patterns against the test environment
set -e
echo "🧪 Testing Wildcard Folder Pattern Functionality"
echo "================================================"
# Make sure we're in the right directory
cd "$(dirname "$0")/.."
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
# Function to run mail2couch with a specific config and capture output
run_test() {
local test_name="$1"
local config_file="$2"
local max_messages="$3"
echo -e "\n${BLUE}Testing: $test_name${NC}"
echo "Config: $config_file"
echo "Max messages: $max_messages"
echo "----------------------------------------"
# Run mail2couch and capture output
cd go
if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then
echo -e "${GREEN}✅ Test completed successfully${NC}"
else
echo -e "${RED}❌ Test failed${NC}"
return 1
fi
cd ..
}
# Function to check if containers are running
check_containers() {
echo "🔍 Checking if test containers are running..."
if ! podman ps | grep -q "greenmail"; then
echo -e "${RED}❌ GreenMail container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
if ! podman ps | grep -q "couchdb"; then
echo -e "${RED}❌ CouchDB container not running${NC}"
echo "Please run: cd test && ./start-test-env.sh"
exit 1
fi
echo -e "${GREEN}✅ Test containers are running${NC}"
}
# Function to populate test data
populate_test_data() {
echo "📧 Populating test data..."
cd test
if python3 populate-greenmail.py; then
echo -e "${GREEN}✅ Test data populated successfully${NC}"
else
echo -e "${RED}❌ Failed to populate test data${NC}"
exit 1
fi
cd ..
}
# Function to build the application
build_app() {
echo "🔨 Building mail2couch..."
cd go
if go build -o mail2couch .; then
echo -e "${GREEN}✅ Build successful${NC}"
else
echo -e "${RED}❌ Build failed${NC}"
exit 1
fi
cd ..
}
# Main test execution
main() {
echo "Starting wildcard pattern tests..."
# Pre-test checks
check_containers
build_app
populate_test_data
# Wait a moment for test data to be fully ready
echo "⏳ Waiting for test data to settle..."
sleep 3
# Test 1: Wildcard all folders (*)
echo -e "\n${YELLOW}Test 1: Wildcard All Folders Pattern (*)${NC}"
echo "Expected: Should process all folders except Drafts and Trash"
run_test "Wildcard All Folders" "config-test.json" 3
# Test 2: Work pattern (Work*)
echo -e "\n${YELLOW}Test 2: Work Pattern (Work*)${NC}"
echo "Expected: Should process Work/Projects, Work/Archive but not Work/Temp (excluded by *Temp*)"
run_test "Work Pattern" "config-test.json" 3
# Test 3: Specific folders only
echo -e "\n${YELLOW}Test 3: Specific Folders Only${NC}"
echo "Expected: Should only process INBOX, Sent, and Personal folders"
run_test "Specific Folders" "config-test.json" 3
# Test 4: Advanced wildcard examples
echo -e "\n${YELLOW}Test 4: Advanced Wildcard Examples${NC}"
echo "Expected: Various complex patterns should work correctly"
run_test "Advanced Patterns" "config-wildcard-examples.json" 2
echo -e "\n${GREEN}🎉 All wildcard pattern tests completed!${NC}"
echo ""
echo "To verify results, check the CouchDB databases:"
echo " http://localhost:5984/_utils"
echo ""
echo "Expected databases should be created for each account:"
echo " - wildcard_all_folders_test"
echo " - work_pattern_test"
echo " - specific_folders_only"
echo ""
echo "Each database should contain documents with 'mailbox' field showing origin folder."
}
# Run main function if executed directly
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
main "$@"
fi