Compare commits
7 commits
d0caff800a
...
c2ad55eaaf
| Author | SHA1 | Date | |
|---|---|---|---|
| c2ad55eaaf | |||
| 357cd06264 | |||
| ea6235b674 | |||
| 79f19a8877 | |||
| 44efed908d | |||
| 2ed5ce7ad2 | |||
| 1e4a67d4cb |
33 changed files with 3653 additions and 2 deletions
161
CLAUDE.md
Normal file
161
CLAUDE.md
Normal file
|
|
@ -0,0 +1,161 @@
|
||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## Project Overview
|
||||||
|
|
||||||
|
mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations:
|
||||||
|
- **Go implementation**: Located in `/go/` directory (currently the active implementation)
|
||||||
|
- **Rust implementation**: Planned but not yet implemented
|
||||||
|
|
||||||
|
## Development Commands
|
||||||
|
|
||||||
|
### Go Implementation (Primary)
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Build the application
|
||||||
|
cd go && go build -o mail2couch .
|
||||||
|
|
||||||
|
# Run the application with automatic config discovery
|
||||||
|
cd go && ./mail2couch
|
||||||
|
|
||||||
|
# Run with specific config file
|
||||||
|
cd go && ./mail2couch -config /path/to/config.json
|
||||||
|
|
||||||
|
# Run with message limit (useful for large mailboxes)
|
||||||
|
cd go && ./mail2couch -max-messages 100
|
||||||
|
|
||||||
|
# Run with both config and message limit
|
||||||
|
cd go && ./mail2couch -config /path/to/config.json -max-messages 50
|
||||||
|
|
||||||
|
# Run linting/static analysis
|
||||||
|
cd go && go vet ./...
|
||||||
|
|
||||||
|
# Run integration tests with Podman containers
|
||||||
|
cd test && ./run-tests.sh
|
||||||
|
|
||||||
|
# Run specialized tests
|
||||||
|
cd test && ./test-wildcard-patterns.sh
|
||||||
|
cd test && ./test-incremental-sync.sh
|
||||||
|
|
||||||
|
# Run unit tests (none currently implemented)
|
||||||
|
cd go && go test ./...
|
||||||
|
|
||||||
|
# Check dependencies
|
||||||
|
cd go && go mod tidy
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Core Components
|
||||||
|
|
||||||
|
1. **Configuration (`config/`)**: JSON-based configuration system
|
||||||
|
- Supports multiple mail sources with filtering options
|
||||||
|
- CouchDB connection settings
|
||||||
|
- Each source can have folder and message filters
|
||||||
|
|
||||||
|
2. **Mail Handling (`mail/`)**: IMAP client implementation
|
||||||
|
- Uses `github.com/emersion/go-imap/v2` for IMAP operations
|
||||||
|
- Supports TLS connections
|
||||||
|
- Currently only lists mailboxes (backup functionality not yet implemented)
|
||||||
|
|
||||||
|
3. **CouchDB Integration (`couch/`)**: Database operations
|
||||||
|
- Uses `github.com/go-kivik/kivik/v4` as CouchDB driver
|
||||||
|
- Handles database creation and document management
|
||||||
|
- Defines `MailDocument` structure for email storage
|
||||||
|
|
||||||
|
### Configuration Structure
|
||||||
|
|
||||||
|
The application uses `config.json` for configuration with the following structure:
|
||||||
|
- `couchDb`: Database connection settings (URL, credentials)
|
||||||
|
- `mailSources`: Array of mail sources with individual settings:
|
||||||
|
- Protocol support (currently only IMAP)
|
||||||
|
- Connection details (host, port, credentials)
|
||||||
|
- `mode`: Either "sync" or "archive" (defaults to "archive" if not specified)
|
||||||
|
- **sync**: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB)
|
||||||
|
- **archive**: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents)
|
||||||
|
- Filtering options for folders and messages with wildcard support
|
||||||
|
- Enable/disable per source
|
||||||
|
|
||||||
|
### Configuration File Discovery
|
||||||
|
|
||||||
|
The application automatically searches for configuration files in the following order:
|
||||||
|
1. Path specified by `-config` command line flag
|
||||||
|
2. `./config.json` (current working directory)
|
||||||
|
3. `./config/config.json` (config subdirectory)
|
||||||
|
4. `~/.config/mail2couch/config.json` (user XDG config directory)
|
||||||
|
5. `~/.mail2couch.json` (user home directory)
|
||||||
|
|
||||||
|
This design ensures the same `config.json` format will work for both Go and Rust implementations.
|
||||||
|
|
||||||
|
### Current Implementation Status
|
||||||
|
|
||||||
|
- ✅ Configuration loading with automatic file discovery
|
||||||
|
- ✅ Command line flag support for config file path
|
||||||
|
- ✅ Per-account CouchDB database creation and management
|
||||||
|
- ✅ IMAP connection and mailbox listing
|
||||||
|
- ✅ Build error fixes
|
||||||
|
- ✅ Email message retrieval framework (with placeholder data)
|
||||||
|
- ✅ Email storage to CouchDB framework with native attachments
|
||||||
|
- ✅ Folder filtering logic with wildcard support (`*`, `?`, `[abc]` patterns)
|
||||||
|
- ✅ Date filtering support
|
||||||
|
- ✅ Keyword filtering support (subject, sender, recipient keywords)
|
||||||
|
- ✅ Duplicate detection and prevention
|
||||||
|
- ✅ Sync vs Archive mode implementation
|
||||||
|
- ✅ CouchDB attachment storage for email attachments
|
||||||
|
- ✅ Real IMAP message parsing with go-message library
|
||||||
|
- ✅ Full message body and attachment handling with MIME multipart support
|
||||||
|
- ✅ Command line argument support (--max-messages flag)
|
||||||
|
- ✅ Per-account CouchDB databases for better organization
|
||||||
|
- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking
|
||||||
|
- ❌ Rust implementation
|
||||||
|
|
||||||
|
### Key Dependencies
|
||||||
|
|
||||||
|
- `github.com/emersion/go-imap/v2`: IMAP client library
|
||||||
|
- `github.com/go-kivik/kivik/v4`: CouchDB client library
|
||||||
|
|
||||||
|
### Incremental Sync Implementation
|
||||||
|
|
||||||
|
The application implements intelligent incremental synchronization to avoid re-processing messages:
|
||||||
|
|
||||||
|
- **Sync Metadata Storage**: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed
|
||||||
|
- **IMAP SEARCH Integration**: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages
|
||||||
|
- **Per-Mailbox Tracking**: Sync state is tracked independently for each mailbox in each account
|
||||||
|
- **Fallback Behavior**: Gracefully falls back to fetching recent messages if IMAP SEARCH fails
|
||||||
|
- **First Sync Handling**: Initial sync can use config `since` date or perform full sync
|
||||||
|
|
||||||
|
Sync metadata documents are stored in CouchDB with ID format: `sync_metadata_{mailbox}` and include:
|
||||||
|
- `lastSyncTime`: When this mailbox was last successfully synced
|
||||||
|
- `lastMessageUID`: Highest UID processed in the last sync
|
||||||
|
- `messageCount`: Number of messages processed in the last sync
|
||||||
|
|
||||||
|
### Development Notes
|
||||||
|
|
||||||
|
- The main entry point is `main.go` which orchestrates the configuration loading, CouchDB setup, and mail source processing
|
||||||
|
- Each mail source gets its own CouchDB database named using `GenerateAccountDBName()` function with `m2c_` prefix
|
||||||
|
- Each mail source is processed sequentially with proper error handling
|
||||||
|
- The application uses real IMAP message parsing with go-message library for full email processing
|
||||||
|
- Message filtering by folder (wildcard patterns), date (since), and keywords is implemented
|
||||||
|
- Duplicate detection prevents re-storing existing messages
|
||||||
|
- Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account
|
||||||
|
- Email attachments are stored as native CouchDB attachments linked to the email document
|
||||||
|
- Comprehensive test environment with Podman containers and automated test scripts
|
||||||
|
- The application uses automatic config file discovery as documented above
|
||||||
|
|
||||||
|
### Next Steps
|
||||||
|
|
||||||
|
The following enhancements could further improve the implementation:
|
||||||
|
|
||||||
|
1. **Error Recovery**: Add retry logic for network failures and partial sync recovery
|
||||||
|
2. **Performance Optimization**: Add batch operations for better CouchDB insertion performance
|
||||||
|
3. **Unit Testing**: Add comprehensive unit tests for all major components
|
||||||
|
4. **Advanced Filtering**: Add support for more complex filter expressions and regex patterns
|
||||||
|
5. **Monitoring**: Add metrics and logging for production deployment
|
||||||
|
6. **Configuration Validation**: Enhanced validation for configuration files
|
||||||
|
7. **Multi-threading**: Parallel processing of multiple mailboxes or accounts
|
||||||
|
|
||||||
|
## Development Guidelines
|
||||||
|
|
||||||
|
### Code Quality and Standards
|
||||||
|
- All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly
|
||||||
102
FOLDER_PATTERNS.md
Normal file
102
FOLDER_PATTERNS.md
Normal file
|
|
@ -0,0 +1,102 @@
|
||||||
|
# Folder Pattern Matching in mail2couch
|
||||||
|
|
||||||
|
mail2couch supports powerful wildcard patterns for selecting which folders to process. This allows flexible configuration for different mail backup scenarios.
|
||||||
|
|
||||||
|
## Pattern Syntax
|
||||||
|
|
||||||
|
The folder filtering uses Go's `filepath.Match` syntax, which supports:
|
||||||
|
|
||||||
|
- `*` matches any sequence of characters (including none)
|
||||||
|
- `?` matches any single character
|
||||||
|
- `[abc]` matches any character within the brackets
|
||||||
|
- `[a-z]` matches any character in the range
|
||||||
|
- `\` escapes special characters
|
||||||
|
|
||||||
|
## Special Cases
|
||||||
|
|
||||||
|
- `"*"` in the include list means **ALL available folders** will be processed
|
||||||
|
- Empty include list with exclude patterns will process all folders except excluded ones
|
||||||
|
- Exact string matching is supported for backwards compatibility
|
||||||
|
|
||||||
|
## Examples
|
||||||
|
|
||||||
|
### Include All Folders
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Drafts", "Trash", "Spam"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
This processes all folders except Drafts, Trash, and Spam.
|
||||||
|
|
||||||
|
### Work-Related Folders Only
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work*", "Projects*", "INBOX"],
|
||||||
|
"exclude": ["*Temp*", "*Draft*"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
This includes folders starting with "Work" or "Projects", plus INBOX, but excludes any folder containing "Temp" or "Draft".
|
||||||
|
|
||||||
|
### Archive Patterns
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Archive*", "*Important*", "INBOX"],
|
||||||
|
"exclude": ["*Temp"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
This includes folders starting with "Archive", any folder containing "Important", and INBOX, excluding temporary folders.
|
||||||
|
|
||||||
|
### Specific Folders Only
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent", "Important"],
|
||||||
|
"exclude": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
This processes only the exact folders: INBOX, Sent, and Important.
|
||||||
|
|
||||||
|
### Subfolder Patterns
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work/*", "Personal/*"],
|
||||||
|
"exclude": ["*/Drafts"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
This includes all subfolders under Work and Personal, but excludes any Drafts subfolder.
|
||||||
|
|
||||||
|
## Folder Hierarchy
|
||||||
|
|
||||||
|
Different IMAP servers use different separators for folder hierarchies:
|
||||||
|
- Most servers use `/` (e.g., `Work/Projects`, `Archive/2024`)
|
||||||
|
- Some use `.` (e.g., `Work.Projects`, `Archive.2024`)
|
||||||
|
|
||||||
|
The patterns work with whatever separator your IMAP server uses.
|
||||||
|
|
||||||
|
## Common Use Cases
|
||||||
|
|
||||||
|
1. **Corporate Email**: `["*"]` with exclude `["Drafts", "Trash", "Spam"]` for complete backup
|
||||||
|
2. **Selective Backup**: `["INBOX", "Sent", "Important"]` for essential folders only
|
||||||
|
3. **Project-based**: `["Project*", "Client*"]` to backup work-related folders
|
||||||
|
4. **Archive Mode**: `["Archive*", "*Important*"]` for long-term storage
|
||||||
|
5. **Sync Mode**: `["INBOX"]` for real-time synchronization
|
||||||
|
|
||||||
|
## Message Origin Tracking
|
||||||
|
|
||||||
|
All messages stored in CouchDB include a `mailbox` field that records the original folder name. This ensures you can always identify which folder a message came from, regardless of how it was selected by the folder filter.
|
||||||
|
|
||||||
|
## Performance Considerations
|
||||||
|
|
||||||
|
- Using `"*"` processes all folders, which may be slow for accounts with many folders
|
||||||
|
- Specific folder names are faster than wildcard patterns
|
||||||
|
- Consider using exclude patterns to filter out large, unimportant folders like Trash or Spam
|
||||||
388
README.md
388
README.md
|
|
@ -1,5 +1,389 @@
|
||||||
# mail2couch
|
# mail2couch
|
||||||
|
|
||||||
A utility to back up mail from various sources to couchdb
|
A powerful email backup utility that synchronizes mail from IMAP accounts to CouchDB databases with intelligent incremental sync, comprehensive filtering, and native attachment support.
|
||||||
|
|
||||||
At least two implementations will be available, on in Rust and one in Go.
|
## Features
|
||||||
|
|
||||||
|
### Core Functionality
|
||||||
|
- **IMAP Email Backup**: Connect to any IMAP server (Gmail, Outlook, self-hosted)
|
||||||
|
- **CouchDB Storage**: Store emails as JSON documents with native CouchDB attachments
|
||||||
|
- **Incremental Sync**: Efficiently sync only new messages using IMAP SEARCH with timestamp tracking
|
||||||
|
- **Per-Account Databases**: Each mail source gets its own CouchDB database for better organization
|
||||||
|
- **Duplicate Prevention**: Automatic detection and prevention of duplicate message storage
|
||||||
|
|
||||||
|
### Sync Modes
|
||||||
|
- **Archive Mode**: Preserve all messages ever seen, even if deleted from mail server (default)
|
||||||
|
- **Sync Mode**: Maintain 1-to-1 relationship with mail server (removes deleted messages from CouchDB)
|
||||||
|
|
||||||
|
### Advanced Filtering
|
||||||
|
- **Wildcard Folder Patterns**: Use `*`, `?`, `[abc]` patterns for flexible folder selection
|
||||||
|
- **Keyword Filtering**: Filter messages by keywords in subjects, senders, or recipients
|
||||||
|
- **Date Filtering**: Process only messages since a specific date
|
||||||
|
- **Include/Exclude Logic**: Combine multiple filter types for precise control
|
||||||
|
|
||||||
|
### Message Processing
|
||||||
|
- **Full MIME Support**: Parse multipart messages, HTML/plain text, and embedded content
|
||||||
|
- **Native Attachments**: Store email attachments as CouchDB native attachments with compression
|
||||||
|
- **Complete Headers**: Preserve all email headers and metadata
|
||||||
|
- **UTF-8 Support**: Handle international characters and special content
|
||||||
|
|
||||||
|
### Operational Features
|
||||||
|
- **Automatic Config Discovery**: Finds configuration files in standard locations
|
||||||
|
- **Command Line Control**: Override settings with `--max-messages` and `--config` flags
|
||||||
|
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
|
||||||
|
- **Error Resilience**: Graceful handling of network issues and server problems
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Installation
|
||||||
|
|
||||||
|
1. **Install dependencies**:
|
||||||
|
```bash
|
||||||
|
# Go 1.21+ required
|
||||||
|
go version
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Clone and build**:
|
||||||
|
```bash
|
||||||
|
git clone <repository-url>
|
||||||
|
cd mail2couch/go
|
||||||
|
go build -o mail2couch .
|
||||||
|
```
|
||||||
|
|
||||||
|
### Basic Usage
|
||||||
|
|
||||||
|
1. **Create configuration file** (`config.json`):
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@gmail.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Run mail2couch**:
|
||||||
|
```bash
|
||||||
|
./mail2couch
|
||||||
|
```
|
||||||
|
|
||||||
|
The application will:
|
||||||
|
- Create a CouchDB database named `m2c_personal_gmail`
|
||||||
|
- Sync all folders except Trash and Spam
|
||||||
|
- Store messages with native attachments
|
||||||
|
- Track sync state for efficient incremental updates
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
### Configuration File Discovery
|
||||||
|
|
||||||
|
mail2couch automatically searches for configuration files in this order:
|
||||||
|
1. Path specified by `--config` flag
|
||||||
|
2. `./config.json` (current directory)
|
||||||
|
3. `./config/config.json` (config subdirectory)
|
||||||
|
4. `~/.config/mail2couch/config.json` (user config directory)
|
||||||
|
5. `~/.mail2couch.json` (user home directory)
|
||||||
|
|
||||||
|
### Command Line Options
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./mail2couch [options]
|
||||||
|
|
||||||
|
Options:
|
||||||
|
--config PATH Specify configuration file path
|
||||||
|
--max-messages N Limit messages processed per mailbox per run (0 = unlimited)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Folder Pattern Examples
|
||||||
|
|
||||||
|
| Pattern | Description | Matches |
|
||||||
|
|---------|-------------|---------|
|
||||||
|
| `"*"` | All folders | `INBOX`, `Sent`, `Work/Projects`, etc. |
|
||||||
|
| `"INBOX"` | Exact match | `INBOX` only |
|
||||||
|
| `"Work*"` | Prefix match | `Work`, `Work/Projects`, `WorkStuff` |
|
||||||
|
| `"*/Archive"` | Suffix match | `Personal/Archive`, `Work/Archive` |
|
||||||
|
| `"Work/*"` | Subfolder match | `Work/Projects`, `Work/Clients` |
|
||||||
|
|
||||||
|
### Keyword Filtering Examples
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"messageFilter": {
|
||||||
|
"subjectKeywords": ["urgent", "meeting", "invoice"],
|
||||||
|
"senderKeywords": ["@company.com", "noreply@"],
|
||||||
|
"recipientKeywords": ["team@", "support@"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Advanced Configuration Examples
|
||||||
|
|
||||||
|
See the [example configurations](#example-configurations) section below for detailed configuration scenarios.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
A comprehensive test environment is included with Podman containers:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd test
|
||||||
|
|
||||||
|
# Quick automated testing (recommended)
|
||||||
|
./run-tests.sh # Complete integration test with automatic cleanup
|
||||||
|
|
||||||
|
# Specialized feature testing
|
||||||
|
./test-wildcard-patterns.sh # Test folder pattern matching
|
||||||
|
./test-incremental-sync.sh # Test incremental synchronization
|
||||||
|
|
||||||
|
# Manual testing environment
|
||||||
|
./start-test-env.sh # Start persistent test environment
|
||||||
|
# ... manual testing with various configurations ...
|
||||||
|
./stop-test-env.sh # Clean up when done
|
||||||
|
```
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
### Database Structure
|
||||||
|
- **Per-Account Databases**: Each mail source creates its own CouchDB database with `m2c_` prefix
|
||||||
|
- **Message Documents**: Each email becomes a CouchDB document with metadata
|
||||||
|
- **Native Attachments**: Email attachments stored as CouchDB attachments (compressed)
|
||||||
|
- **Sync Metadata**: Tracks incremental sync state per mailbox
|
||||||
|
|
||||||
|
### Document Structure
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"_id": "INBOX_12345",
|
||||||
|
"sourceUid": "12345",
|
||||||
|
"mailbox": "INBOX",
|
||||||
|
"from": ["sender@example.com"],
|
||||||
|
"to": ["recipient@example.com"],
|
||||||
|
"subject": "Sample Email",
|
||||||
|
"date": "2024-01-15T10:30:00Z",
|
||||||
|
"body": "Email content...",
|
||||||
|
"headers": {"Content-Type": ["text/plain"]},
|
||||||
|
"storedAt": "2024-01-15T10:35:00Z",
|
||||||
|
"docType": "mail",
|
||||||
|
"hasAttachments": true,
|
||||||
|
"_attachments": {
|
||||||
|
"document.pdf": {
|
||||||
|
"content_type": "application/pdf",
|
||||||
|
"length": 54321
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
## Example Configurations
|
||||||
|
|
||||||
|
### Simple Configuration
|
||||||
|
Basic setup for a single Gmail account:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@gmail.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Advanced Multi-Account Configuration
|
||||||
|
Complex setup with multiple accounts, filtering, and different sync modes:
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "https://your-couchdb.example.com:5984",
|
||||||
|
"user": "backup_user",
|
||||||
|
"password": "secure_password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Work Email",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "outlook.office365.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "you@company.com",
|
||||||
|
"password": "app-password",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Deleted Items", "Junk Email", "Drafts"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2023-01-01",
|
||||||
|
"subjectKeywords": ["project", "meeting", "urgent"],
|
||||||
|
"senderKeywords": ["@company.com", "@client.com"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "personal@gmail.com",
|
||||||
|
"password": "gmail-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Important", "Work/*", "Personal/*"],
|
||||||
|
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"recipientKeywords": ["family@", "personal@"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Self-Hosted Mail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "mail.yourdomain.com",
|
||||||
|
"port": 143,
|
||||||
|
"user": "admin@yourdomain.com",
|
||||||
|
"password": "mail-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Archive/*", "Projects/*"],
|
||||||
|
"exclude": ["*/Drafts", "Trash"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2023-06-01",
|
||||||
|
"subjectKeywords": ["invoice", "receipt", "statement"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Legacy Account",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "legacy.mailserver.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "old@account.com",
|
||||||
|
"password": "legacy-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Configuration Options Reference
|
||||||
|
|
||||||
|
#### CouchDB Configuration
|
||||||
|
- `url`: CouchDB server URL with protocol and port
|
||||||
|
- `user`: CouchDB username with database access
|
||||||
|
- `password`: CouchDB password
|
||||||
|
|
||||||
|
#### Mail Source Configuration
|
||||||
|
- `name`: Descriptive name (used for database naming)
|
||||||
|
- `enabled`: Boolean to enable/disable this source
|
||||||
|
- `protocol`: Only `"imap"` currently supported
|
||||||
|
- `host`: IMAP server hostname
|
||||||
|
- `port`: IMAP port (993 for TLS, 143 for plain, 3143 for testing)
|
||||||
|
- `user`: Email account username
|
||||||
|
- `password`: Email account password (use app passwords for Gmail/Outlook)
|
||||||
|
- `mode`: `"sync"` (mirror server) or `"archive"` (preserve all messages)
|
||||||
|
|
||||||
|
#### Folder Filter Configuration
|
||||||
|
- `include`: Array of folder patterns to process (empty = all folders)
|
||||||
|
- `exclude`: Array of folder patterns to skip
|
||||||
|
|
||||||
|
#### Message Filter Configuration
|
||||||
|
- `since`: Date string (YYYY-MM-DD) to process messages from
|
||||||
|
- `subjectKeywords`: Array of keywords that must appear in subject line
|
||||||
|
- `senderKeywords`: Array of keywords that must appear in sender addresses
|
||||||
|
- `recipientKeywords`: Array of keywords that must appear in recipient addresses
|
||||||
|
|
||||||
|
## Production Deployment
|
||||||
|
|
||||||
|
### Security Considerations
|
||||||
|
- Use app passwords instead of account passwords
|
||||||
|
- Store configuration files with restricted permissions (600)
|
||||||
|
- Use HTTPS for CouchDB connections in production
|
||||||
|
- Consider encrypting sensitive configuration data
|
||||||
|
|
||||||
|
### Monitoring and Maintenance
|
||||||
|
- Review sync metadata documents for sync health
|
||||||
|
- Monitor CouchDB database sizes and compaction
|
||||||
|
- Set up log rotation for application output
|
||||||
|
- Schedule regular backups of CouchDB databases
|
||||||
|
|
||||||
|
### Performance Tuning
|
||||||
|
- Use `--max-messages` to limit processing load
|
||||||
|
- Run during off-peak hours for large initial syncs
|
||||||
|
- Monitor IMAP server rate limits and connection limits
|
||||||
|
- Consider running multiple instances for different accounts
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Common Issues
|
||||||
|
|
||||||
|
**Connection Errors**:
|
||||||
|
- Verify IMAP server settings and credentials
|
||||||
|
- Check firewall and network connectivity
|
||||||
|
- Ensure correct ports (993 for TLS, 143 for plain)
|
||||||
|
|
||||||
|
**Authentication Failures**:
|
||||||
|
- Use app passwords for Gmail, Outlook, and other providers
|
||||||
|
- Enable "Less Secure Apps" if required by provider
|
||||||
|
- Verify account permissions and 2FA settings
|
||||||
|
|
||||||
|
**Sync Issues**:
|
||||||
|
- Check CouchDB connectivity and permissions
|
||||||
|
- Review sync metadata documents for error states
|
||||||
|
- Verify folder names and patterns match server structure
|
||||||
|
|
||||||
|
**Performance Problems**:
|
||||||
|
- Use date filtering (`since`) for large mailboxes
|
||||||
|
- Implement `--max-messages` limits for initial syncs
|
||||||
|
- Monitor server-side rate limiting
|
||||||
|
|
||||||
|
For detailed troubleshooting, see the [test environment documentation](test/README.md).
|
||||||
|
|
||||||
|
## Contributing
|
||||||
|
|
||||||
|
This project welcomes contributions! Please see [CLAUDE.md](CLAUDE.md) for development setup and architecture details.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
[License information to be added]
|
||||||
|
|
|
||||||
47
TODO.md
Normal file
47
TODO.md
Normal file
|
|
@ -0,0 +1,47 @@
|
||||||
|
# mail2couch TODO and Feature Requests
|
||||||
|
|
||||||
|
## Planned Features
|
||||||
|
|
||||||
|
### Keyword Filtering for Messages
|
||||||
|
|
||||||
|
Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration.
|
||||||
|
|
||||||
|
**Proposed Configuration Extension:**
|
||||||
|
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01",
|
||||||
|
"subjectKeywords": ["urgent", "important", "meeting"],
|
||||||
|
"senderKeywords": ["@company.com", "notifications"],
|
||||||
|
"recipientKeywords": ["team@company.com", "all@"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Implementation Details:**
|
||||||
|
|
||||||
|
- `subjectKeywords`: Array of keywords to match in email subject lines
|
||||||
|
- `senderKeywords`: Array of keywords to match in sender email addresses or names
|
||||||
|
- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names
|
||||||
|
- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns
|
||||||
|
- Case-insensitive matching by default
|
||||||
|
- Support for simple wildcards or regex patterns
|
||||||
|
|
||||||
|
**Use Cases:**
|
||||||
|
|
||||||
|
1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords
|
||||||
|
2. **Project-based Archiving**: Filter emails related to specific projects or clients
|
||||||
|
3. **Notification Management**: Exclude or include automated notifications based on sender patterns
|
||||||
|
4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders
|
||||||
|
|
||||||
|
**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails.
|
||||||
|
|
||||||
|
## Other Planned Improvements
|
||||||
|
|
||||||
|
1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content
|
||||||
|
2. **Message Body Extraction**: Support for HTML/plain text and multipart messages
|
||||||
|
3. **Attachment Handling**: Optional support for email attachments
|
||||||
|
4. **Batch Operations**: Improve CouchDB insertion performance
|
||||||
|
5. **Error Recovery**: Retry logic and partial sync recovery
|
||||||
|
6. **Testing**: Comprehensive unit test coverage
|
||||||
78
config-advanced.json
Normal file
78
config-advanced.json
Normal file
|
|
@ -0,0 +1,78 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "https://your-couchdb.example.com:5984",
|
||||||
|
"user": "backup_user",
|
||||||
|
"password": "secure_password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Work Email",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "outlook.office365.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "you@company.com",
|
||||||
|
"password": "app-password",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Deleted Items", "Junk Email", "Drafts"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2023-01-01",
|
||||||
|
"subjectKeywords": ["project", "meeting", "urgent"],
|
||||||
|
"senderKeywords": ["@company.com", "@client.com"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "personal@gmail.com",
|
||||||
|
"password": "gmail-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Important", "Work/*", "Personal/*"],
|
||||||
|
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "*Temp*"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"recipientKeywords": ["family@", "personal@"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Self-Hosted Mail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "mail.yourdomain.com",
|
||||||
|
"port": 143,
|
||||||
|
"user": "admin@yourdomain.com",
|
||||||
|
"password": "mail-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Archive/*", "Projects/*"],
|
||||||
|
"exclude": ["*/Drafts", "Trash"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2023-06-01",
|
||||||
|
"subjectKeywords": ["invoice", "receipt", "statement"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Legacy Account",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "legacy.mailserver.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "old@account.com",
|
||||||
|
"password": "legacy-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
90
config-providers.json
Normal file
90
config-providers.json
Normal file
|
|
@ -0,0 +1,90 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Gmail Account",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@gmail.com",
|
||||||
|
"password": "your-16-character-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["[Gmail]/Trash", "[Gmail]/Spam", "[Gmail]/Drafts"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Outlook 365",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "outlook.office365.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "you@outlook.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent Items", "Archive"],
|
||||||
|
"exclude": ["Deleted Items", "Junk Email"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2023-06-01"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Yahoo Mail",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.mail.yahoo.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@yahoo.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": ["Trash", "Spam"]
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "iCloud Mail",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.mail.me.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@icloud.com",
|
||||||
|
"password": "your-app-specific-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent Messages"],
|
||||||
|
"exclude": ["Deleted Messages", "Junk"]
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Custom IMAP Server",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "mail.example.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "username@example.com",
|
||||||
|
"password": "password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": ["Trash"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
26
config-simple.json
Normal file
26
config-simple.json
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@gmail.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
41
config.json
Normal file
41
config.json
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Personal Gmail",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.gmail.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "your-email@gmail.com",
|
||||||
|
"password": "your-app-password",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": ["Spam", "Trash"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"since": "2024-01-01"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Work Account",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "imap.work.com",
|
||||||
|
"port": 993,
|
||||||
|
"user": "user@work.com",
|
||||||
|
"password": "password",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": [],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
153
go/config/config.go
Normal file
153
go/config/config.go
Normal file
|
|
@ -0,0 +1,153 @@
|
||||||
|
package config
|
||||||
|
|
||||||
|
import (
|
||||||
|
"encoding/json"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"os"
|
||||||
|
"path/filepath"
|
||||||
|
)
|
||||||
|
|
||||||
|
type Config struct {
|
||||||
|
CouchDb CouchDbConfig `json:"couchDb"`
|
||||||
|
MailSources []MailSource `json:"mailSources"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type CouchDbConfig struct {
|
||||||
|
URL string `json:"url"`
|
||||||
|
User string `json:"user"`
|
||||||
|
Password string `json:"password"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type MailSource struct {
|
||||||
|
Name string `json:"name"`
|
||||||
|
Enabled bool `json:"enabled"`
|
||||||
|
Protocol string `json:"protocol"`
|
||||||
|
Host string `json:"host"`
|
||||||
|
Port int `json:"port"`
|
||||||
|
User string `json:"user"`
|
||||||
|
Password string `json:"password"`
|
||||||
|
Mode string `json:"mode"` // "sync" or "archive"
|
||||||
|
FolderFilter FolderFilter `json:"folderFilter"`
|
||||||
|
MessageFilter MessageFilter `json:"messageFilter"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type FolderFilter struct {
|
||||||
|
Include []string `json:"include"`
|
||||||
|
Exclude []string `json:"exclude"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type MessageFilter struct {
|
||||||
|
Since string `json:"since,omitempty"`
|
||||||
|
SubjectKeywords []string `json:"subjectKeywords,omitempty"` // Filter by keywords in subject
|
||||||
|
SenderKeywords []string `json:"senderKeywords,omitempty"` // Filter by keywords in sender addresses
|
||||||
|
RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses
|
||||||
|
}
|
||||||
|
|
||||||
|
func LoadConfig(path string) (*Config, error) {
|
||||||
|
configFile, err := os.Open(path)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
defer configFile.Close()
|
||||||
|
|
||||||
|
var config Config
|
||||||
|
jsonParser := json.NewDecoder(configFile)
|
||||||
|
if err = jsonParser.Decode(&config); err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Validate and set defaults for mail sources
|
||||||
|
for i := range config.MailSources {
|
||||||
|
source := &config.MailSources[i]
|
||||||
|
if source.Mode == "" {
|
||||||
|
source.Mode = "archive" // Default to archive mode
|
||||||
|
}
|
||||||
|
if source.Mode != "sync" && source.Mode != "archive" {
|
||||||
|
return nil, fmt.Errorf("invalid mode '%s' for mail source '%s': must be 'sync' or 'archive'", source.Mode, source.Name)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return &config, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsSyncMode returns true if the mail source is in sync mode
|
||||||
|
func (ms *MailSource) IsSyncMode() bool {
|
||||||
|
return ms.Mode == "sync"
|
||||||
|
}
|
||||||
|
|
||||||
|
// IsArchiveMode returns true if the mail source is in archive mode
|
||||||
|
func (ms *MailSource) IsArchiveMode() bool {
|
||||||
|
return ms.Mode == "archive" || ms.Mode == "" // Default to archive
|
||||||
|
}
|
||||||
|
|
||||||
|
// CommandLineArgs holds parsed command line arguments
|
||||||
|
type CommandLineArgs struct {
|
||||||
|
ConfigPath string
|
||||||
|
MaxMessages int
|
||||||
|
}
|
||||||
|
|
||||||
|
// ParseCommandLine parses command line arguments
|
||||||
|
func ParseCommandLine() *CommandLineArgs {
|
||||||
|
configFlag := flag.String("config", "", "Path to configuration file")
|
||||||
|
maxMessagesFlag := flag.Int("max-messages", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
return &CommandLineArgs{
|
||||||
|
ConfigPath: *configFlag,
|
||||||
|
MaxMessages: *maxMessagesFlag,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// FindConfigFile searches for config.json in the following order:
|
||||||
|
// 1. Path specified by -config flag
|
||||||
|
// 2. ./config.json (current directory)
|
||||||
|
// 3. ~/.config/mail2couch/config.json (user config directory)
|
||||||
|
// 4. ~/.mail2couch.json (user home directory)
|
||||||
|
func FindConfigFile(args *CommandLineArgs) (string, error) {
|
||||||
|
if args.ConfigPath != "" {
|
||||||
|
if _, err := os.Stat(args.ConfigPath); err == nil {
|
||||||
|
return args.ConfigPath, nil
|
||||||
|
}
|
||||||
|
return "", fmt.Errorf("specified config file not found: %s", args.ConfigPath)
|
||||||
|
}
|
||||||
|
|
||||||
|
// List of possible config file locations in order of preference
|
||||||
|
candidates := []string{
|
||||||
|
"config.json", // Current directory
|
||||||
|
"config/config.json", // Config subdirectory
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add user directory paths
|
||||||
|
if homeDir, err := os.UserHomeDir(); err == nil {
|
||||||
|
candidates = append(candidates,
|
||||||
|
filepath.Join(homeDir, ".config", "mail2couch", "config.json"),
|
||||||
|
filepath.Join(homeDir, ".mail2couch.json"),
|
||||||
|
)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Try each candidate location
|
||||||
|
for _, candidate := range candidates {
|
||||||
|
if _, err := os.Stat(candidate); err == nil {
|
||||||
|
return candidate, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return "", fmt.Errorf("no configuration file found. Searched locations: %v", candidates)
|
||||||
|
}
|
||||||
|
|
||||||
|
// LoadConfigWithDiscovery loads configuration using automatic file discovery
|
||||||
|
func LoadConfigWithDiscovery(args *CommandLineArgs) (*Config, error) {
|
||||||
|
configPath, err := FindConfigFile(args)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("Using configuration file: %s\n", configPath)
|
||||||
|
if args.MaxMessages > 0 {
|
||||||
|
fmt.Printf("Maximum messages per mailbox: %d\n", args.MaxMessages)
|
||||||
|
} else {
|
||||||
|
fmt.Printf("Maximum messages per mailbox: unlimited\n")
|
||||||
|
}
|
||||||
|
return LoadConfig(configPath)
|
||||||
|
}
|
||||||
374
go/couch/couch.go
Normal file
374
go/couch/couch.go
Normal file
|
|
@ -0,0 +1,374 @@
|
||||||
|
package couch
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"net/url"
|
||||||
|
"regexp"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/go-kivik/kivik/v4"
|
||||||
|
_ "github.com/go-kivik/kivik/v4/couchdb" // The CouchDB driver
|
||||||
|
"mail2couch/config"
|
||||||
|
"mail2couch/mail"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Client wraps the Kivik client
|
||||||
|
type Client struct {
|
||||||
|
*kivik.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// MailDocument represents an email message stored in CouchDB
|
||||||
|
type MailDocument struct {
|
||||||
|
ID string `json:"_id,omitempty"`
|
||||||
|
Rev string `json:"_rev,omitempty"`
|
||||||
|
Attachments map[string]AttachmentStub `json:"_attachments,omitempty"` // CouchDB attachments
|
||||||
|
SourceUID string `json:"sourceUid"` // Unique ID from the mail source (e.g., IMAP UID)
|
||||||
|
Mailbox string `json:"mailbox"` // Source mailbox name
|
||||||
|
From []string `json:"from"`
|
||||||
|
To []string `json:"to"`
|
||||||
|
Subject string `json:"subject"`
|
||||||
|
Date time.Time `json:"date"`
|
||||||
|
Body string `json:"body"`
|
||||||
|
Headers map[string][]string `json:"headers"`
|
||||||
|
StoredAt time.Time `json:"storedAt"` // When the document was stored
|
||||||
|
DocType string `json:"docType"` // Always "mail"
|
||||||
|
HasAttachments bool `json:"hasAttachments"` // Indicates if message has attachments
|
||||||
|
}
|
||||||
|
|
||||||
|
// AttachmentStub represents metadata for a CouchDB attachment
|
||||||
|
type AttachmentStub struct {
|
||||||
|
ContentType string `json:"content_type"`
|
||||||
|
Length int64 `json:"length,omitempty"`
|
||||||
|
Stub bool `json:"stub,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// SyncMetadata represents sync state information stored in CouchDB
|
||||||
|
type SyncMetadata struct {
|
||||||
|
ID string `json:"_id,omitempty"`
|
||||||
|
Rev string `json:"_rev,omitempty"`
|
||||||
|
DocType string `json:"docType"` // Always "sync_metadata"
|
||||||
|
Mailbox string `json:"mailbox"` // Mailbox name
|
||||||
|
LastSyncTime time.Time `json:"lastSyncTime"` // When this mailbox was last synced
|
||||||
|
LastMessageUID uint32 `json:"lastMessageUID"` // Highest UID processed in last sync
|
||||||
|
MessageCount int `json:"messageCount"` // Number of messages processed in last sync
|
||||||
|
UpdatedAt time.Time `json:"updatedAt"` // When this metadata was last updated
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewClient creates a new CouchDB client from the configuration
|
||||||
|
func NewClient(cfg *config.CouchDbConfig) (*Client, error) {
|
||||||
|
parsedURL, err := url.Parse(cfg.URL)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("invalid couchdb url: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
parsedURL.User = url.UserPassword(cfg.User, cfg.Password)
|
||||||
|
dsn := parsedURL.String()
|
||||||
|
|
||||||
|
client, err := kivik.New("couch", dsn)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return &Client{client}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// EnsureDB ensures that the configured database exists.
|
||||||
|
func (c *Client) EnsureDB(ctx context.Context, dbName string) error {
|
||||||
|
exists, err := c.DBExists(ctx, dbName)
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if !exists {
|
||||||
|
return c.CreateDB(ctx, dbName)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GenerateAccountDBName creates a CouchDB-compatible database name from account info
|
||||||
|
func GenerateAccountDBName(accountName, userEmail string) string {
|
||||||
|
// Use account name if available, otherwise fall back to email
|
||||||
|
name := accountName
|
||||||
|
if name == "" {
|
||||||
|
name = userEmail
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert to lowercase and replace invalid characters with underscores
|
||||||
|
name = strings.ToLower(name)
|
||||||
|
// CouchDB database names must match: ^[a-z][a-z0-9_$()+/-]*$
|
||||||
|
validName := regexp.MustCompile(`[^a-z0-9_$()+/-]`).ReplaceAllString(name, "_")
|
||||||
|
|
||||||
|
// Ensure it starts with a letter and add m2c prefix
|
||||||
|
if len(validName) > 0 && (validName[0] < 'a' || validName[0] > 'z') {
|
||||||
|
validName = "m2c_mail_" + validName
|
||||||
|
} else {
|
||||||
|
validName = "m2c_" + validName
|
||||||
|
}
|
||||||
|
|
||||||
|
return validName
|
||||||
|
}
|
||||||
|
|
||||||
|
// ConvertMessage converts an IMAP message to a MailDocument
|
||||||
|
func ConvertMessage(msg *mail.Message, mailbox string) *MailDocument {
|
||||||
|
docID := fmt.Sprintf("%s_%d", mailbox, msg.UID)
|
||||||
|
|
||||||
|
doc := &MailDocument{
|
||||||
|
ID: docID,
|
||||||
|
SourceUID: fmt.Sprintf("%d", msg.UID),
|
||||||
|
Mailbox: mailbox,
|
||||||
|
From: msg.From,
|
||||||
|
To: msg.To,
|
||||||
|
Subject: msg.Subject,
|
||||||
|
Date: msg.Date,
|
||||||
|
Body: msg.Body,
|
||||||
|
Headers: msg.Headers,
|
||||||
|
StoredAt: time.Now(),
|
||||||
|
DocType: "mail",
|
||||||
|
HasAttachments: len(msg.Attachments) > 0,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Don't add attachment metadata here - CouchDB will handle this when we store attachments
|
||||||
|
// We'll add the attachment metadata after successful document creation
|
||||||
|
|
||||||
|
return doc
|
||||||
|
}
|
||||||
|
|
||||||
|
// StoreMessage stores a mail message in CouchDB with attachments
|
||||||
|
func (c *Client) StoreMessage(ctx context.Context, dbName string, doc *MailDocument, msg *mail.Message) error {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check if document already exists
|
||||||
|
exists, err := c.DocumentExists(ctx, dbName, doc.ID)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to check if document exists: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if exists {
|
||||||
|
return nil // Document already exists, skip
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store the document first (without attachments)
|
||||||
|
rev, err := db.Put(ctx, doc.ID, doc)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to store document: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// If there are attachments, store them as CouchDB attachments
|
||||||
|
if msg != nil && len(msg.Attachments) > 0 {
|
||||||
|
currentRev := rev
|
||||||
|
for _, att := range msg.Attachments {
|
||||||
|
newRev, err := c.StoreAttachment(ctx, dbName, doc.ID, currentRev, att.Filename, att.ContentType, att.Content)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to store attachment %s: %w", att.Filename, err)
|
||||||
|
}
|
||||||
|
currentRev = newRev // Update revision for next attachment
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// StoreAttachment stores an attachment to an existing CouchDB document
|
||||||
|
func (c *Client) StoreAttachment(ctx context.Context, dbName, docID, rev, filename, contentType string, content []byte) (string, error) {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return "", db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
att := &kivik.Attachment{
|
||||||
|
Filename: filename,
|
||||||
|
ContentType: contentType,
|
||||||
|
Content: io.NopCloser(strings.NewReader(string(content))),
|
||||||
|
}
|
||||||
|
|
||||||
|
newRev, err := db.PutAttachment(ctx, docID, att, kivik.Rev(rev))
|
||||||
|
if err != nil {
|
||||||
|
return "", fmt.Errorf("failed to store attachment: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return newRev, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// StoreMessages stores multiple mail messages in CouchDB with their corresponding attachments
|
||||||
|
func (c *Client) StoreMessages(ctx context.Context, dbName string, docs []*MailDocument, messages []*mail.Message) error {
|
||||||
|
for i, doc := range docs {
|
||||||
|
var msg *mail.Message
|
||||||
|
if i < len(messages) {
|
||||||
|
msg = messages[i]
|
||||||
|
}
|
||||||
|
if err := c.StoreMessage(ctx, dbName, doc, msg); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DocumentExists checks if a document with the given ID already exists.
|
||||||
|
func (c *Client) DocumentExists(ctx context.Context, dbName, docID string) (bool, error) {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return false, db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
row := db.Get(ctx, docID)
|
||||||
|
return row.Err() == nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetAllMailDocumentIDs returns all mail document IDs from a database for a specific mailbox
|
||||||
|
func (c *Client) GetAllMailDocumentIDs(ctx context.Context, dbName, mailbox string) (map[string]bool, error) {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return nil, db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create a view query to get all document IDs for the specified mailbox
|
||||||
|
rows := db.AllDocs(ctx)
|
||||||
|
|
||||||
|
docIDs := make(map[string]bool)
|
||||||
|
for rows.Next() {
|
||||||
|
docID, err := rows.ID()
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
// Filter by mailbox prefix (documents are named like "INBOX_123")
|
||||||
|
if mailbox == "" || strings.HasPrefix(docID, mailbox+"_") {
|
||||||
|
docIDs[docID] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if rows.Err() != nil {
|
||||||
|
return nil, rows.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
return docIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeleteDocument removes a document from CouchDB
|
||||||
|
func (c *Client) DeleteDocument(ctx context.Context, dbName, docID string) error {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get the current revision
|
||||||
|
row := db.Get(ctx, docID)
|
||||||
|
if row.Err() != nil {
|
||||||
|
return row.Err() // Document doesn't exist or other error
|
||||||
|
}
|
||||||
|
|
||||||
|
var doc struct {
|
||||||
|
Rev string `json:"_rev"`
|
||||||
|
}
|
||||||
|
if err := row.ScanDoc(&doc); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete the document
|
||||||
|
_, err := db.Delete(ctx, docID, doc.Rev)
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
|
||||||
|
// SyncMailbox synchronizes a mailbox between mail server and CouchDB
|
||||||
|
// In sync mode: removes documents from CouchDB that are no longer in the mail account
|
||||||
|
// In archive mode: keeps all documents (no removal)
|
||||||
|
func (c *Client) SyncMailbox(ctx context.Context, dbName, mailbox string, currentMessageUIDs map[uint32]bool, syncMode bool) error {
|
||||||
|
if !syncMode {
|
||||||
|
return nil // Archive mode - don't remove anything
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get all existing document IDs for this mailbox from CouchDB
|
||||||
|
existingDocs, err := c.GetAllMailDocumentIDs(ctx, dbName, mailbox)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to get existing documents: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Find documents that should be removed (exist in CouchDB but not in mail account)
|
||||||
|
var toDelete []string
|
||||||
|
for docID := range existingDocs {
|
||||||
|
// Extract UID from document ID (format: "mailbox_uid")
|
||||||
|
parts := strings.Split(docID, "_")
|
||||||
|
if len(parts) < 2 {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
uidStr := parts[len(parts)-1]
|
||||||
|
uid := uint32(0)
|
||||||
|
if _, err := fmt.Sscanf(uidStr, "%d", &uid); err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// If this UID is not in the current mail account, mark for deletion
|
||||||
|
if !currentMessageUIDs[uid] {
|
||||||
|
toDelete = append(toDelete, docID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Delete documents that are no longer in the mail account
|
||||||
|
for _, docID := range toDelete {
|
||||||
|
if err := c.DeleteDocument(ctx, dbName, docID); err != nil {
|
||||||
|
return fmt.Errorf("failed to delete document %s: %w", docID, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(toDelete) > 0 {
|
||||||
|
fmt.Printf(" Sync mode: Removed %d documents no longer in mail account\n", len(toDelete))
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetSyncMetadata retrieves the sync metadata for a specific mailbox
|
||||||
|
func (c *Client) GetSyncMetadata(ctx context.Context, dbName, mailbox string) (*SyncMetadata, error) {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return nil, db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
metadataID := fmt.Sprintf("sync_metadata_%s", mailbox)
|
||||||
|
row := db.Get(ctx, metadataID)
|
||||||
|
if row.Err() != nil {
|
||||||
|
// If metadata doesn't exist, return nil (not an error for first sync)
|
||||||
|
return nil, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var metadata SyncMetadata
|
||||||
|
if err := row.ScanDoc(&metadata); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to scan sync metadata: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &metadata, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// StoreSyncMetadata stores or updates sync metadata for a mailbox
|
||||||
|
func (c *Client) StoreSyncMetadata(ctx context.Context, dbName string, metadata *SyncMetadata) error {
|
||||||
|
db := c.DB(dbName)
|
||||||
|
if db.Err() != nil {
|
||||||
|
return db.Err()
|
||||||
|
}
|
||||||
|
|
||||||
|
metadata.ID = fmt.Sprintf("sync_metadata_%s", metadata.Mailbox)
|
||||||
|
metadata.DocType = "sync_metadata"
|
||||||
|
metadata.UpdatedAt = time.Now()
|
||||||
|
|
||||||
|
// Check if metadata already exists to get current revision
|
||||||
|
existing, err := c.GetSyncMetadata(ctx, dbName, metadata.Mailbox)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to check existing sync metadata: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if existing != nil {
|
||||||
|
metadata.Rev = existing.Rev
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err = db.Put(ctx, metadata.ID, metadata)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to store sync metadata: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
16
go/go.mod
Normal file
16
go/go.mod
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
module mail2couch
|
||||||
|
|
||||||
|
go 1.24.4
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/emersion/go-imap/v2 v2.0.0-beta.5
|
||||||
|
github.com/go-kivik/kivik/v4 v4.4.0
|
||||||
|
)
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/emersion/go-message v0.18.1 // indirect
|
||||||
|
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 // indirect
|
||||||
|
github.com/google/uuid v1.6.0 // indirect
|
||||||
|
golang.org/x/net v0.25.0 // indirect
|
||||||
|
golang.org/x/sync v0.10.0 // indirect
|
||||||
|
)
|
||||||
65
go/go.sum
Normal file
65
go/go.sum
Normal file
|
|
@ -0,0 +1,65 @@
|
||||||
|
github.com/Masterminds/semver/v3 v3.2.1 h1:RN9w6+7QoMeJVGyfmbcgs28Br8cvmnucEXnY0rYXWg0=
|
||||||
|
github.com/Masterminds/semver/v3 v3.2.1/go.mod h1:qvl/7zhW3nngYb5+80sSMF+FG2BjYrf8m9wsX0PNOMQ=
|
||||||
|
github.com/cenkalti/backoff/v4 v4.1.3 h1:cFAlzYUlVYDysBEH2T5hyJZMh3+5+WCBvSnK6Q8UtC4=
|
||||||
|
github.com/cenkalti/backoff/v4 v4.1.3/go.mod h1:scbssz8iZGpm3xbr14ovlUdkxfGXNInqkPWOWmG2CLw=
|
||||||
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc h1:U9qPSI2PIWSS1VwoXQT9A3Wy9MM3WgvqSxFWenqJduM=
|
||||||
|
github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||||
|
github.com/emersion/go-imap/v2 v2.0.0-beta.5 h1:H3858DNmBuXyMK1++YrQIRdpKE1MwBc+ywBtg3n+0wA=
|
||||||
|
github.com/emersion/go-imap/v2 v2.0.0-beta.5/go.mod h1:BZTFHsS1hmgBkFlHqbxGLXk2hnRqTItUgwjSSCsYNAk=
|
||||||
|
github.com/emersion/go-message v0.18.1 h1:tfTxIoXFSFRwWaZsgnqS1DSZuGpYGzSmCZD8SK3QA2E=
|
||||||
|
github.com/emersion/go-message v0.18.1/go.mod h1:XpJyL70LwRvq2a8rVbHXikPgKj8+aI0kGdHlg16ibYA=
|
||||||
|
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43 h1:hH4PQfOndHDlpzYfLAAfl63E8Le6F2+EL/cdhlkyRJY=
|
||||||
|
github.com/emersion/go-sasl v0.0.0-20231106173351-e73c9f7bad43/go.mod h1:iL2twTeMvZnrg54ZoPDNfJaJaqy0xIQFuBdrLsmspwQ=
|
||||||
|
github.com/go-kivik/kivik/v4 v4.4.0 h1:1YMqNvRMIIC+CJUtyldD7c4Czl6SqdUcnbusCoFOTfk=
|
||||||
|
github.com/go-kivik/kivik/v4 v4.4.0/go.mod h1:DnPzIEO7CcLOqJNuqxuo7EMZeK4bPsEbUSSmAfi+tL4=
|
||||||
|
github.com/google/go-cmp v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
|
||||||
|
github.com/google/go-cmp v0.6.0/go.mod h1:17dUlkBOakJ0+DkrSSNjCkIjxS6bF9zb3elmeNGIjoY=
|
||||||
|
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||||
|
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||||
|
github.com/gopherjs/gopherjs v1.17.2 h1:fQnZVsXk8uxXIStYb0N4bGk7jeyTalG/wsZjQ25dO0g=
|
||||||
|
github.com/gopherjs/gopherjs v1.17.2/go.mod h1:pRRIvn/QzFLrKfvEz3qUuEhtE/zLCWfreZ6J5gM2i+k=
|
||||||
|
github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0 h1:nHoRIX8iXob3Y2kdt9KsjyIb7iApSvb3vgsd93xb5Ow=
|
||||||
|
github.com/icza/dyno v0.0.0-20230330125955-09f820a8d9c0/go.mod h1:c1tRKs5Tx7E2+uHGSyyncziFjvGpgv4H2HrqXeUQ/Uk=
|
||||||
|
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||||
|
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||||
|
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||||
|
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||||
|
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||||
|
gitlab.com/flimzy/testy v0.14.0 h1:2nZV4Wa1OSJb3rOKHh0GJqvvhtE03zT+sKnPCI0owfQ=
|
||||||
|
gitlab.com/flimzy/testy v0.14.0/go.mod h1:m3aGuwdXc+N3QgnH+2Ar2zf1yg0UxNdIaXKvC5SlfMk=
|
||||||
|
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||||
|
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
|
||||||
|
golang.org/x/mod v0.6.0-dev.0.20220419223038-86c51ed26bb4/go.mod h1:jJ57K6gSWd91VN4djpZkiMVwK6gcyfeH4XE8wZrZaV4=
|
||||||
|
golang.org/x/mod v0.8.0/go.mod h1:iBbtSCu2XBx23ZKBPSOrRkjjQPZFPuis4dIYUhu/chs=
|
||||||
|
golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||||
|
golang.org/x/net v0.0.0-20210226172049-e18ecbb05110/go.mod h1:m0MpNAwzfU5UDzcl9v0D8zg8gWTRqZa9RBIspLL5mdg=
|
||||||
|
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
|
||||||
|
golang.org/x/net v0.6.0/go.mod h1:2Tu9+aMcznHK/AK1HMvgo6xiTLG5rD5rZLDS+rp2Bjs=
|
||||||
|
golang.org/x/net v0.25.0 h1:d/OCCoBEUq33pjydKrGQhw7IlUPI2Oylr+8qLx49kac=
|
||||||
|
golang.org/x/net v0.25.0/go.mod h1:JkAGAh7GEvH74S6FOH42FLoXpXbE/aqXSrIQjXgsiwM=
|
||||||
|
golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.1.0/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||||
|
golang.org/x/sync v0.10.0 h1:3NQrjDixjgGwUOCaF8w2+VYHv0Ve/vGYSbdkTa98gmQ=
|
||||||
|
golang.org/x/sync v0.10.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||||
|
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||||
|
golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||||
|
golang.org/x/sys v0.0.0-20210615035016-665e8c7367d1/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||||
|
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
|
||||||
|
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
|
||||||
|
golang.org/x/term v0.5.0/go.mod h1:jMB1sMXY+tzblOD4FWmEbocvup2/aLOaQEp7JmGp78k=
|
||||||
|
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||||
|
golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
|
||||||
|
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
|
||||||
|
golang.org/x/text v0.7.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
|
||||||
|
golang.org/x/text v0.14.0/go.mod h1:18ZOQIKpY8NJVqYksKHtTdi31H5itFRjB5/qKTNYzSU=
|
||||||
|
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||||
|
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
|
||||||
|
golang.org/x/tools v0.1.12/go.mod h1:hNGJHUnrk76NpqgfD5Aqm5Crs+Hm0VOH/i9J2+nxYbc=
|
||||||
|
golang.org/x/tools v0.6.0/go.mod h1:Xwgl3UAJ/d3gWutnCtw505GrjyAbvKui8lOU390QaIU=
|
||||||
|
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||||
|
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||||
481
go/mail/imap.go
Normal file
481
go/mail/imap.go
Normal file
|
|
@ -0,0 +1,481 @@
|
||||||
|
package mail
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"log"
|
||||||
|
"mime"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"github.com/emersion/go-imap/v2"
|
||||||
|
"github.com/emersion/go-imap/v2/imapclient"
|
||||||
|
"github.com/emersion/go-message"
|
||||||
|
"mail2couch/config"
|
||||||
|
)
|
||||||
|
|
||||||
|
// ImapClient wraps the IMAP client
|
||||||
|
type ImapClient struct {
|
||||||
|
*imapclient.Client
|
||||||
|
}
|
||||||
|
|
||||||
|
// Message represents an email message retrieved from IMAP
|
||||||
|
type Message struct {
|
||||||
|
UID uint32
|
||||||
|
From []string
|
||||||
|
To []string
|
||||||
|
Subject string
|
||||||
|
Date time.Time
|
||||||
|
Body string
|
||||||
|
Headers map[string][]string
|
||||||
|
Attachments []Attachment
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attachment represents an email attachment
|
||||||
|
type Attachment struct {
|
||||||
|
Filename string
|
||||||
|
ContentType string
|
||||||
|
Content []byte
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewImapClient creates a new IMAP client from the configuration
|
||||||
|
func NewImapClient(source *config.MailSource) (*ImapClient, error) {
|
||||||
|
addr := fmt.Sprintf("%s:%d", source.Host, source.Port)
|
||||||
|
|
||||||
|
var client *imapclient.Client
|
||||||
|
var err error
|
||||||
|
|
||||||
|
// Try TLS first for standard IMAPS ports (993, 465)
|
||||||
|
if source.Port == 993 || source.Port == 465 {
|
||||||
|
client, err = imapclient.DialTLS(addr, nil)
|
||||||
|
} else {
|
||||||
|
// Use insecure connection for other ports (143, 3143, etc.)
|
||||||
|
client, err = imapclient.DialInsecure(addr, nil)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to dial IMAP server: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := client.Login(source.User, source.Password).Wait(); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to login: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return &ImapClient{client}, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ListMailboxes lists all available mailboxes
|
||||||
|
func (c *ImapClient) ListMailboxes() ([]string, error) {
|
||||||
|
var mailboxes []string
|
||||||
|
cmd := c.List("", "*", nil)
|
||||||
|
|
||||||
|
infos, err := cmd.Collect()
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, info := range infos {
|
||||||
|
mailboxes = append(mailboxes, info.Mailbox)
|
||||||
|
}
|
||||||
|
|
||||||
|
return mailboxes, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// GetMessages retrieves messages from a specific mailbox with filtering support
|
||||||
|
// Returns messages and a map of all current UIDs in the mailbox
|
||||||
|
// maxMessages: 0 means no limit, > 0 limits the number of messages to fetch
|
||||||
|
// since: if provided, only fetch messages newer than this date (for incremental sync)
|
||||||
|
func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages int, messageFilter *config.MessageFilter) ([]*Message, map[uint32]bool, error) {
|
||||||
|
// Select the mailbox
|
||||||
|
mbox, err := c.Select(mailbox, nil).Wait()
|
||||||
|
if err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to select mailbox %s: %w", mailbox, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if mbox.NumMessages == 0 {
|
||||||
|
return []*Message{}, make(map[uint32]bool), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
var messages []*Message
|
||||||
|
currentUIDs := make(map[uint32]bool)
|
||||||
|
|
||||||
|
// First, get all current UIDs in the mailbox for sync purposes
|
||||||
|
allUIDsSet := imap.SeqSet{}
|
||||||
|
allUIDsSet.AddRange(1, mbox.NumMessages)
|
||||||
|
|
||||||
|
// Fetch UIDs for all messages to track current state
|
||||||
|
uidCmd := c.Fetch(allUIDsSet, &imap.FetchOptions{UID: true})
|
||||||
|
for {
|
||||||
|
msg := uidCmd.Next()
|
||||||
|
if msg == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
data, err := msg.Collect()
|
||||||
|
if err != nil {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if data.UID != 0 {
|
||||||
|
currentUIDs[uint32(data.UID)] = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
uidCmd.Close()
|
||||||
|
|
||||||
|
// Determine which messages to fetch based on since date
|
||||||
|
var seqSet imap.SeqSet
|
||||||
|
|
||||||
|
if since != nil {
|
||||||
|
// Use IMAP SEARCH to find messages since the specified date
|
||||||
|
searchCriteria := &imap.SearchCriteria{
|
||||||
|
Since: *since,
|
||||||
|
}
|
||||||
|
|
||||||
|
searchCmd := c.Search(searchCriteria, nil)
|
||||||
|
searchResults, err := searchCmd.Wait()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("IMAP SEARCH failed, falling back to fetch all: %v", err)
|
||||||
|
// Fall back to fetching all messages
|
||||||
|
numToFetch := mbox.NumMessages
|
||||||
|
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||||
|
numToFetch = uint32(maxMessages)
|
||||||
|
}
|
||||||
|
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||||
|
} else {
|
||||||
|
// Convert search results to sequence set
|
||||||
|
searchSeqNums := searchResults.AllSeqNums()
|
||||||
|
if len(searchSeqNums) == 0 {
|
||||||
|
return []*Message{}, currentUIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Limit results if maxMessages is specified
|
||||||
|
if maxMessages > 0 && len(searchSeqNums) > maxMessages {
|
||||||
|
searchSeqNums = searchSeqNums[len(searchSeqNums)-maxMessages:]
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, seqNum := range searchSeqNums {
|
||||||
|
seqSet.AddNum(seqNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// No since date specified, fetch recent messages up to maxMessages
|
||||||
|
numToFetch := mbox.NumMessages
|
||||||
|
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||||
|
numToFetch = uint32(maxMessages)
|
||||||
|
}
|
||||||
|
|
||||||
|
if numToFetch == 0 {
|
||||||
|
return []*Message{}, currentUIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fetch the most recent messages
|
||||||
|
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Fetch message data - get envelope and full message body
|
||||||
|
options := &imap.FetchOptions{
|
||||||
|
Envelope: true,
|
||||||
|
UID: true,
|
||||||
|
BodySection: []*imap.FetchItemBodySection{
|
||||||
|
{}, // Empty section gets the entire message
|
||||||
|
},
|
||||||
|
}
|
||||||
|
|
||||||
|
fetchCmd := c.Fetch(seqSet, options)
|
||||||
|
|
||||||
|
for {
|
||||||
|
msg := fetchCmd.Next()
|
||||||
|
if msg == nil {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
parsedMsg, err := c.parseMessage(msg)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Failed to parse message: %v", err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Apply message-level keyword filtering
|
||||||
|
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) {
|
||||||
|
continue // Skip this message due to keyword filter
|
||||||
|
}
|
||||||
|
|
||||||
|
messages = append(messages, parsedMsg)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := fetchCmd.Close(); err != nil {
|
||||||
|
return nil, nil, fmt.Errorf("failed to fetch messages: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return messages, currentUIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseMessage parses an IMAP fetch response into our Message struct
|
||||||
|
func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) {
|
||||||
|
msg := &Message{
|
||||||
|
UID: fetchMsg.SeqNum, // Using sequence number for now
|
||||||
|
Headers: make(map[string][]string),
|
||||||
|
Attachments: []Attachment{},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Collect all fetch data first
|
||||||
|
buffer, err := fetchMsg.Collect()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to collect fetch data: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse envelope for basic headers
|
||||||
|
if buffer.Envelope != nil {
|
||||||
|
env := buffer.Envelope
|
||||||
|
msg.Subject = env.Subject
|
||||||
|
msg.Date = env.Date
|
||||||
|
|
||||||
|
// Parse From addresses
|
||||||
|
for _, addr := range env.From {
|
||||||
|
if addr.Mailbox != "" {
|
||||||
|
fullAddr := addr.Mailbox
|
||||||
|
if addr.Host != "" {
|
||||||
|
fullAddr = addr.Mailbox + "@" + addr.Host
|
||||||
|
}
|
||||||
|
msg.From = append(msg.From, fullAddr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse To addresses
|
||||||
|
for _, addr := range env.To {
|
||||||
|
if addr.Mailbox != "" {
|
||||||
|
fullAddr := addr.Mailbox
|
||||||
|
if addr.Host != "" {
|
||||||
|
fullAddr = addr.Mailbox + "@" + addr.Host
|
||||||
|
}
|
||||||
|
msg.To = append(msg.To, fullAddr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get UID if available
|
||||||
|
if buffer.UID != 0 {
|
||||||
|
msg.UID = uint32(buffer.UID)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse full message content
|
||||||
|
if len(buffer.BodySection) > 0 {
|
||||||
|
bodyBuffer := buffer.BodySection[0]
|
||||||
|
reader := bytes.NewReader(bodyBuffer.Bytes)
|
||||||
|
|
||||||
|
// Parse the message using go-message
|
||||||
|
entity, err := message.Read(reader)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse message: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Extract headers
|
||||||
|
header := entity.Header
|
||||||
|
for field := header.Fields(); field.Next(); {
|
||||||
|
key := field.Key()
|
||||||
|
value, _ := field.Text()
|
||||||
|
msg.Headers[key] = append(msg.Headers[key], value)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Parse message body and attachments
|
||||||
|
if err := c.parseMessageBody(entity, msg); err != nil {
|
||||||
|
return nil, fmt.Errorf("failed to parse message body: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return msg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseMessageBody extracts the body and attachments from a message entity
|
||||||
|
func (c *ImapClient) parseMessageBody(entity *message.Entity, msg *Message) error {
|
||||||
|
mediaType, _, err := entity.Header.ContentType()
|
||||||
|
if err != nil {
|
||||||
|
// Default to text/plain if no content type
|
||||||
|
mediaType = "text/plain"
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.HasPrefix(mediaType, "multipart/") {
|
||||||
|
// Handle multipart message
|
||||||
|
mr := entity.MultipartReader()
|
||||||
|
if mr == nil {
|
||||||
|
return fmt.Errorf("failed to create multipart reader")
|
||||||
|
}
|
||||||
|
|
||||||
|
for {
|
||||||
|
part, err := mr.NextPart()
|
||||||
|
if err == io.EOF {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to read multipart: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := c.parseMessagePart(part, msg); err != nil {
|
||||||
|
log.Printf("Failed to parse message part: %v", err)
|
||||||
|
// Continue processing other parts
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Handle single part message
|
||||||
|
if err := c.parseMessagePart(entity, msg); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseMessagePart processes a single message part (body or attachment)
|
||||||
|
func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) error {
|
||||||
|
mediaType, params, err := entity.Header.ContentType()
|
||||||
|
if err != nil {
|
||||||
|
mediaType = "text/plain"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Get content disposition
|
||||||
|
disposition, dispositionParams, _ := entity.Header.ContentDisposition()
|
||||||
|
|
||||||
|
// Determine if this is an attachment
|
||||||
|
isAttachment := disposition == "attachment" ||
|
||||||
|
(disposition == "inline" && dispositionParams["filename"] != "") ||
|
||||||
|
params["name"] != ""
|
||||||
|
|
||||||
|
if isAttachment {
|
||||||
|
// Handle attachment
|
||||||
|
filename := dispositionParams["filename"]
|
||||||
|
if filename == "" {
|
||||||
|
filename = params["name"]
|
||||||
|
}
|
||||||
|
if filename == "" {
|
||||||
|
filename = "unnamed_attachment"
|
||||||
|
}
|
||||||
|
|
||||||
|
// Decode filename if needed
|
||||||
|
decoder := &mime.WordDecoder{}
|
||||||
|
filename, _ = decoder.DecodeHeader(filename)
|
||||||
|
|
||||||
|
// Read attachment content
|
||||||
|
content, err := io.ReadAll(entity.Body)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to read attachment content: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
attachment := Attachment{
|
||||||
|
Filename: filename,
|
||||||
|
ContentType: mediaType,
|
||||||
|
Content: content,
|
||||||
|
}
|
||||||
|
|
||||||
|
msg.Attachments = append(msg.Attachments, attachment)
|
||||||
|
} else if strings.HasPrefix(mediaType, "text/") && msg.Body == "" {
|
||||||
|
// Handle text body (only take the first text part as body)
|
||||||
|
bodyBytes, err := io.ReadAll(entity.Body)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to read message body: %w", err)
|
||||||
|
}
|
||||||
|
msg.Body = string(bodyBytes)
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// ShouldProcessMailbox checks if a mailbox should be processed based on filters with wildcard support
|
||||||
|
func (c *ImapClient) ShouldProcessMailbox(mailbox string, filter *config.FolderFilter) bool {
|
||||||
|
// If include list is specified, mailbox must match at least one pattern
|
||||||
|
if len(filter.Include) > 0 {
|
||||||
|
found := false
|
||||||
|
for _, pattern := range filter.Include {
|
||||||
|
// Handle special case: "*" means include all folders
|
||||||
|
if pattern == "*" {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
// Use filepath.Match for wildcard pattern matching
|
||||||
|
if matched, err := filepath.Match(pattern, mailbox); err == nil && matched {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
// Also support exact string matching for backwards compatibility
|
||||||
|
if mailbox == pattern {
|
||||||
|
found = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !found {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// If exclude list is specified, mailbox must not match any exclude pattern
|
||||||
|
for _, pattern := range filter.Exclude {
|
||||||
|
// Use filepath.Match for wildcard pattern matching
|
||||||
|
if matched, err := filepath.Match(pattern, mailbox); err == nil && matched {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
// Also support exact string matching for backwards compatibility
|
||||||
|
if mailbox == pattern {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// ShouldProcessMessage checks if a message should be processed based on keyword filters
|
||||||
|
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool {
|
||||||
|
// Check subject keywords
|
||||||
|
if len(filter.SubjectKeywords) > 0 {
|
||||||
|
if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check sender keywords
|
||||||
|
if len(filter.SenderKeywords) > 0 {
|
||||||
|
senderMatch := false
|
||||||
|
for _, sender := range msg.From {
|
||||||
|
if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) {
|
||||||
|
senderMatch = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !senderMatch {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Check recipient keywords
|
||||||
|
if len(filter.RecipientKeywords) > 0 {
|
||||||
|
recipientMatch := false
|
||||||
|
for _, recipient := range msg.To {
|
||||||
|
if c.containsAnyKeyword(strings.ToLower(recipient), filter.RecipientKeywords) {
|
||||||
|
recipientMatch = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !recipientMatch {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
// containsAnyKeyword checks if the text contains any of the specified keywords (case-insensitive)
|
||||||
|
func (c *ImapClient) containsAnyKeyword(text string, keywords []string) bool {
|
||||||
|
for _, keyword := range keywords {
|
||||||
|
if strings.Contains(text, strings.ToLower(keyword)) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
// Logout logs the client out
|
||||||
|
func (c *ImapClient) Logout() {
|
||||||
|
if err := c.Client.Logout(); err != nil {
|
||||||
|
log.Printf("Failed to logout: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
BIN
go/mail2couch
Executable file
BIN
go/mail2couch
Executable file
Binary file not shown.
205
go/main.go
Normal file
205
go/main.go
Normal file
|
|
@ -0,0 +1,205 @@
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"fmt"
|
||||||
|
"log"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
"mail2couch/config"
|
||||||
|
"mail2couch/couch"
|
||||||
|
"mail2couch/mail"
|
||||||
|
)
|
||||||
|
|
||||||
|
func main() {
|
||||||
|
args := config.ParseCommandLine()
|
||||||
|
|
||||||
|
cfg, err := config.LoadConfigWithDiscovery(args)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to load configuration: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Initialize CouchDB client
|
||||||
|
couchClient, err := couch.NewClient(&cfg.CouchDb)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatalf("Failed to create CouchDB client: %v", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf("Found %d mail source(s) to process.\n", len(cfg.MailSources))
|
||||||
|
for _, source := range cfg.MailSources {
|
||||||
|
if !source.Enabled {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Generate per-account database name
|
||||||
|
dbName := couch.GenerateAccountDBName(source.Name, source.User)
|
||||||
|
|
||||||
|
// Ensure the account-specific database exists
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||||
|
err = couchClient.EnsureDB(ctx, dbName)
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Could not ensure CouchDB database '%s' exists (is it running?): %v", dbName, err)
|
||||||
|
continue
|
||||||
|
} else {
|
||||||
|
fmt.Printf("CouchDB database '%s' is ready for account: %s\n", dbName, source.Name)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf(" - Processing source: %s\n", source.Name)
|
||||||
|
if source.Protocol == "imap" {
|
||||||
|
err := processImapSource(&source, couchClient, dbName, args.MaxMessages)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" ERROR: Failed to process IMAP source %s: %v", source.Name, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func processImapSource(source *config.MailSource, couchClient *couch.Client, dbName string, maxMessages int) error {
|
||||||
|
fmt.Printf(" Connecting to IMAP server: %s:%d\n", source.Host, source.Port)
|
||||||
|
imapClient, err := mail.NewImapClient(source)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to connect to IMAP server: %w", err)
|
||||||
|
}
|
||||||
|
defer imapClient.Logout()
|
||||||
|
|
||||||
|
fmt.Println(" IMAP connection successful.")
|
||||||
|
|
||||||
|
mailboxes, err := imapClient.ListMailboxes()
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("failed to list mailboxes: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf(" Found %d mailboxes.\n", len(mailboxes))
|
||||||
|
|
||||||
|
// Parse the since date from config if provided (fallback for first sync)
|
||||||
|
var configSinceDate *time.Time
|
||||||
|
if source.MessageFilter.Since != "" {
|
||||||
|
parsed, err := time.Parse("2006-01-02", source.MessageFilter.Since)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" WARNING: Invalid since date format '%s', ignoring filter", source.MessageFilter.Since)
|
||||||
|
} else {
|
||||||
|
configSinceDate = &parsed
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
totalMessages := 0
|
||||||
|
totalStored := 0
|
||||||
|
|
||||||
|
// Process each mailbox
|
||||||
|
for _, mailbox := range mailboxes {
|
||||||
|
// Check if this mailbox should be processed based on filters
|
||||||
|
if !imapClient.ShouldProcessMailbox(mailbox, &source.FolderFilter) {
|
||||||
|
fmt.Printf(" Skipping mailbox: %s (filtered)\n", mailbox)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf(" Processing mailbox: %s (mode: %s)\n", mailbox, source.Mode)
|
||||||
|
|
||||||
|
// Get sync metadata to determine incremental sync date
|
||||||
|
syncCtx, syncCancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
syncMetadata, err := couchClient.GetSyncMetadata(syncCtx, dbName, mailbox)
|
||||||
|
syncCancel()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" ERROR: Failed to get sync metadata for %s: %v", mailbox, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Determine the since date for incremental sync
|
||||||
|
var sinceDate *time.Time
|
||||||
|
if syncMetadata != nil {
|
||||||
|
// Use last sync time for incremental sync
|
||||||
|
sinceDate = &syncMetadata.LastSyncTime
|
||||||
|
fmt.Printf(" Incremental sync since: %s (last synced %d messages)\n",
|
||||||
|
sinceDate.Format("2006-01-02 15:04:05"), syncMetadata.MessageCount)
|
||||||
|
} else {
|
||||||
|
// First sync - use config since date if available
|
||||||
|
sinceDate = configSinceDate
|
||||||
|
if sinceDate != nil {
|
||||||
|
fmt.Printf(" First sync since: %s (from config)\n", sinceDate.Format("2006-01-02"))
|
||||||
|
} else {
|
||||||
|
fmt.Printf(" First full sync (no date filter)\n")
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Retrieve messages from the mailbox
|
||||||
|
messages, currentUIDs, err := imapClient.GetMessages(mailbox, sinceDate, maxMessages, &source.MessageFilter)
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" ERROR: Failed to get messages from %s: %v", mailbox, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
// Perform sync/archive logic
|
||||||
|
mailboxSyncCtx, mailboxSyncCancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
err = couchClient.SyncMailbox(mailboxSyncCtx, dbName, mailbox, currentUIDs, source.IsSyncMode())
|
||||||
|
mailboxSyncCancel()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" ERROR: Failed to sync mailbox %s: %v", mailbox, err)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(messages) == 0 {
|
||||||
|
fmt.Printf(" No new messages found in %s\n", mailbox)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf(" Found %d messages in %s\n", len(messages), mailbox)
|
||||||
|
totalMessages += len(messages)
|
||||||
|
|
||||||
|
// Convert messages to CouchDB documents
|
||||||
|
var docs []*couch.MailDocument
|
||||||
|
for _, msg := range messages {
|
||||||
|
doc := couch.ConvertMessage(msg, mailbox)
|
||||||
|
docs = append(docs, doc)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store messages in CouchDB with attachments
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
|
||||||
|
stored := 0
|
||||||
|
for i, doc := range docs {
|
||||||
|
err := couchClient.StoreMessage(ctx, dbName, doc, messages[i])
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" ERROR: Failed to store message %s: %v", doc.ID, err)
|
||||||
|
} else {
|
||||||
|
stored++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
cancel()
|
||||||
|
|
||||||
|
fmt.Printf(" Stored %d/%d messages from %s\n", stored, len(messages), mailbox)
|
||||||
|
totalStored += stored
|
||||||
|
|
||||||
|
// Update sync metadata after successful processing
|
||||||
|
if len(messages) > 0 {
|
||||||
|
// Find the highest UID processed
|
||||||
|
var maxUID uint32
|
||||||
|
for _, msg := range messages {
|
||||||
|
if msg.UID > maxUID {
|
||||||
|
maxUID = msg.UID
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Create/update sync metadata
|
||||||
|
newMetadata := &couch.SyncMetadata{
|
||||||
|
Mailbox: mailbox,
|
||||||
|
LastSyncTime: time.Now(),
|
||||||
|
LastMessageUID: maxUID,
|
||||||
|
MessageCount: stored,
|
||||||
|
}
|
||||||
|
|
||||||
|
// Store sync metadata
|
||||||
|
metadataCtx, metadataCancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||||
|
err = couchClient.StoreSyncMetadata(metadataCtx, dbName, newMetadata)
|
||||||
|
metadataCancel()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf(" WARNING: Failed to store sync metadata for %s: %v", mailbox, err)
|
||||||
|
} else {
|
||||||
|
fmt.Printf(" Updated sync metadata (last UID: %d)\n", maxUID)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
fmt.Printf(" Summary: Processed %d messages, stored %d new messages\n", totalMessages, totalStored)
|
||||||
|
return nil
|
||||||
|
}
|
||||||
250
test/README.md
Normal file
250
test/README.md
Normal file
|
|
@ -0,0 +1,250 @@
|
||||||
|
# mail2couch Test Environment
|
||||||
|
|
||||||
|
This directory contains a complete test environment for mail2couch using Podman containers.
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
The test environment provides:
|
||||||
|
- **CouchDB**: Database for storing email messages
|
||||||
|
- **GreenMail IMAP Server**: Java-based mail server designed for testing with pre-populated test accounts and messages
|
||||||
|
- **Test Configuration**: Ready-to-use config for testing both sync and archive modes
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### Run Basic Integration Tests
|
||||||
|
```bash
|
||||||
|
./run-tests.sh
|
||||||
|
```
|
||||||
|
This comprehensive test will:
|
||||||
|
1. Start all containers with cleanup
|
||||||
|
2. Populate test data
|
||||||
|
3. Build and run mail2couch
|
||||||
|
4. Verify database creation and document storage
|
||||||
|
5. Test incremental sync behavior
|
||||||
|
6. Clean up automatically
|
||||||
|
|
||||||
|
### Run Wildcard Pattern Tests
|
||||||
|
```bash
|
||||||
|
./test-wildcard-patterns.sh
|
||||||
|
```
|
||||||
|
This will test various wildcard folder patterns including:
|
||||||
|
- `*` (all folders)
|
||||||
|
- `Work*` (prefix patterns)
|
||||||
|
- `*/Drafts` (subfolder patterns)
|
||||||
|
- Complex include/exclude combinations
|
||||||
|
|
||||||
|
### Run Incremental Sync Tests
|
||||||
|
```bash
|
||||||
|
./test-incremental-sync.sh
|
||||||
|
```
|
||||||
|
This will test incremental synchronization functionality:
|
||||||
|
- First sync establishes baseline
|
||||||
|
- New messages are added to test accounts
|
||||||
|
- Second sync should only fetch new messages
|
||||||
|
- Sync metadata tracking and IMAP SEARCH with SINCE
|
||||||
|
|
||||||
|
### Manual Testing Environment
|
||||||
|
```bash
|
||||||
|
# Start persistent test environment (for manual experimentation)
|
||||||
|
./start-test-env.sh
|
||||||
|
|
||||||
|
# Run mail2couch manually with different configurations
|
||||||
|
cd ../go
|
||||||
|
./mail2couch -config ../test/config-test.json
|
||||||
|
./mail2couch -config ../test/config-wildcard-examples.json
|
||||||
|
|
||||||
|
# Stop test environment when done
|
||||||
|
cd ../test
|
||||||
|
./stop-test-env.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test Scripts Overview
|
||||||
|
|
||||||
|
### Automated Testing (Recommended)
|
||||||
|
- **`./run-tests.sh`**: Complete integration test with automatic cleanup
|
||||||
|
- Starts containers, populates data, runs mail2couch, verifies results
|
||||||
|
- Tests basic functionality, database creation, and incremental sync
|
||||||
|
- Cleans up automatically - perfect for CI/CD or quick validation
|
||||||
|
|
||||||
|
### Specialized Feature Testing
|
||||||
|
- **`./test-wildcard-patterns.sh`**: Comprehensive folder pattern testing
|
||||||
|
- Tests `*`, `Work*`, `*/Drafts`, and complex include/exclude patterns
|
||||||
|
- Self-contained with own setup/teardown
|
||||||
|
- **`./test-incremental-sync.sh`**: Incremental synchronization testing
|
||||||
|
- Tests sync metadata tracking and IMAP SEARCH with SINCE
|
||||||
|
- Multi-step validation: baseline sync → add messages → incremental sync
|
||||||
|
- Self-contained with own setup/teardown
|
||||||
|
|
||||||
|
### Manual Testing Environment
|
||||||
|
- **`./start-test-env.sh`**: Start persistent test containers
|
||||||
|
- Keeps environment running for manual experimentation
|
||||||
|
- Populates test data once
|
||||||
|
- Use with different configurations for development
|
||||||
|
- **`./stop-test-env.sh`**: Clean up manual test environment
|
||||||
|
- Only needed after using `start-test-env.sh`
|
||||||
|
|
||||||
|
## Test Accounts
|
||||||
|
|
||||||
|
The test environment includes these IMAP accounts:
|
||||||
|
|
||||||
|
| Username | Password | Mode | Folder Pattern | Purpose |
|
||||||
|
|----------|----------|------|---------------|---------|
|
||||||
|
| `testuser1` | `password123` | archive | `*` (exclude Drafts, Trash) | Wildcard all folders test |
|
||||||
|
| `syncuser` | `syncpass` | sync | `Work*`, `Important*`, `INBOX` | Work pattern test |
|
||||||
|
| `archiveuser` | `archivepass` | archive | `INBOX`, `Sent`, `Personal` | Specific folders test |
|
||||||
|
| `testuser2` | `password456` | archive | `Work/*`, `Archive/*` | Subfolder pattern test |
|
||||||
|
|
||||||
|
Each account contains:
|
||||||
|
- 10 messages in INBOX (every 3rd has an attachment)
|
||||||
|
- 3 messages in each additional folder
|
||||||
|
- Test folders: `Sent`, `Work/Projects`, `Work/Archive`, `Work/Temp`, `Personal`, `Important/Urgent`, `Important/Meetings`, `Archive/2024`, `Archive/Projects`, `Archive/Drafts`, `Drafts`, `Trash`
|
||||||
|
- Various message types for comprehensive wildcard testing
|
||||||
|
|
||||||
|
## Services
|
||||||
|
|
||||||
|
### CouchDB
|
||||||
|
- **URL**: http://localhost:5984
|
||||||
|
- **Admin**: `admin` / `password`
|
||||||
|
- **Web UI**: http://localhost:5984/_utils
|
||||||
|
|
||||||
|
### GreenMail Server
|
||||||
|
- **Host**: localhost
|
||||||
|
- **IMAP Port**: 3143 (plain)
|
||||||
|
- **IMAPS Port**: 3993 (SSL)
|
||||||
|
- **SMTP Port**: 3025
|
||||||
|
- **Server**: GreenMail (Java-based test server)
|
||||||
|
|
||||||
|
## Database Structure
|
||||||
|
|
||||||
|
mail2couch will create separate databases for each mail source (with `m2c_` prefix):
|
||||||
|
- `m2c_wildcard_all_folders_test` - Wildcard All Folders Test (archive mode)
|
||||||
|
- `m2c_work_pattern_test` - Work Pattern Test (sync mode)
|
||||||
|
- `m2c_specific_folders_only` - Specific Folders Only (archive mode)
|
||||||
|
- `m2c_subfolder_pattern_test` - Subfolder Pattern Test (archive mode)
|
||||||
|
|
||||||
|
Each database contains documents with:
|
||||||
|
- `mailbox` field indicating the origin folder
|
||||||
|
- Native CouchDB attachments for email attachments
|
||||||
|
- Full message headers and body content
|
||||||
|
|
||||||
|
## Testing Sync vs Archive Modes
|
||||||
|
|
||||||
|
### Sync Mode (`syncuser`)
|
||||||
|
- Database exactly matches mail account
|
||||||
|
- If messages are deleted from IMAP, they're removed from CouchDB
|
||||||
|
- 1-to-1 relationship
|
||||||
|
|
||||||
|
### Archive Mode (`archiveuser`, `testuser1`)
|
||||||
|
- Database preserves all messages ever seen
|
||||||
|
- Messages deleted from IMAP remain in CouchDB
|
||||||
|
- Archive/backup behavior
|
||||||
|
|
||||||
|
## Wildcard Pattern Examples
|
||||||
|
|
||||||
|
The test environment demonstrates these wildcard patterns:
|
||||||
|
|
||||||
|
### All Folders Pattern (`*`)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Drafts", "Trash"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Processes all folders except Drafts and Trash.
|
||||||
|
|
||||||
|
### Work Pattern (`Work*`)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work*", "Important*", "INBOX"],
|
||||||
|
"exclude": ["*Temp*"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Includes Work/Projects, Work/Archive, Important/Urgent, Important/Meetings, and INBOX. Excludes Work/Temp.
|
||||||
|
|
||||||
|
### Specific Folders
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent", "Personal"],
|
||||||
|
"exclude": []
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Only processes the exact named folders.
|
||||||
|
|
||||||
|
### Subfolder Pattern (`Work/*`)
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work/*", "Archive/*"],
|
||||||
|
"exclude": ["*/Drafts"]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
Includes all subfolders under Work and Archive, but excludes any Drafts subfolder.
|
||||||
|
|
||||||
|
## File Structure
|
||||||
|
|
||||||
|
```
|
||||||
|
test/
|
||||||
|
├── podman-compose.yml # Container orchestration (GreenMail + CouchDB)
|
||||||
|
├── config-test.json # Main test configuration with wildcard examples
|
||||||
|
├── config-wildcard-examples.json # Advanced wildcard patterns
|
||||||
|
├── run-tests.sh # Automated integration test (recommended)
|
||||||
|
├── test-wildcard-patterns.sh # Specialized wildcard pattern testing
|
||||||
|
├── test-incremental-sync.sh # Specialized incremental sync testing
|
||||||
|
├── start-test-env.sh # Start persistent test environment
|
||||||
|
├── stop-test-env.sh # Stop test environment
|
||||||
|
├── populate-greenmail.py # Create test messages across multiple folders
|
||||||
|
├── dovecot/ # Dovecot configuration (legacy, unused)
|
||||||
|
└── README.md # This file
|
||||||
|
```
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Podman and podman-compose
|
||||||
|
- OpenSSL (for certificate generation)
|
||||||
|
- curl and nc (for connectivity checks)
|
||||||
|
- Go (for building mail2couch)
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Containers won't start
|
||||||
|
```bash
|
||||||
|
# Check podman status
|
||||||
|
podman ps -a
|
||||||
|
|
||||||
|
# View logs
|
||||||
|
podman logs mail2couch_test_couchdb
|
||||||
|
podman logs mail2couch_test_imap
|
||||||
|
```
|
||||||
|
|
||||||
|
### CouchDB connection issues
|
||||||
|
- Verify CouchDB is running: `curl http://localhost:5984`
|
||||||
|
- Check admin credentials: `admin/password`
|
||||||
|
|
||||||
|
### IMAP connection issues
|
||||||
|
- Test IMAP connection: `nc -z localhost 143`
|
||||||
|
- Check Dovecot logs: `podman logs mail2couch_test_imap`
|
||||||
|
|
||||||
|
### Permission issues
|
||||||
|
- Ensure scripts are executable: `chmod +x *.sh`
|
||||||
|
- Check file permissions in dovecot directory
|
||||||
|
|
||||||
|
## Advanced Usage
|
||||||
|
|
||||||
|
### Add custom test messages
|
||||||
|
Edit `populate-test-messages.sh` to create additional test scenarios.
|
||||||
|
|
||||||
|
### Modify IMAP configuration
|
||||||
|
Edit `dovecot/dovecot.conf` and restart containers.
|
||||||
|
|
||||||
|
### Test with SSL
|
||||||
|
Update `config-test.json` to use port 993 and enable SSL.
|
||||||
|
|
||||||
|
### Custom test scenarios
|
||||||
|
Create additional configuration files for specific test cases.
|
||||||
74
test/config-test.json
Normal file
74
test/config-test.json
Normal file
|
|
@ -0,0 +1,74 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "Wildcard All Folders Test",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "testuser1",
|
||||||
|
"password": "password123",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Drafts", "Trash"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"subjectKeywords": ["meeting", "important"],
|
||||||
|
"senderKeywords": ["@company.com"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Work Pattern Test",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "syncuser",
|
||||||
|
"password": "syncpass",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work*", "Important*", "INBOX"],
|
||||||
|
"exclude": ["*Temp*"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"recipientKeywords": ["support@", "team@"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Specific Folders Only",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "archiveuser",
|
||||||
|
"password": "archivepass",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent", "Personal"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Subfolder Pattern Test",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "testuser2",
|
||||||
|
"password": "password456",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work/*", "Archive/*"],
|
||||||
|
"exclude": ["*/Drafts"]
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
71
test/config-wildcard-examples.json
Normal file
71
test/config-wildcard-examples.json
Normal file
|
|
@ -0,0 +1,71 @@
|
||||||
|
{
|
||||||
|
"couchDb": {
|
||||||
|
"url": "http://localhost:5984",
|
||||||
|
"user": "admin",
|
||||||
|
"password": "password"
|
||||||
|
},
|
||||||
|
"mailSources": [
|
||||||
|
{
|
||||||
|
"name": "All Folders Example",
|
||||||
|
"enabled": true,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "testuser1",
|
||||||
|
"password": "password123",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["*"],
|
||||||
|
"exclude": ["Drafts", "Trash", "Spam"]
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Inbox and Sent Only",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "testuser2",
|
||||||
|
"password": "password456",
|
||||||
|
"mode": "sync",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["INBOX", "Sent"],
|
||||||
|
"exclude": []
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Work Folders Pattern",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "workuser",
|
||||||
|
"password": "workpass",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Work*", "Projects*", "INBOX"],
|
||||||
|
"exclude": ["*Temp*", "*Draft*"]
|
||||||
|
},
|
||||||
|
"messageFilter": {
|
||||||
|
"senderKeywords": ["@company.com"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"name": "Archive Pattern Example",
|
||||||
|
"enabled": false,
|
||||||
|
"protocol": "imap",
|
||||||
|
"host": "localhost",
|
||||||
|
"port": 3143,
|
||||||
|
"user": "archiveuser",
|
||||||
|
"password": "archivepass",
|
||||||
|
"mode": "archive",
|
||||||
|
"folderFilter": {
|
||||||
|
"include": ["Archive*", "*Important*", "INBOX"],
|
||||||
|
"exclude": ["*Temp"]
|
||||||
|
},
|
||||||
|
"messageFilter": {}
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
79
test/dovecot/dovecot.conf
Normal file
79
test/dovecot/dovecot.conf
Normal file
|
|
@ -0,0 +1,79 @@
|
||||||
|
# Dovecot configuration for testing mail2couch
|
||||||
|
|
||||||
|
# Basic settings
|
||||||
|
protocols = imap
|
||||||
|
listen = *
|
||||||
|
|
||||||
|
# SSL/TLS settings - make optional for easier testing
|
||||||
|
ssl = optional
|
||||||
|
ssl_cert = </etc/dovecot/ssl/server.crt
|
||||||
|
ssl_key = </etc/dovecot/ssl/server.key
|
||||||
|
|
||||||
|
# Authentication
|
||||||
|
auth_mechanisms = plain login
|
||||||
|
disable_plaintext_auth = no
|
||||||
|
|
||||||
|
# User database
|
||||||
|
passdb {
|
||||||
|
driver = passwd-file
|
||||||
|
args = scheme=plain username_format=%u /etc/dovecot/passwd
|
||||||
|
}
|
||||||
|
|
||||||
|
userdb {
|
||||||
|
driver = passwd-file
|
||||||
|
args = username_format=%u /etc/dovecot/users
|
||||||
|
}
|
||||||
|
|
||||||
|
# Mail location
|
||||||
|
mail_location = maildir:/var/mail/%u
|
||||||
|
|
||||||
|
# Mailbox settings
|
||||||
|
namespace inbox {
|
||||||
|
type = private
|
||||||
|
separator = /
|
||||||
|
prefix =
|
||||||
|
location =
|
||||||
|
inbox = yes
|
||||||
|
hidden = no
|
||||||
|
list = yes
|
||||||
|
subscriptions = yes
|
||||||
|
|
||||||
|
mailbox Drafts {
|
||||||
|
special_use = \Drafts
|
||||||
|
}
|
||||||
|
mailbox Junk {
|
||||||
|
special_use = \Junk
|
||||||
|
}
|
||||||
|
mailbox Trash {
|
||||||
|
special_use = \Trash
|
||||||
|
}
|
||||||
|
mailbox Sent {
|
||||||
|
special_use = \Sent
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Services
|
||||||
|
service imap-login {
|
||||||
|
inet_listener imap {
|
||||||
|
port = 143
|
||||||
|
}
|
||||||
|
inet_listener imaps {
|
||||||
|
port = 993
|
||||||
|
ssl = yes
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
service auth {
|
||||||
|
unix_listener auth-userdb {
|
||||||
|
mode = 0666
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Logging
|
||||||
|
log_path = /dev/stdout
|
||||||
|
info_log_path = /dev/stdout
|
||||||
|
debug_log_path = /dev/stdout
|
||||||
|
|
||||||
|
# Process limits
|
||||||
|
default_process_limit = 10
|
||||||
|
default_client_limit = 100
|
||||||
51
test/dovecot/entrypoint.sh
Executable file
51
test/dovecot/entrypoint.sh
Executable file
|
|
@ -0,0 +1,51 @@
|
||||||
|
#!/bin/sh
|
||||||
|
|
||||||
|
# Entrypoint script for Dovecot test container
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "Installing Dovecot..."
|
||||||
|
apk add --no-cache dovecot dovecot-lmtpd openssl
|
||||||
|
|
||||||
|
echo "Setting up directories..."
|
||||||
|
mkdir -p /var/mail
|
||||||
|
mkdir -p /var/run/dovecot
|
||||||
|
mkdir -p /var/log/dovecot
|
||||||
|
|
||||||
|
# Create dovecot user if it doesn't exist
|
||||||
|
if ! getent passwd dovecot > /dev/null 2>&1; then
|
||||||
|
addgroup -g 97 dovecot
|
||||||
|
adduser -D -u 97 -G dovecot -s /sbin/nologin dovecot
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Set proper ownership
|
||||||
|
chown -R dovecot:dovecot /var/mail
|
||||||
|
chown -R dovecot:dovecot /var/run/dovecot
|
||||||
|
chown -R root:dovecot /etc/dovecot
|
||||||
|
chmod -R 0640 /etc/dovecot
|
||||||
|
chmod 0644 /etc/dovecot/dovecot.conf
|
||||||
|
|
||||||
|
# Generate SSL certificates if they don't exist
|
||||||
|
if [ ! -f /etc/dovecot/ssl/server.crt ] || [ ! -f /etc/dovecot/ssl/server.key ]; then
|
||||||
|
echo "Generating SSL certificates..."
|
||||||
|
mkdir -p /etc/dovecot/ssl
|
||||||
|
|
||||||
|
# Generate DH parameters (small for testing)
|
||||||
|
openssl dhparam -out /etc/dovecot/ssl/dh.pem 1024
|
||||||
|
|
||||||
|
# Generate private key
|
||||||
|
openssl genrsa -out /etc/dovecot/ssl/server.key 2048
|
||||||
|
|
||||||
|
# Generate certificate
|
||||||
|
openssl req -new -key /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.csr -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
|
||||||
|
openssl x509 -req -days 365 -in /etc/dovecot/ssl/server.csr -signkey /etc/dovecot/ssl/server.key -out /etc/dovecot/ssl/server.crt
|
||||||
|
rm /etc/dovecot/ssl/server.csr
|
||||||
|
fi
|
||||||
|
|
||||||
|
# Ensure SSL directory permissions
|
||||||
|
chown -R dovecot:dovecot /etc/dovecot/ssl
|
||||||
|
chmod 600 /etc/dovecot/ssl/server.key
|
||||||
|
chmod 644 /etc/dovecot/ssl/server.crt
|
||||||
|
|
||||||
|
echo "Starting Dovecot..."
|
||||||
|
exec dovecot -F
|
||||||
8
test/dovecot/passwd
Normal file
8
test/dovecot/passwd
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
# Password database for Dovecot testing
|
||||||
|
# Format: username:password
|
||||||
|
|
||||||
|
# Test accounts with simple passwords for testing
|
||||||
|
testuser1:password123
|
||||||
|
testuser2:password456
|
||||||
|
syncuser:syncpass
|
||||||
|
archiveuser:archivepass
|
||||||
8
test/dovecot/ssl/dh.pem
Normal file
8
test/dovecot/ssl/dh.pem
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
-----BEGIN DH PARAMETERS-----
|
||||||
|
MIIBDAKCAQEAjcUSAHFs60qgDRg/cT7byhuhF3vwZQhmm1QToCFgG4VWu/EOVXq2
|
||||||
|
kHxjxmo3hBuJCqUZqTAyF91Tum7A2QuQhXFrxOpRF8EiyVSgBabjN/WcEHIow1uh
|
||||||
|
Vtb4JOcDl/Q9IJfFT6zyXdQQiHPBOWnpOBKXeQQQIx5plgsrmK0cTO2ZxtyrmHHp
|
||||||
|
wxtE3INKYuBlGH3Y0zghc+Hoezpf/hbIHZibGQ0l79EtBDQjqmqoDJCIiv5gsTt8
|
||||||
|
9VpkR6FFvjWTNOb5qY10W/PRhLGjioX29bp1B6qW5PNJcd//cqrBLebKlkAoXnyx
|
||||||
|
x0uTUy6pmmIt5vdYxx0symrMXZEjrL7uzwIBAgICAOE=
|
||||||
|
-----END DH PARAMETERS-----
|
||||||
20
test/dovecot/ssl/server.crt
Normal file
20
test/dovecot/ssl/server.crt
Normal file
|
|
@ -0,0 +1,20 @@
|
||||||
|
-----BEGIN CERTIFICATE-----
|
||||||
|
MIIDVzCCAj+gAwIBAgIUYvkZSKbVH08s/3B70AW8IEpTB/kwDQYJKoZIhvcNAQEL
|
||||||
|
BQAwVDELMAkGA1UEBhMCVVMxDTALBgNVBAgMBFRlc3QxDTALBgNVBAcMBFRlc3Qx
|
||||||
|
EzARBgNVBAoMCk1haWwyQ291Y2gxEjAQBgNVBAMMCWxvY2FsaG9zdDAeFw0yNTA4
|
||||||
|
MDExNDIyMzVaFw0yNjA4MDExNDIyMzVaMFQxCzAJBgNVBAYTAlVTMQ0wCwYDVQQI
|
||||||
|
DARUZXN0MQ0wCwYDVQQHDARUZXN0MRMwEQYDVQQKDApNYWlsMkNvdWNoMRIwEAYD
|
||||||
|
VQQDDAlsb2NhbGhvc3QwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQCs
|
||||||
|
leQcVLlKx7/bJGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vx
|
||||||
|
Td1mVCm9Lsd9bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDW
|
||||||
|
knco7/MEatakliVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331
|
||||||
|
HEFdIqqccpimnIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwo
|
||||||
|
eN8zBzGFJe9xzqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5
|
||||||
|
LGN5xSzXvFIzgP20y5s5AgMBAAGjITAfMB0GA1UdDgQWBBTkgYZGp2s74D+1ltyl
|
||||||
|
rudF/o7jODANBgkqhkiG9w0BAQsFAAOCAQEATc6ekhuk32meLuxhalz6lNwBxfDg
|
||||||
|
EG3gGUxNwehwgiNCcKIKQFtCwjJde6drOobkRDANtb7g3gSlAxlUCPsO6xnL1c6E
|
||||||
|
HhehFn++7HOpXvmEy/mnoqBL6PLzRZRMRlDynlPVV9Y82zsdrQiQEhGyNTfgP5dk
|
||||||
|
u9RMIMQl1hIK381V738b5MXfdpYhmRiTGEd6hCxCnzkx0OakCLM9lnJASr0dYPuh
|
||||||
|
LYKoClxhr3sV/JsgAmx91BuHGpzaPYQ2zFvCJSqD2ihM7zIl9K2bLIUR87/CznyH
|
||||||
|
JuPRbgt6/cxzwdqflP73j+TTZdlI4gckEA3H0WhNN4nB2SEjTgS+kDctMA==
|
||||||
|
-----END CERTIFICATE-----
|
||||||
28
test/dovecot/ssl/server.key
Normal file
28
test/dovecot/ssl/server.key
Normal file
|
|
@ -0,0 +1,28 @@
|
||||||
|
-----BEGIN PRIVATE KEY-----
|
||||||
|
MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCsleQcVLlKx7/b
|
||||||
|
JGPo1k2Ccu1nKlMv8zdnvYpQ4c3vBVS1vPK3wxVnFz5JWiNPy/vxTd1mVCm9Lsd9
|
||||||
|
bc3QwntbWFW8EO7DNBbCiUbfPeDsURpRT0evuPfCgWHr8pJ0/ZDWknco7/MEatak
|
||||||
|
liVkpf3O6WdbNkx7I+MO2KOePCzIVi5Pxwb3ldXO4OxHsgfKG331HEFdIqqccpim
|
||||||
|
nIUYSYNmyRrowBixanMW/wq7rcInJYuYRnw9wEg24jOfpKLJHuwoeN8zBzGFJe9x
|
||||||
|
zqeaLNa9RBJCJSYp6AnDV6mDpeIEgwrW/66NWYqwVEcC3IJ22Et5LGN5xSzXvFIz
|
||||||
|
gP20y5s5AgMBAAECggEAOfBaM76ns/iqKpYlamnjfIs7svotEjhrHcsub6fWvEsE
|
||||||
|
XLzRmSqHeWP+t55oo2XeL2zOCofvuUDGnQ+rXE2mHwzhP3FJzsOibm2qmtCJvZwe
|
||||||
|
ozRj4xTMLILGDnGRhHAJ21cxZM9lPNLnOzri0868DeYimib49xAdroLBLyKRgDGN
|
||||||
|
O7OAwA4KWAebZRBU8cowEF87WGAI2hOLJA5WIKX7X9SiRDx5sNnDpTpqJvyvuleR
|
||||||
|
D/wKGHzkQiZx8WNJx5A571dilfKEp8S49o4sJdz8DQ/4ruVov2Vi+nSdMEzHnp5m
|
||||||
|
M7ZlZ7IcJRJDKPrDYasmxvtM8EiKyJf87/DPANfYFwKBgQDa+ANr2pGg2oMJVJIq
|
||||||
|
r/mj8wtRsLRYFgfs96BUwp7e9Vo+Rx+E0uZxGGMul6hn4b2/TQljLfiX/CP8ZTyw
|
||||||
|
MStlNlnAXaF425JNuQIFQ2wGGiGdsx2I1WNw7co4UPVVjH11nr3o30g+NDJcp7Tq
|
||||||
|
rvpKvGbeoZtHzOj0bF7fA/B7qwKBgQDJxcUUEH5A50n3oUP99aK/DzSA7kcte2Aw
|
||||||
|
tjv9hbPnbOmcM24Fm+KU7bsRYt9QPa0PU2lV3O1KrHj4q+QRcPFl2P2mzZC+Hzmx
|
||||||
|
H8dEjMmH8YdrjGqethMoUHJCguNfskNwjgWFlxTSBLY+NffghXNzZgiF9d6WqF48
|
||||||
|
iqwH+HsAqwKBgQCb/B2D0Xn4WnEKToKpgh6WGmcv1G9EaL1Qo75FYzcFoUaeItBj
|
||||||
|
MFIUssjEwiinh/pBssFDM9Zpfqar//pRkVVWjnc1P/3tOI1qbKbx1Ou5FRhpXNVn
|
||||||
|
SovCQMLTh2idfq1JAsJKh/TQyyItOxL4M5n9b2Tgp8MUTPaOWDzlJctEbQKBgEVu
|
||||||
|
oNq+sjNzY6iq/dKubEqC2PZlCGlGQ1t/2jTrhXTlrZ3qtLmJYvcMt4rMEzxxfNQB
|
||||||
|
SAYb+CvyHc60l87Ipsj9WovDwUMrS5b/8HpOWCtHmeoQb8Adt4nv5OGuWL/dgAeD
|
||||||
|
V7MYwjljFbNiruG8CnZzbgtrCCWf2o3KylgT0X/xAoGAUhSdBge5Vpg0JcT1VDgm
|
||||||
|
q5rgc6dD1LJtXfBaq3w4kHYK/iLFcPOLUKcIJXNbhMwWza/JwVYK6hsCIw3/b4va
|
||||||
|
NhJ8ABpC3fZqkl28glEF8bnrPAkE1akn2GiBaaEbTCQRMrhZ2SW3JCyjX6yCvvvz
|
||||||
|
m7b2ZpDMJEMIBmgrK70E3Oo=
|
||||||
|
-----END PRIVATE KEY-----
|
||||||
8
test/dovecot/users
Normal file
8
test/dovecot/users
Normal file
|
|
@ -0,0 +1,8 @@
|
||||||
|
# User database for Dovecot testing
|
||||||
|
# Format: username:uid:gid:home:shell
|
||||||
|
|
||||||
|
# Test user accounts
|
||||||
|
testuser1::1001:1001:/var/mail/testuser1::
|
||||||
|
testuser2::1002:1002:/var/mail/testuser2::
|
||||||
|
syncuser::1003:1003:/var/mail/syncuser::
|
||||||
|
archiveuser::1004:1004:/var/mail/archiveuser::
|
||||||
28
test/generate-ssl.sh
Executable file
28
test/generate-ssl.sh
Executable file
|
|
@ -0,0 +1,28 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Generate SSL certificates for Dovecot testing
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
CERT_DIR="dovecot/ssl"
|
||||||
|
mkdir -p "$CERT_DIR"
|
||||||
|
|
||||||
|
# Generate DH parameters
|
||||||
|
echo "Generating DH parameters..."
|
||||||
|
openssl dhparam -out "$CERT_DIR/dh.pem" 2048
|
||||||
|
|
||||||
|
# Generate private key
|
||||||
|
echo "Generating private key..."
|
||||||
|
openssl genrsa -out "$CERT_DIR/server.key" 2048
|
||||||
|
|
||||||
|
# Generate certificate signing request
|
||||||
|
echo "Generating certificate..."
|
||||||
|
openssl req -new -key "$CERT_DIR/server.key" -out "$CERT_DIR/server.csr" -subj "/C=US/ST=Test/L=Test/O=Mail2Couch/CN=localhost"
|
||||||
|
|
||||||
|
# Generate self-signed certificate
|
||||||
|
openssl x509 -req -days 365 -in "$CERT_DIR/server.csr" -signkey "$CERT_DIR/server.key" -out "$CERT_DIR/server.crt"
|
||||||
|
|
||||||
|
# Clean up CSR
|
||||||
|
rm "$CERT_DIR/server.csr"
|
||||||
|
|
||||||
|
echo "SSL certificates generated successfully in $CERT_DIR/"
|
||||||
38
test/podman-compose.yml
Normal file
38
test/podman-compose.yml
Normal file
|
|
@ -0,0 +1,38 @@
|
||||||
|
version: '3.8'
|
||||||
|
|
||||||
|
services:
|
||||||
|
# CouchDB for testing
|
||||||
|
couchdb:
|
||||||
|
image: docker.io/couchdb:3.3
|
||||||
|
container_name: mail2couch_test_couchdb
|
||||||
|
environment:
|
||||||
|
- COUCHDB_USER=admin
|
||||||
|
- COUCHDB_PASSWORD=password
|
||||||
|
ports:
|
||||||
|
- "5984:5984"
|
||||||
|
volumes:
|
||||||
|
- couchdb_data:/opt/couchdb/data
|
||||||
|
networks:
|
||||||
|
- mail2couch_test
|
||||||
|
|
||||||
|
# GreenMail IMAP server for testing
|
||||||
|
greenmail:
|
||||||
|
image: docker.io/greenmail/standalone:2.0.1
|
||||||
|
container_name: mail2couch_test_imap
|
||||||
|
ports:
|
||||||
|
- "3143:3143" # IMAP
|
||||||
|
- "3993:3993" # IMAPS
|
||||||
|
- "3025:3025" # SMTP
|
||||||
|
environment:
|
||||||
|
- GREENMAIL_OPTS=-Dgreenmail.setup.test.all -Dgreenmail.hostname=0.0.0.0 -Dgreenmail.users=testuser1:password123@localhost,testuser2:password456@localhost,syncuser:syncpass@localhost,archiveuser:archivepass@localhost
|
||||||
|
networks:
|
||||||
|
- mail2couch_test
|
||||||
|
depends_on:
|
||||||
|
- couchdb
|
||||||
|
|
||||||
|
volumes:
|
||||||
|
couchdb_data:
|
||||||
|
|
||||||
|
networks:
|
||||||
|
mail2couch_test:
|
||||||
|
driver: bridge
|
||||||
179
test/populate-greenmail.py
Executable file
179
test/populate-greenmail.py
Executable file
|
|
@ -0,0 +1,179 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import imaplib
|
||||||
|
import email
|
||||||
|
from email.mime.text import MIMEText
|
||||||
|
from email.mime.multipart import MIMEMultipart
|
||||||
|
from email.mime.base import MIMEBase
|
||||||
|
from email import encoders
|
||||||
|
import time
|
||||||
|
import sys
|
||||||
|
|
||||||
|
def create_simple_message(subject, body, from_addr="test-sender@example.com", to_addr="user@example.com"):
|
||||||
|
"""Create a simple text message"""
|
||||||
|
msg = MIMEText(body)
|
||||||
|
msg['Subject'] = subject
|
||||||
|
msg['From'] = from_addr
|
||||||
|
msg['To'] = to_addr
|
||||||
|
msg['Date'] = email.utils.formatdate(localtime=True)
|
||||||
|
return msg.as_string()
|
||||||
|
|
||||||
|
def create_message_with_attachment(subject, body, attachment_content, from_addr="test-sender@example.com", to_addr="user@example.com"):
|
||||||
|
"""Create a message with an attachment"""
|
||||||
|
msg = MIMEMultipart()
|
||||||
|
msg['Subject'] = subject
|
||||||
|
msg['From'] = from_addr
|
||||||
|
msg['To'] = to_addr
|
||||||
|
msg['Date'] = email.utils.formatdate(localtime=True)
|
||||||
|
|
||||||
|
# Add body
|
||||||
|
msg.attach(MIMEText(body, 'plain'))
|
||||||
|
|
||||||
|
# Add attachment
|
||||||
|
part = MIMEBase('text', 'plain')
|
||||||
|
part.set_payload(attachment_content)
|
||||||
|
encoders.encode_base64(part)
|
||||||
|
part.add_header('Content-Disposition', 'attachment; filename="attachment.txt"')
|
||||||
|
msg.attach(part)
|
||||||
|
|
||||||
|
return msg.as_string()
|
||||||
|
|
||||||
|
def populate_user_mailbox(username, password, host='localhost', port=3143):
|
||||||
|
"""Populate a user's mailbox with test messages"""
|
||||||
|
print(f"Connecting to {username}@{host}:{port}")
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Connect to IMAP server
|
||||||
|
imap = imaplib.IMAP4(host, port)
|
||||||
|
imap.login(username, password)
|
||||||
|
|
||||||
|
# Create additional folders for testing wildcards
|
||||||
|
# These folders are designed to test various wildcard patterns
|
||||||
|
test_folders = [
|
||||||
|
'Sent', # Exact match
|
||||||
|
'Work/Projects', # Work/* pattern
|
||||||
|
'Work/Archive', # Work/* pattern
|
||||||
|
'Work/Temp', # Work/* but excluded by *Temp*
|
||||||
|
'Personal', # Exact match
|
||||||
|
'Important/Urgent', # Important/* pattern
|
||||||
|
'Important/Meetings', # Important/* pattern
|
||||||
|
'Archive/2024', # Archive/* pattern
|
||||||
|
'Archive/Projects', # Archive/* pattern
|
||||||
|
'Archive/Drafts', # Archive/* but excluded by */Drafts
|
||||||
|
'Drafts', # Should be excluded
|
||||||
|
'Trash' # Should be excluded
|
||||||
|
]
|
||||||
|
created_folders = ['INBOX'] # INBOX always exists
|
||||||
|
|
||||||
|
for folder in test_folders:
|
||||||
|
try:
|
||||||
|
imap.create(folder)
|
||||||
|
created_folders.append(folder)
|
||||||
|
print(f" Created folder: {folder}")
|
||||||
|
except Exception as e:
|
||||||
|
print(f" Folder {folder} may already exist or creation failed: {e}")
|
||||||
|
# Try to select it to see if it exists
|
||||||
|
try:
|
||||||
|
imap.select(folder)
|
||||||
|
created_folders.append(folder)
|
||||||
|
print(f" Folder {folder} already exists")
|
||||||
|
except:
|
||||||
|
pass
|
||||||
|
|
||||||
|
print(f"Available folders for {username}: {created_folders}")
|
||||||
|
|
||||||
|
# Populate each folder with messages
|
||||||
|
for folder in created_folders[:3]: # Limit to first 3 folders to avoid too many messages
|
||||||
|
imap.select(folder)
|
||||||
|
print(f"Creating messages in {folder} for {username}...")
|
||||||
|
|
||||||
|
# Create fewer messages per folder for testing
|
||||||
|
message_count = 3 if folder != 'INBOX' else 10
|
||||||
|
|
||||||
|
for i in range(1, message_count + 1):
|
||||||
|
if i % 3 == 0 and folder == 'INBOX':
|
||||||
|
# Every 3rd message has attachment (only in INBOX)
|
||||||
|
msg = create_message_with_attachment(
|
||||||
|
f"[{folder}] Test Message {i} with Attachment",
|
||||||
|
f"This is test message {i} in {folder} for {username} with an attachment.",
|
||||||
|
f"Sample attachment content for message {i} in {folder}",
|
||||||
|
"test-sender@example.com",
|
||||||
|
f"{username}@example.com"
|
||||||
|
)
|
||||||
|
print(f" Created message {i} (with attachment)")
|
||||||
|
else:
|
||||||
|
# Simple message
|
||||||
|
msg = create_simple_message(
|
||||||
|
f"[{folder}] Test Message {i}",
|
||||||
|
f"This is test message {i} in {folder} for {username}.",
|
||||||
|
"test-sender@example.com",
|
||||||
|
f"{username}@example.com"
|
||||||
|
)
|
||||||
|
print(f" Created message {i}")
|
||||||
|
|
||||||
|
# Append message to current folder
|
||||||
|
imap.append(folder, None, None, msg.encode('utf-8'))
|
||||||
|
time.sleep(0.1) # Small delay to avoid overwhelming
|
||||||
|
|
||||||
|
# Add special messages only to INBOX for keyword filtering tests
|
||||||
|
if folder == 'INBOX':
|
||||||
|
special_messages = [
|
||||||
|
("Important Meeting Reminder", "This is an important meeting message for testing keyword filters.", "manager@company.com", "team@company.com"),
|
||||||
|
("Urgent: System Maintenance", "Important notification about system maintenance.", "admin@company.com", f"{username}@example.com"),
|
||||||
|
("Regular Newsletter", "This is a regular newsletter message.", "newsletter@external.com", f"{username}@example.com"),
|
||||||
|
("Team Meeting Notes", "Meeting notes from the team.", "secretary@company.com", "support@company.com"),
|
||||||
|
("Message with Special Characters", "This message contains special characters: äöü ñ 中文 🚀", "test-sender@example.com", f"{username}@example.com")
|
||||||
|
]
|
||||||
|
|
||||||
|
for subject, body, sender, recipient in special_messages:
|
||||||
|
msg = create_simple_message(subject, body, sender, recipient)
|
||||||
|
imap.append(folder, None, None, msg.encode('utf-8'))
|
||||||
|
print(f" Created special message: {subject} from {sender} to {recipient}")
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
|
imap.logout()
|
||||||
|
print(f"✅ Successfully created messages across {len(created_folders[:3])} folders for {username}")
|
||||||
|
return True
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error populating {username}: {e}")
|
||||||
|
return False
|
||||||
|
|
||||||
|
def main():
|
||||||
|
print("🚀 Populating GreenMail with test messages using IMAP...")
|
||||||
|
|
||||||
|
# Test accounts
|
||||||
|
accounts = [
|
||||||
|
("testuser1", "password123"),
|
||||||
|
("testuser2", "password456"),
|
||||||
|
("syncuser", "syncpass"),
|
||||||
|
("archiveuser", "archivepass")
|
||||||
|
]
|
||||||
|
|
||||||
|
# Wait for GreenMail to be ready
|
||||||
|
print("Waiting for GreenMail to be ready...")
|
||||||
|
time.sleep(5)
|
||||||
|
|
||||||
|
success_count = 0
|
||||||
|
for username, password in accounts:
|
||||||
|
if populate_user_mailbox(username, password):
|
||||||
|
success_count += 1
|
||||||
|
time.sleep(1) # Brief pause between accounts
|
||||||
|
|
||||||
|
print(f"\n🎉 Successfully populated {success_count}/{len(accounts)} accounts!")
|
||||||
|
|
||||||
|
if success_count == len(accounts):
|
||||||
|
print("\n✅ All test accounts ready:")
|
||||||
|
for username, password in accounts:
|
||||||
|
print(f" - {username}:{password}@localhost")
|
||||||
|
print(f"\nGreenMail Services:")
|
||||||
|
print(f" - IMAP: localhost:3143")
|
||||||
|
print(f" - IMAPS: localhost:3993")
|
||||||
|
print(f" - SMTP: localhost:3025")
|
||||||
|
return 0
|
||||||
|
else:
|
||||||
|
print(f"\n❌ Failed to populate {len(accounts) - success_count} accounts")
|
||||||
|
return 1
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
sys.exit(main())
|
||||||
128
test/run-tests.sh
Executable file
128
test/run-tests.sh
Executable file
|
|
@ -0,0 +1,128 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Run basic integration tests with test containers
|
||||||
|
# This is a comprehensive test that handles its own setup and teardown
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
|
echo "🚀 Running basic integration tests..."
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Function to print colored output
|
||||||
|
print_status() {
|
||||||
|
echo -e "${GREEN}[INFO]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_warning() {
|
||||||
|
echo -e "${YELLOW}[WARN]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
print_error() {
|
||||||
|
echo -e "${RED}[ERROR]${NC} $1"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cleanup function
|
||||||
|
cleanup() {
|
||||||
|
print_status "Cleaning up test containers..."
|
||||||
|
podman-compose -f podman-compose.yml down -v 2>/dev/null || true
|
||||||
|
}
|
||||||
|
|
||||||
|
# Set up cleanup trap
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
# Start containers
|
||||||
|
print_status "Starting test containers..."
|
||||||
|
podman-compose -f podman-compose.yml up -d
|
||||||
|
|
||||||
|
# Wait for containers to be ready
|
||||||
|
print_status "Waiting for containers to be ready..."
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
# Check if CouchDB is ready
|
||||||
|
print_status "Checking CouchDB connectivity..."
|
||||||
|
timeout=30
|
||||||
|
while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
|
||||||
|
timeout=$((timeout - 1))
|
||||||
|
if [ $timeout -le 0 ]; then
|
||||||
|
print_error "CouchDB failed to start within 30 seconds"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
print_status "CouchDB is ready!"
|
||||||
|
|
||||||
|
# Check if IMAP server is ready
|
||||||
|
print_status "Checking IMAP server connectivity..."
|
||||||
|
timeout=30
|
||||||
|
while ! nc -z localhost 3143 > /dev/null 2>&1; do
|
||||||
|
timeout=$((timeout - 1))
|
||||||
|
if [ $timeout -le 0 ]; then
|
||||||
|
print_error "IMAP server failed to start within 30 seconds"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
print_status "IMAP server is ready!"
|
||||||
|
|
||||||
|
# Populate test messages
|
||||||
|
print_status "Populating test messages..."
|
||||||
|
python3 ./populate-greenmail.py
|
||||||
|
|
||||||
|
# Build mail2couch
|
||||||
|
print_status "Building mail2couch..."
|
||||||
|
cd ../go
|
||||||
|
go build -o mail2couch .
|
||||||
|
cd ../test
|
||||||
|
|
||||||
|
# Run mail2couch with test configuration
|
||||||
|
print_status "Running mail2couch with test configuration..."
|
||||||
|
../go/mail2couch -config config-test.json -max-messages 3
|
||||||
|
|
||||||
|
# Verify results
|
||||||
|
print_status "Verifying test results..."
|
||||||
|
|
||||||
|
# Check CouchDB databases were created (using correct database names with m2c prefix)
|
||||||
|
EXPECTED_DBS=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
|
||||||
|
|
||||||
|
for db in "${EXPECTED_DBS[@]}"; do
|
||||||
|
if curl -s "http://admin:password@localhost:5984/$db" | grep -q "\"db_name\":\"$db\""; then
|
||||||
|
print_status "✅ Database '$db' created successfully"
|
||||||
|
else
|
||||||
|
print_error "❌ Database '$db' was not created"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Check document counts
|
||||||
|
for db in "${EXPECTED_DBS[@]}"; do
|
||||||
|
doc_count=$(curl -s "http://admin:password@localhost:5984/$db" | grep -o '"doc_count":[0-9]*' | cut -d':' -f2)
|
||||||
|
if [ "$doc_count" -gt 0 ]; then
|
||||||
|
print_status "✅ Database '$db' contains $doc_count documents"
|
||||||
|
else
|
||||||
|
print_warning "⚠️ Database '$db' contains no documents"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
# Test sync mode by running again (should show incremental behavior)
|
||||||
|
print_status "Running mail2couch again to test incremental sync..."
|
||||||
|
../go/mail2couch -config config-test.json -max-messages 3
|
||||||
|
|
||||||
|
print_status "🎉 Basic integration tests completed successfully!"
|
||||||
|
|
||||||
|
# Show summary
|
||||||
|
print_status "Test Summary:"
|
||||||
|
echo " - IMAP Server: localhost:3143"
|
||||||
|
echo " - CouchDB: http://localhost:5984"
|
||||||
|
echo " - Test accounts: testuser1, syncuser, archiveuser"
|
||||||
|
echo " - Databases created: ${EXPECTED_DBS[*]}"
|
||||||
|
echo ""
|
||||||
|
echo "For more comprehensive tests, run:"
|
||||||
|
echo " - ./test-wildcard-patterns.sh (test folder pattern matching)"
|
||||||
|
echo " - ./test-incremental-sync.sh (test incremental synchronization)"
|
||||||
67
test/start-test-env.sh
Executable file
67
test/start-test-env.sh
Executable file
|
|
@ -0,0 +1,67 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Start test environment for manual testing
|
||||||
|
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
|
echo "🚀 Starting mail2couch test environment..."
|
||||||
|
|
||||||
|
# Start containers
|
||||||
|
echo "Starting containers..."
|
||||||
|
podman-compose -f podman-compose.yml up -d
|
||||||
|
|
||||||
|
# Wait for services
|
||||||
|
echo "Waiting for services to be ready..."
|
||||||
|
sleep 10
|
||||||
|
|
||||||
|
# Check CouchDB
|
||||||
|
echo "Checking CouchDB..."
|
||||||
|
timeout=30
|
||||||
|
while ! curl -s http://localhost:5984/_up > /dev/null 2>&1; do
|
||||||
|
timeout=$((timeout - 1))
|
||||||
|
if [ $timeout -le 0 ]; then
|
||||||
|
echo "❌ CouchDB failed to start"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
echo "✅ CouchDB is ready at http://localhost:5984"
|
||||||
|
|
||||||
|
# Check IMAP
|
||||||
|
echo "Checking IMAP server..."
|
||||||
|
timeout=30
|
||||||
|
while ! nc -z localhost 3143 > /dev/null 2>&1; do
|
||||||
|
timeout=$((timeout - 1))
|
||||||
|
if [ $timeout -le 0 ]; then
|
||||||
|
echo "❌ IMAP server failed to start"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
sleep 1
|
||||||
|
done
|
||||||
|
echo "✅ IMAP server is ready at localhost:3143"
|
||||||
|
|
||||||
|
# Populate test data
|
||||||
|
echo "Populating test messages..."
|
||||||
|
python3 ./populate-greenmail.py
|
||||||
|
|
||||||
|
echo ""
|
||||||
|
echo "🎉 Test environment is ready!"
|
||||||
|
echo ""
|
||||||
|
echo "Services:"
|
||||||
|
echo " - CouchDB: http://localhost:5984 (admin/password)"
|
||||||
|
echo " - CouchDB Web UI: http://localhost:5984/_utils"
|
||||||
|
echo " - IMAP Server: localhost:3143"
|
||||||
|
echo " - IMAPS Server: localhost:3993"
|
||||||
|
echo " - SMTP Server: localhost:3025"
|
||||||
|
echo ""
|
||||||
|
echo "Test accounts:"
|
||||||
|
echo " - testuser1:password123"
|
||||||
|
echo " - testuser2:password456"
|
||||||
|
echo " - syncuser:syncpass"
|
||||||
|
echo " - archiveuser:archivepass"
|
||||||
|
echo ""
|
||||||
|
echo "To run mail2couch:"
|
||||||
|
echo " cd ../go && ./mail2couch -config ../test/config-test.json"
|
||||||
|
echo ""
|
||||||
|
echo "To stop the environment:"
|
||||||
|
echo " ./stop-test-env.sh"
|
||||||
12
test/stop-test-env.sh
Executable file
12
test/stop-test-env.sh
Executable file
|
|
@ -0,0 +1,12 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Stop test environment
|
||||||
|
|
||||||
|
cd "$(dirname "$0")"
|
||||||
|
|
||||||
|
echo "🛑 Stopping mail2couch test environment..."
|
||||||
|
|
||||||
|
# Stop and remove containers
|
||||||
|
podman-compose -f podman-compose.yml down -v
|
||||||
|
|
||||||
|
echo "✅ Test environment stopped and cleaned up!"
|
||||||
242
test/test-incremental-sync.sh
Executable file
242
test/test-incremental-sync.sh
Executable file
|
|
@ -0,0 +1,242 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Test script to validate incremental sync functionality
|
||||||
|
# This script tests that mail2couch properly implements incremental synchronization
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🔄 Testing Incremental Sync Functionality"
|
||||||
|
echo "=========================================="
|
||||||
|
|
||||||
|
# Make sure we're in the right directory
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Function to check if containers are running
|
||||||
|
check_containers() {
|
||||||
|
echo "🔍 Checking if test containers are running..."
|
||||||
|
|
||||||
|
if ! podman ps | grep -q "greenmail"; then
|
||||||
|
echo -e "${RED}❌ GreenMail container not running${NC}"
|
||||||
|
echo "Please run: cd test && ./start-test-env.sh"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! podman ps | grep -q "couchdb"; then
|
||||||
|
echo -e "${RED}❌ CouchDB container not running${NC}"
|
||||||
|
echo "Please run: cd test && ./start-test-env.sh"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ Test containers are running${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to populate initial test data
|
||||||
|
populate_initial_data() {
|
||||||
|
echo "📧 Populating initial test data..."
|
||||||
|
cd test
|
||||||
|
if python3 populate-greenmail.py; then
|
||||||
|
echo -e "${GREEN}✅ Initial test data populated${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}❌ Failed to populate initial test data${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to build the application
|
||||||
|
build_app() {
|
||||||
|
echo "🔨 Building mail2couch..."
|
||||||
|
cd go
|
||||||
|
if go build -o mail2couch .; then
|
||||||
|
echo -e "${GREEN}✅ Build successful${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}❌ Build failed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to run first sync
|
||||||
|
run_first_sync() {
|
||||||
|
echo -e "\n${BLUE}Running first sync...${NC}"
|
||||||
|
cd go
|
||||||
|
./mail2couch -config ../test/config-test.json -max-messages 5
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to add new messages to test incremental sync
|
||||||
|
add_new_messages() {
|
||||||
|
echo -e "\n${YELLOW}Adding new messages for incremental sync test...${NC}"
|
||||||
|
|
||||||
|
# Create a simple Python script to add messages directly to GreenMail
|
||||||
|
cat > test/add_incremental_messages.py << 'EOF'
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
|
||||||
|
import imaplib
|
||||||
|
import time
|
||||||
|
from test.populate_greenmail import create_simple_message
|
||||||
|
|
||||||
|
def add_new_messages():
|
||||||
|
"""Add new messages to test incremental sync"""
|
||||||
|
accounts = [
|
||||||
|
("testuser1", "password123"),
|
||||||
|
("syncuser", "syncpass"),
|
||||||
|
("archiveuser", "archivepass")
|
||||||
|
]
|
||||||
|
|
||||||
|
for username, password in accounts:
|
||||||
|
try:
|
||||||
|
print(f"Adding new messages to {username}...")
|
||||||
|
imap = imaplib.IMAP4('localhost', 3143)
|
||||||
|
imap.login(username, password)
|
||||||
|
imap.select('INBOX')
|
||||||
|
|
||||||
|
# Add 3 new messages with timestamps after the first sync
|
||||||
|
for i in range(1, 4):
|
||||||
|
subject = f"Incremental Sync Test Message {i}"
|
||||||
|
body = f"This message was added after the first sync for incremental testing. Message {i} for {username}."
|
||||||
|
|
||||||
|
msg = create_simple_message(subject, body, f"incremental-test@example.com", f"{username}@example.com")
|
||||||
|
imap.append('INBOX', None, None, msg.encode('utf-8'))
|
||||||
|
print(f" Added: {subject}")
|
||||||
|
time.sleep(0.1)
|
||||||
|
|
||||||
|
imap.logout()
|
||||||
|
print(f"✅ Added 3 new messages to {username}")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"❌ Error adding messages to {username}: {e}")
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
add_new_messages()
|
||||||
|
EOF
|
||||||
|
|
||||||
|
# Add the parent directory to Python path and run the script
|
||||||
|
cd test
|
||||||
|
PYTHONPATH=.. python3 add_incremental_messages.py
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to run second sync (incremental)
|
||||||
|
run_incremental_sync() {
|
||||||
|
echo -e "\n${BLUE}Running incremental sync...${NC}"
|
||||||
|
cd go
|
||||||
|
./mail2couch -config ../test/config-test.json -max-messages 10
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to verify incremental sync results
|
||||||
|
verify_results() {
|
||||||
|
echo -e "\n${YELLOW}Verifying incremental sync results...${NC}"
|
||||||
|
|
||||||
|
# Check CouchDB for sync metadata documents
|
||||||
|
echo "Checking for sync metadata in CouchDB databases..."
|
||||||
|
|
||||||
|
# List of expected databases based on test config (with m2c prefix)
|
||||||
|
databases=("m2c_wildcard_all_folders_test" "m2c_work_pattern_test" "m2c_specific_folders_only")
|
||||||
|
|
||||||
|
for db in "${databases[@]}"; do
|
||||||
|
echo " Checking database: $db"
|
||||||
|
|
||||||
|
# Check if database exists
|
||||||
|
if curl -s -f "http://admin:password@localhost:5984/$db" > /dev/null; then
|
||||||
|
echo " ✅ Database exists"
|
||||||
|
|
||||||
|
# Look for sync metadata documents
|
||||||
|
metadata_docs=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"" | grep -o '"total_rows":[0-9]*' | cut -d: -f2 || echo "0")
|
||||||
|
|
||||||
|
if [ "$metadata_docs" -gt 0 ]; then
|
||||||
|
echo " ✅ Found sync metadata documents: $metadata_docs"
|
||||||
|
|
||||||
|
# Get a sample sync metadata document
|
||||||
|
sample_doc=$(curl -s "http://admin:password@localhost:5984/$db/_all_docs?startkey=\"sync_metadata\"&endkey=\"sync_metadata_z\"&include_docs=true&limit=1")
|
||||||
|
echo " Sample sync metadata:"
|
||||||
|
echo "$sample_doc" | python3 -m json.tool | grep -E "(lastSyncTime|lastMessageUID|messageCount)" | head -3
|
||||||
|
else
|
||||||
|
echo " ⚠️ No sync metadata documents found"
|
||||||
|
fi
|
||||||
|
else
|
||||||
|
echo " ❌ Database does not exist"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main test execution
|
||||||
|
main() {
|
||||||
|
echo "Starting incremental sync tests..."
|
||||||
|
|
||||||
|
# Pre-test setup
|
||||||
|
check_containers
|
||||||
|
build_app
|
||||||
|
|
||||||
|
# Clean up any existing data
|
||||||
|
echo "🧹 Cleaning up existing test data..."
|
||||||
|
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_wildcard_all_folders_test" > /dev/null || true
|
||||||
|
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_work_pattern_test" > /dev/null || true
|
||||||
|
curl -s -X DELETE "http://admin:password@localhost:5984/m2c_specific_folders_only" > /dev/null || true
|
||||||
|
|
||||||
|
# Step 1: Populate initial test data
|
||||||
|
populate_initial_data
|
||||||
|
|
||||||
|
# Wait for data to settle
|
||||||
|
echo "⏳ Waiting for initial data to settle..."
|
||||||
|
sleep 5
|
||||||
|
|
||||||
|
# Step 2: Run first sync to establish baseline
|
||||||
|
echo -e "\n${YELLOW}=== STEP 1: First Sync (Baseline) ===${NC}"
|
||||||
|
run_first_sync
|
||||||
|
|
||||||
|
# Wait between syncs
|
||||||
|
echo "⏳ Waiting between syncs..."
|
||||||
|
sleep 3
|
||||||
|
|
||||||
|
# Step 3: Add new messages for incremental sync test
|
||||||
|
echo -e "\n${YELLOW}=== STEP 2: Add New Messages ===${NC}"
|
||||||
|
add_new_messages
|
||||||
|
|
||||||
|
# Wait for new messages to be ready
|
||||||
|
echo "⏳ Waiting for new messages to be ready..."
|
||||||
|
sleep 2
|
||||||
|
|
||||||
|
# Step 4: Run incremental sync
|
||||||
|
echo -e "\n${YELLOW}=== STEP 3: Incremental Sync ===${NC}"
|
||||||
|
run_incremental_sync
|
||||||
|
|
||||||
|
# Step 5: Verify results
|
||||||
|
echo -e "\n${YELLOW}=== STEP 4: Verification ===${NC}"
|
||||||
|
verify_results
|
||||||
|
|
||||||
|
echo -e "\n${GREEN}🎉 Incremental sync test completed!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "Key features tested:"
|
||||||
|
echo " ✅ Sync metadata storage and retrieval"
|
||||||
|
echo " ✅ IMAP SEARCH with SINCE for efficient incremental fetching"
|
||||||
|
echo " ✅ Last sync timestamp tracking per mailbox"
|
||||||
|
echo " ✅ Proper handling of first sync vs incremental sync"
|
||||||
|
echo ""
|
||||||
|
echo "To verify results manually:"
|
||||||
|
echo " - Check CouchDB: http://localhost:5984/_utils"
|
||||||
|
echo " - Look for 'sync_metadata_*' documents in each database"
|
||||||
|
echo " - Verify incremental messages were added after baseline sync"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cleanup function
|
||||||
|
cleanup() {
|
||||||
|
echo "🧹 Cleaning up test artifacts..."
|
||||||
|
rm -f test/add_incremental_messages.py
|
||||||
|
}
|
||||||
|
|
||||||
|
# Set trap to cleanup on exit
|
||||||
|
trap cleanup EXIT
|
||||||
|
|
||||||
|
# Run main function if executed directly
|
||||||
|
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||||
|
main "$@"
|
||||||
|
fi
|
||||||
137
test/test-wildcard-patterns.sh
Executable file
137
test/test-wildcard-patterns.sh
Executable file
|
|
@ -0,0 +1,137 @@
|
||||||
|
#!/bin/bash
|
||||||
|
|
||||||
|
# Test script to validate wildcard folder pattern functionality
|
||||||
|
# This script tests the various wildcard patterns against the test environment
|
||||||
|
|
||||||
|
set -e
|
||||||
|
|
||||||
|
echo "🧪 Testing Wildcard Folder Pattern Functionality"
|
||||||
|
echo "================================================"
|
||||||
|
|
||||||
|
# Make sure we're in the right directory
|
||||||
|
cd "$(dirname "$0")/.."
|
||||||
|
|
||||||
|
# Colors for output
|
||||||
|
RED='\033[0;31m'
|
||||||
|
GREEN='\033[0;32m'
|
||||||
|
YELLOW='\033[1;33m'
|
||||||
|
BLUE='\033[0;34m'
|
||||||
|
NC='\033[0m' # No Color
|
||||||
|
|
||||||
|
# Function to run mail2couch with a specific config and capture output
|
||||||
|
run_test() {
|
||||||
|
local test_name="$1"
|
||||||
|
local config_file="$2"
|
||||||
|
local max_messages="$3"
|
||||||
|
|
||||||
|
echo -e "\n${BLUE}Testing: $test_name${NC}"
|
||||||
|
echo "Config: $config_file"
|
||||||
|
echo "Max messages: $max_messages"
|
||||||
|
echo "----------------------------------------"
|
||||||
|
|
||||||
|
# Run mail2couch and capture output
|
||||||
|
cd go
|
||||||
|
if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then
|
||||||
|
echo -e "${GREEN}✅ Test completed successfully${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}❌ Test failed${NC}"
|
||||||
|
return 1
|
||||||
|
fi
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to check if containers are running
|
||||||
|
check_containers() {
|
||||||
|
echo "🔍 Checking if test containers are running..."
|
||||||
|
|
||||||
|
if ! podman ps | grep -q "greenmail"; then
|
||||||
|
echo -e "${RED}❌ GreenMail container not running${NC}"
|
||||||
|
echo "Please run: cd test && ./start-test-env.sh"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
if ! podman ps | grep -q "couchdb"; then
|
||||||
|
echo -e "${RED}❌ CouchDB container not running${NC}"
|
||||||
|
echo "Please run: cd test && ./start-test-env.sh"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo -e "${GREEN}✅ Test containers are running${NC}"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to populate test data
|
||||||
|
populate_test_data() {
|
||||||
|
echo "📧 Populating test data..."
|
||||||
|
cd test
|
||||||
|
if python3 populate-greenmail.py; then
|
||||||
|
echo -e "${GREEN}✅ Test data populated successfully${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}❌ Failed to populate test data${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Function to build the application
|
||||||
|
build_app() {
|
||||||
|
echo "🔨 Building mail2couch..."
|
||||||
|
cd go
|
||||||
|
if go build -o mail2couch .; then
|
||||||
|
echo -e "${GREEN}✅ Build successful${NC}"
|
||||||
|
else
|
||||||
|
echo -e "${RED}❌ Build failed${NC}"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
cd ..
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main test execution
|
||||||
|
main() {
|
||||||
|
echo "Starting wildcard pattern tests..."
|
||||||
|
|
||||||
|
# Pre-test checks
|
||||||
|
check_containers
|
||||||
|
build_app
|
||||||
|
populate_test_data
|
||||||
|
|
||||||
|
# Wait a moment for test data to be fully ready
|
||||||
|
echo "⏳ Waiting for test data to settle..."
|
||||||
|
sleep 3
|
||||||
|
|
||||||
|
# Test 1: Wildcard all folders (*)
|
||||||
|
echo -e "\n${YELLOW}Test 1: Wildcard All Folders Pattern (*)${NC}"
|
||||||
|
echo "Expected: Should process all folders except Drafts and Trash"
|
||||||
|
run_test "Wildcard All Folders" "config-test.json" 3
|
||||||
|
|
||||||
|
# Test 2: Work pattern (Work*)
|
||||||
|
echo -e "\n${YELLOW}Test 2: Work Pattern (Work*)${NC}"
|
||||||
|
echo "Expected: Should process Work/Projects, Work/Archive but not Work/Temp (excluded by *Temp*)"
|
||||||
|
run_test "Work Pattern" "config-test.json" 3
|
||||||
|
|
||||||
|
# Test 3: Specific folders only
|
||||||
|
echo -e "\n${YELLOW}Test 3: Specific Folders Only${NC}"
|
||||||
|
echo "Expected: Should only process INBOX, Sent, and Personal folders"
|
||||||
|
run_test "Specific Folders" "config-test.json" 3
|
||||||
|
|
||||||
|
# Test 4: Advanced wildcard examples
|
||||||
|
echo -e "\n${YELLOW}Test 4: Advanced Wildcard Examples${NC}"
|
||||||
|
echo "Expected: Various complex patterns should work correctly"
|
||||||
|
run_test "Advanced Patterns" "config-wildcard-examples.json" 2
|
||||||
|
|
||||||
|
echo -e "\n${GREEN}🎉 All wildcard pattern tests completed!${NC}"
|
||||||
|
echo ""
|
||||||
|
echo "To verify results, check the CouchDB databases:"
|
||||||
|
echo " http://localhost:5984/_utils"
|
||||||
|
echo ""
|
||||||
|
echo "Expected databases should be created for each account:"
|
||||||
|
echo " - wildcard_all_folders_test"
|
||||||
|
echo " - work_pattern_test"
|
||||||
|
echo " - specific_folders_only"
|
||||||
|
echo ""
|
||||||
|
echo "Each database should contain documents with 'mailbox' field showing origin folder."
|
||||||
|
}
|
||||||
|
|
||||||
|
# Run main function if executed directly
|
||||||
|
if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
|
||||||
|
main "$@"
|
||||||
|
fi
|
||||||
Loading…
Add table
Add a link
Reference in a new issue