mail2couch/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations:
- **Go implementation**: Located in `/go/` directory, builds as `mail2couch-go`
- **Rust implementation**: Located in `/rust/` directory, builds as `mail2couch-rs` (fully functional, more advanced)

## Development Commands

### Universal Build Commands (Recommended)

```bash
# Build both implementations
just build

# Build individual implementations
just build-go      # Builds go/mail2couch-go
just build-rust    # Builds rust/target/release/mail2couch-rs

# Build optimized release versions
just build-release

# Install both binaries to ~/bin (user-local)
just install

# Install both binaries to /usr/local/bin (system-wide)
sudo just system-install

# Uninstall from ~/bin
just uninstall

# Uninstall from /usr/local/bin
sudo just system-uninstall

# Run tests for both implementations
just test

# Clean all build artifacts
just clean

# Format and check code
just fmt
just check

# Show binary sizes
just sizes

# Compare versions
just versions

# List all available recipes
just --list
```

### Go Implementation

```bash
# Build the application
cd go && go build -o mail2couch-go .

# Run the application with automatic config discovery
cd go && ./mail2couch-go

# Run with specific config file
cd go && ./mail2couch-go --config /path/to/config.json

# Run with message limit (useful for large mailboxes)
cd go && ./mail2couch-go --max-messages 100

# Run with dry-run mode
cd go && ./mail2couch-go --dry-run

# Run linting/static analysis
cd go && go vet ./...

# Run unit tests
cd go && go test ./...

# Check dependencies
cd go && go mod tidy
```

### Rust Implementation

```bash
# Build the application (release mode recommended)
cd rust && cargo build --release

# Run the application with automatic config discovery
cd rust && ./target/release/mail2couch-rs

# Run with specific config file
cd rust && ./target/release/mail2couch-rs --config /path/to/config.json

# Run with message limit
cd rust && ./target/release/mail2couch-rs --max-messages 100

# Run with dry-run mode
cd rust && ./target/release/mail2couch-rs --dry-run

# Run linting/static analysis
cd rust && cargo clippy -- -D warnings

# Run unit tests
cd rust && cargo test

# Format code
cd rust && cargo fmt
```

### Integration Testing

```bash
# Run integration tests with Podman containers
cd test && ./run-tests.sh

# Run specialized tests
cd test && ./test-wildcard-patterns.sh
cd test && ./test-incremental-sync.sh
```

## Architecture

### Core Components

1. **Configuration (`config/`)**: JSON-based configuration system
   - Supports multiple mail sources with filtering options
   - CouchDB connection settings
   - Each source can have folder and message filters

2. **Mail Handling (`mail/`)**: IMAP client implementation
   - Uses `github.com/emersion/go-imap/v2` for IMAP operations
   - Supports TLS connections
   - Fetches and processes email messages from IMAP mailboxes

3. **CouchDB Integration (`couch/`)**: Database operations
   - Uses `github.com/go-kivik/kivik/v4` as CouchDB driver
   - Handles database creation and document management
   - Defines `MailDocument` structure for email storage

### Configuration Structure

The application uses `config.json` for configuration with the following structure:
- `couchDb`: Database connection settings (URL, credentials)
- `mailSources`: Array of mail sources with individual settings:
  - Protocol support (currently only IMAP)
  - Connection details (host, port, credentials)
  - `mode`: Either "sync" or "archive" (defaults to "archive" if not specified)
    - **sync**: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB)
    - **archive**: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents)
  - Filtering options for folders and messages with wildcard support
  - Enable/disable per source

### Configuration File Discovery

The application automatically searches for configuration files in the following order:
1. Path specified by `--config`/`-c` command line flag
2. `./config.json` (current working directory)
3. `./config/config.json` (config subdirectory)
4. `~/.config/mail2couch/config.json` (user XDG config directory)
5. `~/.mail2couch.json` (user home directory)

This design ensures the same `config.json` format will work for both Go and Rust implementations.

### Current Implementation Status

#### Both Implementations
- ✅ Configuration loading with automatic file discovery
- ✅ Command line flag support (--config/-c, --max-messages/-m, --dry-run/-n)
- ✅ Per-account CouchDB database creation and management
- ✅ IMAP connection and mailbox listing
- ✅ Real IMAP message retrieval and parsing
- ✅ Email storage to CouchDB framework with native attachments
- ✅ Folder filtering logic with wildcard support (`*`, `?`, `[abc]` patterns)
- ✅ Date filtering support
- ✅ Keyword filtering support (subject, sender, recipient keywords)
- ✅ Duplicate detection and prevention
- ✅ Sync vs Archive mode implementation
- ✅ CouchDB attachment storage for email attachments
- ✅ Full message body and attachment handling with MIME multipart support
- ✅ Per-account CouchDB databases for better organization
- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking
- ✅ Comprehensive --dry-run mode for safe configuration testing

#### Rust Implementation Additional Features
- ✅ Asynchronous processing with concurrent network operations
- ✅ Server-side IMAP keyword filtering (more efficient)
- ✅ Automatic retry logic with exponential backoff
- ✅ Structured error handling with detailed context
- ✅ Enhanced CLI with rich help system
- ✅ Comprehensive unit test coverage
- ✅ Emoji-enhanced logging for better user experience

#### Go Implementation Characteristics
- ✅ Sequential processing (simple and reliable)
- ✅ Minimal dependencies and fast compilation
- ✅ Client-side keyword filtering
- ✅ Basic error handling with continue-on-error semantics

### Key Dependencies

#### Go Implementation
- `github.com/emersion/go-imap/v2`: IMAP client library
- `github.com/go-kivik/kivik/v4`: CouchDB client library
- `github.com/spf13/pflag`: GNU-style command line flags
- `github.com/emersion/go-message`: Email parsing

#### Rust Implementation
- `async-imap`: Asynchronous IMAP client
- `reqwest`: HTTP client for CouchDB API
- `tokio`: Async runtime and utilities
- `clap`: Command line argument parsing
- `serde`: Serialization framework
- `mail-parser`: Email parsing with MIME support

### Incremental Sync Implementation

The application implements intelligent incremental synchronization to avoid re-processing messages:

- **Sync Metadata Storage**: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed
- **IMAP SEARCH Integration**: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages
- **Per-Mailbox Tracking**: Sync state is tracked independently for each mailbox in each account
- **Fallback Behavior**: Gracefully falls back to fetching recent messages if IMAP SEARCH fails
- **First Sync Handling**: Initial sync can use config `since` date or perform full sync

Sync metadata documents are stored in CouchDB with ID format: `sync_metadata_{mailbox}` and include:
- `lastSyncTime`: When this mailbox was last successfully synced
- `lastMessageUID`: Highest UID processed in the last sync
- `messageCount`: Number of messages processed in the last sync

### Development Notes

- The main entry point is `main.go` which orchestrates the configuration loading, CouchDB setup, and mail source processing
- Each mail source gets its own CouchDB database named using `GenerateAccountDBName()` function with `m2c_` prefix
- Each mail source is processed sequentially with proper error handling
- The application uses real IMAP message parsing with go-message library for full email processing
- Message filtering by folder (wildcard patterns), date (since), and keywords is implemented
- Duplicate detection prevents re-storing existing messages
- Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account
- Email attachments are stored as native CouchDB attachments linked to the email document
- Comprehensive test environment with Podman containers and automated test scripts
- The application uses automatic config file discovery as documented above


### Next Steps

The following enhancements could further improve the implementation:

1. **Error Recovery**: Add retry logic for network failures and partial sync recovery
2. **Performance Optimization**: Add batch operations for better CouchDB insertion performance
3. **Unit Testing**: Add comprehensive unit tests for all major components
4. **Advanced Filtering**: Add support for more complex filter expressions and regex patterns
5. **Monitoring**: Add metrics and logging for production deployment
6. **Configuration Validation**: Enhanced validation for configuration files
7. **Multi-threading**: Parallel processing of multiple mailboxes or accounts

## Development Guidelines

### Code Quality and Standards
- All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly