mail2couch/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

mail2couch is a utility for backing up mail from various sources (primarily IMAP) to CouchDB. The project supports two implementations:

• Go implementation: Located in /go/ directory, builds as mail2couch-go
• Rust implementation: Located in /rust/ directory, builds as mail2couch-rs (fully functional, more advanced)

Development Commands

Universal Build Commands (Recommended)

# Build both implementations
just build

# Build individual implementations
just build-go      # Builds go/mail2couch-go
just build-rust    # Builds rust/target/release/mail2couch-rs

# Build optimized release versions
just build-release

# Install both binaries to ~/bin (user-local)
just install

# Install both binaries to /usr/local/bin (system-wide)
sudo just system-install

# Uninstall from ~/bin
just uninstall

# Uninstall from /usr/local/bin
sudo just system-uninstall

# Run tests for both implementations
just test

# Clean all build artifacts
just clean

# Format and check code
just fmt
just check

# Show binary sizes
just sizes

# Compare versions
just versions

# List all available recipes
just --list

Go Implementation

# Build the application
cd go && go build -o mail2couch-go .

# Run the application with automatic config discovery
cd go && ./mail2couch-go

# Run with specific config file
cd go && ./mail2couch-go --config /path/to/config.json

# Run with message limit (useful for large mailboxes)
cd go && ./mail2couch-go --max-messages 100

# Run with dry-run mode
cd go && ./mail2couch-go --dry-run

# Run linting/static analysis
cd go && go vet ./...

# Run unit tests
cd go && go test ./...

# Check dependencies
cd go && go mod tidy

Rust Implementation

# Build the application (release mode recommended)
cd rust && cargo build --release

# Run the application with automatic config discovery
cd rust && ./target/release/mail2couch-rs

# Run with specific config file
cd rust && ./target/release/mail2couch-rs --config /path/to/config.json

# Run with message limit
cd rust && ./target/release/mail2couch-rs --max-messages 100

# Run with dry-run mode
cd rust && ./target/release/mail2couch-rs --dry-run

# Run linting/static analysis
cd rust && cargo clippy -- -D warnings

# Run unit tests
cd rust && cargo test

# Format code
cd rust && cargo fmt

Integration Testing

# Run integration tests with Podman containers
cd test && ./run-tests.sh

# Run specialized tests
cd test && ./test-wildcard-patterns.sh
cd test && ./test-incremental-sync.sh

Architecture

Core Components

1. Configuration (config/): JSON-based configuration system

   • Supports multiple mail sources with filtering options
   • CouchDB connection settings
   • Each source can have folder and message filters

2. Mail Handling (mail/): IMAP client implementation

   • Uses github.com/emersion/go-imap/v2 for IMAP operations
   • Supports TLS connections
   • Fetches and processes email messages from IMAP mailboxes

3. CouchDB Integration (couch/): Database operations

   • Uses github.com/go-kivik/kivik/v4 as CouchDB driver
   • Handles database creation and document management
   • Defines MailDocument structure for email storage

Configuration Structure

The application uses config.json for configuration with the following structure:

• couchDb: Database connection settings (URL, credentials)
• mailSources: Array of mail sources with individual settings:
  • Protocol support (currently only IMAP)
  • Connection details (host, port, credentials)
  • mode: Either "sync" or "archive" (defaults to "archive" if not specified)
    • sync: 1-to-1 relationship - CouchDB documents match exactly what's in the mail account (may remove documents from CouchDB)
    • archive: Archive mode - CouchDB keeps all messages ever seen, even if deleted from mail account (never removes documents)
  • Filtering options for folders and messages with wildcard support
  • Enable/disable per source

Configuration File Discovery

The application automatically searches for configuration files in the following order:

1. Path specified by --config/-c command line flag
2. ./config.json (current working directory)
3. ./config/config.json (config subdirectory)
4. ~/.config/mail2couch/config.json (user XDG config directory)
5. ~/.mail2couch.json (user home directory)

This design ensures the same config.json format will work for both Go and Rust implementations.

Current Implementation Status

Both Implementations

• ✅ Configuration loading with automatic file discovery
• ✅ Command line flag support (--config/-c, --max-messages/-m, --dry-run/-n)
• ✅ Per-account CouchDB database creation and management
• ✅ IMAP connection and mailbox listing
• ✅ Real IMAP message retrieval and parsing
• ✅ Email storage to CouchDB framework with native attachments
• ✅ Folder filtering logic with wildcard support (*, ?, [abc] patterns)
• ✅ Date filtering support
• ✅ Keyword filtering support (subject, sender, recipient keywords)
• ✅ Duplicate detection and prevention
• ✅ Sync vs Archive mode implementation
• ✅ CouchDB attachment storage for email attachments
• ✅ Full message body and attachment handling with MIME multipart support
• ✅ Per-account CouchDB databases for better organization
• ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking
• ✅ Comprehensive --dry-run mode for safe configuration testing

Rust Implementation Additional Features

• ✅ Asynchronous processing with concurrent network operations
• ✅ Server-side IMAP keyword filtering (more efficient)
• ✅ Automatic retry logic with exponential backoff
• ✅ Structured error handling with detailed context
• ✅ Enhanced CLI with rich help system
• ✅ Comprehensive unit test coverage
• ✅ Emoji-enhanced logging for better user experience

Go Implementation Characteristics

• ✅ Sequential processing (simple and reliable)
• ✅ Minimal dependencies and fast compilation
• ✅ Client-side keyword filtering
• ✅ Basic error handling with continue-on-error semantics

Key Dependencies

Go Implementation

• github.com/emersion/go-imap/v2: IMAP client library
• github.com/go-kivik/kivik/v4: CouchDB client library
• github.com/spf13/pflag: GNU-style command line flags
• github.com/emersion/go-message: Email parsing

Rust Implementation

• async-imap: Asynchronous IMAP client
• reqwest: HTTP client for CouchDB API
• tokio: Async runtime and utilities
• clap: Command line argument parsing
• serde: Serialization framework
• mail-parser: Email parsing with MIME support

Incremental Sync Implementation

The application implements intelligent incremental synchronization to avoid re-processing messages:

• Sync Metadata Storage: Each mailbox sync operation stores metadata including last sync timestamp and highest UID processed
• IMAP SEARCH Integration: Uses IMAP SEARCH with SINCE criteria for efficient server-side filtering of new messages
• Per-Mailbox Tracking: Sync state is tracked independently for each mailbox in each account
• Fallback Behavior: Gracefully falls back to fetching recent messages if IMAP SEARCH fails
• First Sync Handling: Initial sync can use config since date or perform full sync

Sync metadata documents are stored in CouchDB with ID format: sync_metadata_{mailbox} and include:

• lastSyncTime: When this mailbox was last successfully synced
• lastMessageUID: Highest UID processed in the last sync
• messageCount: Number of messages processed in the last sync

Development Notes

• The main entry point is main.go which orchestrates the configuration loading, CouchDB setup, and mail source processing
• Each mail source gets its own CouchDB database named using GenerateAccountDBName() function with m2c_ prefix
• Each mail source is processed sequentially with proper error handling
• The application uses real IMAP message parsing with go-message library for full email processing
• Message filtering by folder (wildcard patterns), date (since), and keywords is implemented
• Duplicate detection prevents re-storing existing messages
• Sync vs Archive mode determines whether to remove documents from CouchDB when they're no longer in the mail account
• Email attachments are stored as native CouchDB attachments linked to the email document
• Comprehensive test environment with Podman containers and automated test scripts
• The application uses automatic config file discovery as documented above

Next Steps

The following enhancements could further improve the implementation:

1. Error Recovery: Add retry logic for network failures and partial sync recovery
2. Performance Optimization: Add batch operations for better CouchDB insertion performance
3. Unit Testing: Add comprehensive unit tests for all major components
4. Advanced Filtering: Add support for more complex filter expressions and regex patterns
5. Monitoring: Add metrics and logging for production deployment
6. Configuration Validation: Enhanced validation for configuration files
7. Multi-threading: Parallel processing of multiple mailboxes or accounts

Development Guidelines

Code Quality and Standards

• All code requires perfect linting and tool-formatting, exceptions are allowed only if documented properly