diff --git a/ANALYSIS.md b/ANALYSIS.md new file mode 100644 index 0000000..64f46c6 --- /dev/null +++ b/ANALYSIS.md @@ -0,0 +1,112 @@ +### Comprehensive Analysis of `mail2couch` Implementations + +This document provides an updated, in-depth analysis of the `mail2couch` project, integrating findings from the original `ANALYSIS.md` with a fresh review of the current Go and Rust codebases. It evaluates the current state, compares the two implementations, and outlines a roadmap for future improvements. + +--- + +### 1. Current State of the Implementations + +The project currently consists of two distinct implementations of the same core tool. + +* **The Go Implementation**: This is a mature, functional, and straightforward command-line tool. It is built on a simple, sequential architecture and effectively synchronizes emails from IMAP servers to CouchDB. It serves as a solid baseline for the project's core functionality. + +* **The Rust Implementation**: Contrary to the description in the original `ANALYSIS.md`, the Rust version is **no longer a non-functional placeholder**. It is now a complete, and in many ways, more advanced alternative to the Go version. It is built on a highly modular, asynchronous architecture, prioritizing performance, robustness, and an expanded feature set. + +--- + +### 2. Analysis of Points from Original `ANALYSIS.md` + +Several key issues and suggestions were raised in the original analysis. Here is their current status: + +* **`Incomplete Rust Implementation`**: **(Addressed)** The Rust implementation is now fully functional and surpasses the Go version in features and robustness. +* **`Performance for Large-Scale Use (Concurrency)`**: **(Addressed in Rust)** The Go version remains sequential. The Rust version, however, is fully asynchronous, allowing for concurrent network operations, which directly addresses this performance concern. +* **`Inefficient Keyword Filtering`**: **(Addressed in Rust)** The Go version still performs keyword filtering client-side. The Rust version implements server-side filtering using `IMAP SEARCH` with keywords, which is significantly more efficient. +* **`Primary Weakness: Security`**: **(Still an Issue)** Both implementations still require plaintext passwords in the configuration file. This remains a primary weakness. +* **`Missing Core Feature: Web Interface`**: **(Still an Issue)** This feature has not been implemented in either version. +* **`Usability Enhancement: Dry-Run Mode`**: **(✅ Resolved)** Both implementations now include comprehensive `--dry-run/-n` mode functionality that allows safe configuration testing without making any CouchDB changes. + +--- + +### 3. Comparative Analysis: Go vs. Rust + +#### **The Go Version** + +* **Pros**: + * **Simplicity**: The code is sequential and easy to follow, making it highly approachable for new contributors. + * **Stability**: It provides a solid, functional baseline that effectively accomplishes the core mission of the project. + * **Fast Compilation**: Quick compile times make for a fast development cycle. + * **Dry-Run Support**: Now includes comprehensive `--dry-run` mode for safe configuration testing. +* **Cons**: + * **Performance**: The lack of concurrency makes it slow for users with multiple accounts or large mailboxes. + * **Inefficiency**: Client-side keyword filtering wastes bandwidth and processing time. + * **Basic Error Handling**: The absence of retry logic makes it brittle in the face of transient network errors. + +#### **The Rust Version** + +* **Pros**: + * **Performance**: The `async` architecture provides superior performance through concurrency. + * **Robustness**: Automatic retry logic for network calls makes it highly resilient to temporary failures. + * **Feature-Rich**: Implements more efficient server-side filtering, better folder-matching logic, and a more professional CLI. + * **Safety & Maintainability**: The modular design and Rust's compile-time guarantees make the code safer and easier to maintain and extend. + * **Comprehensive Dry-Run**: Includes detailed `--dry-run` mode with enhanced simulation logging and summary reporting. +* **Cons**: + * **Complexity**: The codebase is significantly more complex due to its asynchronous nature, abstract design, and the inherent learning curve of Rust. + * **Slower Compilation**: Longer compile times can slow down development. + +--- + +### 4. Recent Implementation Updates + +#### **Dry-Run Mode Implementation (August 2025)** + +Both Go and Rust implementations now include comprehensive `--dry-run` functionality: + +##### **Go Implementation Features:** +- **CLI Integration**: Added `--dry-run/-n` flag using pflag with GNU-style options +- **Comprehensive Skipping**: All CouchDB write operations bypassed in dry-run mode +- **IMAP Preservation**: Maintains full IMAP operations for realistic email discovery +- **Detailed Simulation**: Shows what would be done with informative logging +- **Enhanced Reporting**: Clear distinction between dry-run and normal mode output +- **Bash Completion**: Updated completion script includes new flag + +##### **Rust Implementation Features:** +- **CLI Integration**: Added `--dry-run/-n` flag using clap with structured argument parsing +- **Advanced Simulation**: Detailed logging of what would be stored including message subjects +- **Async-Safe Skipping**: All async CouchDB operations properly bypassed +- **Enhanced Summary**: Comprehensive dry-run vs normal mode reporting with emoji indicators +- **Test Coverage**: All tests updated to include new dry_run field + +##### **Implementation Benefits:** +- **Risk Mitigation**: Users can validate configurations without database changes +- **Debugging Aid**: Shows exactly what emails would be processed and stored +- **Development Tool**: Enables safe testing of configuration changes +- **Documentation**: Demonstrates the full sync process without side effects + +This addresses the critical usability requirement identified in the original analysis and significantly improves the user experience for configuration validation and troubleshooting. + +--- + +### 5. Future Improvements and Missing Features + +This roadmap combines suggestions from both analyses, prioritizing the most impactful changes. + +#### **Tier 1: Critical Needs** + +1. **Fix the Security Model (Both)**: This is the most urgent issue. + * **Short-Term**: Add support for reading credentials from environment variables (e.g., `M2C_IMAP_PASSWORD`). + * **Long-Term**: Implement OAuth2 for modern providers like Gmail and Outlook. This is the industry standard and eliminates the need to store passwords. +2. **Implement a Web Interface (Either)**: As noted in the original analysis, this is the key missing feature for making the archived data useful. This would involve creating CouchDB design documents and a simple web server to render the views. +3. ~~**Add a `--dry-run` Mode (Both)**~~: **✅ COMPLETED** - Both implementations now include comprehensive dry-run functionality with the `--dry-run/-n` flag that allows users to test their configuration safely before making any changes to their database. + +#### **Tier 2: High-Impact Enhancements** + +1. **Add Concurrency to the Go Version**: To bring the Go implementation closer to the performance of the Rust version, it should be updated to use goroutines to process accounts and/or mailboxes in parallel. +2. **Improve Attachment Handling in Rust**: The `TODO` in the Rust IMAP client for parsing binary attachments should be completed to ensure all attachment types are saved correctly. +3. **URL-Encode Document IDs in Rust**: The CouchDB client in the Rust version should URL-encode document IDs to prevent errors when mailbox names contain special characters. +4. **Add Progress Indicators (Rust)**: For a better user experience during long syncs, the Rust version would benefit greatly from progress bars (e.g., using the `indicatif` crate). + +#### **Tier 3: "Nice-to-Have" Features** + +1. **Interactive Setup (Either)**: A `mail2couch setup` command to interactively generate the `config.json` file would significantly improve first-time user experience. +2. **Support for Other Protocols/Backends (Either)**: Extend the tool to support POP3 or JMAP, or to use other databases like PostgreSQL or Elasticsearch as a storage backend. +3. **Backfill Command (Either)**: A `--backfill-all` flag to ignore existing sync metadata and perform a complete re-sync of an account. \ No newline at end of file diff --git a/docs/FOLDER_PATTERNS.md b/FOLDER_PATTERNS.md similarity index 100% rename from docs/FOLDER_PATTERNS.md rename to FOLDER_PATTERNS.md diff --git a/IMPLEMENTATION_COMPARISON.md b/IMPLEMENTATION_COMPARISON.md new file mode 100644 index 0000000..4288f27 --- /dev/null +++ b/IMPLEMENTATION_COMPARISON.md @@ -0,0 +1,560 @@ +# Go vs Rust Implementation Comparison + +This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch, helping users and developers choose the best implementation for their needs. + +## Executive Summary + +The mail2couch project offers two distinct architectural approaches to email backup: + +- **Go Implementation**: A straightforward, sequential approach emphasizing simplicity and ease of understanding +- **Rust Implementation**: A sophisticated, asynchronous architecture prioritizing performance, reliability, and advanced features + +**Key Finding**: The Rust implementation (~3,056 LOC across 9 modules) is significantly more feature-complete and architecturally advanced than the Go implementation (~1,355 LOC across 4 modules), representing a mature evolution rather than a simple port. + +--- + +## Architecture & Design Philosophy + +### Go Implementation: Sequential Simplicity + +**Design Philosophy**: Straightforward, imperative programming with minimal abstraction + +- **Processing Model**: Sequential processing of sources → mailboxes → messages +- **Error Handling**: Basic error propagation with continue-on-error for non-critical failures +- **Modularity**: Simple package structure (`config`, `couch`, `mail`, `main`) +- **State Management**: Minimal state, mostly function-based operations + +```go +// Example: Sequential processing approach +func processImapSource(source *config.MailSource, couchClient *couch.Client, + dbName string, maxMessages int, dryRun bool) error { + // Connect to IMAP server + imapClient, err := mail.NewImapClient(source) + if err != nil { + return fmt.Errorf("failed to connect to IMAP server: %w", err) + } + defer imapClient.Logout() + + // Process each mailbox sequentially + for _, mailbox := range mailboxes { + // Process messages one by one + messages, currentUIDs, err := imapClient.GetMessages(...) + // Store messages synchronously + } +} +``` + +### Rust Implementation: Async Orchestration + +**Design Philosophy**: Modular, type-safe architecture with comprehensive error handling + +- **Processing Model**: Asynchronous coordination with concurrent network operations +- **Error Handling**: Sophisticated retry logic, structured error types, graceful degradation +- **Modularity**: Well-separated concerns (`cli`, `config`, `couch`, `imap`, `sync`, `filters`, `schemas`) +- **State Management**: Stateful coordinator pattern with proper resource management + +```rust +// Example: Asynchronous coordination approach +impl SyncCoordinator { + pub async fn sync_all_sources(&mut self) -> Result> { + let mut results = Vec::new(); + let sources = self.config.mail_sources.clone(); + + for source in &sources { + if !source.enabled { + info!("Skipping disabled source: {}", source.name); + continue; + } + + match self.sync_source(source).await { + Ok(result) => { + info!("✅ Completed sync for {}: {} messages across {} mailboxes", + result.source_name, result.total_messages, result.mailboxes_processed); + results.push(result); + } + Err(e) => { + error!("❌ Failed to sync source {}: {}", source.name, e); + // Continue with other sources even if one fails + } + } + } + Ok(results) + } +} +``` + +--- + +## Performance & Scalability + +### Concurrency Models + +| Aspect | Go Implementation | Rust Implementation | +|--------|------------------|-------------------| +| **Processing Model** | Sequential (blocking) | Asynchronous (non-blocking) | +| **Account Processing** | One at a time | One at a time with internal concurrency | +| **Mailbox Processing** | One at a time | One at a time with async I/O | +| **Message Processing** | One at a time | Batch processing with async operations | +| **Network Operations** | Blocking I/O | Non-blocking async I/O | + +### IMAP Filtering Efficiency + +**Go: Client-Side Filtering** +```go +// Downloads ALL messages first, then filters locally +messages := imap.FetchAll() +filtered := []Message{} +for _, msg := range messages { + if ShouldProcessMessage(msg, filter) { + filtered = append(filtered, msg) + } +} +``` + +**Rust: Server-Side Filtering** +```rust +// Filters on server, only downloads matching messages +pub async fn search_messages_advanced( + &mut self, + since_date: Option<&DateTime>, + subject_keywords: Option<&[String]>, + from_keywords: Option<&[String]>, +) -> Result> { + let mut search_parts = Vec::new(); + + if let Some(keywords) = subject_keywords { + for keyword in keywords { + search_parts.push(format!("SUBJECT \"{}\"", keyword)); + } + } + // Server processes the filter, returns only matching UIDs +} +``` + +**Performance Impact**: For a mailbox with 10,000 emails where you only want recent messages: +- **Go**: Downloads all 10,000 emails, then filters locally +- **Rust**: Server filters first, downloads only matching emails (potentially 10x less data transfer) + +### Error Recovery and Resilience + +**Go: Basic Error Handling** +```go +err := processImapSource(&source, couchClient, dbName, args.MaxMessages, args.DryRun) +if err != nil { + log.Printf("ERROR: Failed to process IMAP source %s: %v", source.Name, err) +} +// Continues with next source, no retry logic +``` + +**Rust: Intelligent Retry Logic** +```rust +async fn retry_operation(&self, operation_name: &str, operation: F) -> Result +where F: Fn() -> Fut, Fut: std::future::Future> +{ + const MAX_RETRIES: u32 = 3; + const RETRY_DELAY_MS: u64 = 1000; + + for attempt in 1..=MAX_RETRIES { + match operation().await { + Ok(result) => return Ok(result), + Err(e) => { + let is_retryable = match &e.downcast_ref::() { + Some(CouchError::Http(_)) => true, + Some(CouchError::CouchDb { status, .. }) => *status >= 500, + _ => false, + }; + + if is_retryable && attempt < MAX_RETRIES { + warn!("Attempt {}/{} failed for {}: {}. Retrying in {}ms...", + attempt, MAX_RETRIES, operation_name, e, RETRY_DELAY_MS); + async_std::task::sleep(Duration::from_millis(RETRY_DELAY_MS)).await; + } else { + error!("Operation {} failed after {} attempts: {}", + operation_name, attempt, e); + return Err(e); + } + } + } + } + unreachable!() +} +``` + +--- + +## Developer Experience + +### Code Complexity and Learning Curve + +| Aspect | Go Implementation | Rust Implementation | +|--------|------------------|-------------------| +| **Lines of Code** | 1,355 | 3,056 | +| **Number of Files** | 4 | 9 | +| **Dependencies** | 4 external | 14+ external | +| **Compilation Time** | 2-3 seconds | 6+ seconds | +| **Learning Curve** | Low | Medium-High | +| **Debugging Ease** | Simple stack traces | Rich error context | + +### Dependency Management + +**Go Dependencies (minimal approach):** +```go +require ( + github.com/emersion/go-imap/v2 v2.0.0-beta.5 + github.com/emersion/go-message v0.18.1 + github.com/go-kivik/kivik/v4 v4.4.0 + github.com/spf13/pflag v1.0.7 +) +``` + +**Rust Dependencies (rich ecosystem):** +```toml +[dependencies] +anyhow = "1.0" +serde = { version = "1.0", features = ["derive"] } +serde_json = "1.0" +tokio = { version = "1.0", features = ["full"] } +reqwest = { version = "0.11", features = ["json"] } +clap = { version = "4.0", features = ["derive"] } +log = "0.4" +env_logger = "0.10" +chrono = { version = "0.4", features = ["serde"] } +async-imap = "0.9" +mail-parser = "0.6" +thiserror = "1.0" +glob = "0.3" +dirs = "5.0" +``` + +**Trade-offs**: +- **Go**: Faster builds, fewer potential security vulnerabilities, simpler dependency tree +- **Rust**: Richer functionality, better error types, more battle-tested async ecosystem + +--- + +## Feature Comparison Matrix + +| Feature | Go Implementation | Rust Implementation | Notes | +|---------|------------------|-------------------|-------| +| **Core Functionality** | +| IMAP Email Sync | ✅ | ✅ | Both fully functional | +| CouchDB Storage | ✅ | ✅ | Both support attachments | +| Incremental Sync | ✅ | ✅ | Both use metadata tracking | +| **Configuration** | +| JSON Config Files | ✅ | ✅ | Same format, auto-discovery | +| Folder Filtering | ✅ | ✅ | Both support wildcards | +| Date Filtering | ✅ | ✅ | Since date support | +| Keyword Filtering | ✅ (client-side) | ✅ (server-side) | Rust is more efficient | +| **CLI Features** | +| GNU-style Arguments | ✅ | ✅ | Both use standard conventions | +| Dry-run Mode | ✅ | ✅ | Both recently implemented | +| Bash Completion | ✅ | ✅ | Auto-generated scripts | +| Help System | Basic | Rich | Rust uses clap framework | +| **Reliability** | +| Error Handling | Basic | Advanced | Rust has retry logic | +| Connection Recovery | Manual | Automatic | Rust handles reconnections | +| Resource Management | Manual (defer) | Automatic (RAII) | Rust prevents leaks | +| **Performance** | +| Concurrent Processing | ❌ | ✅ | Rust uses async/await | +| Server-side Filtering | ❌ | ✅ | Rust reduces bandwidth | +| Memory Efficiency | Good | Excellent | Rust zero-copy where possible | +| **Development** | +| Test Coverage | Minimal | Comprehensive | Rust has extensive tests | +| Documentation | Basic | Rich | Rust has detailed docs | +| Type Safety | Good | Excellent | Rust prevents more errors | + +--- + +## Use Case Recommendations + +### Choose Go Implementation When: + +#### 🎯 **Personal Use & Simplicity** +- Single email account or small number of accounts +- Infrequent synchronization (daily/weekly) +- Simple setup requirements +- You want to understand/modify the code easily + +#### 🎯 **Resource Constraints** +- Memory-limited environments +- CPU-constrained systems +- Quick deployment needed +- Minimal disk space for binaries + +#### 🎯 **Development Preferences** +- Team familiar with Go +- Preference for simple, readable code +- Fast compilation important for development cycle +- Minimal external dependencies preferred + +**Example Use Case**: Personal backup of 1-2 Gmail accounts, running weekly on a Raspberry Pi. + +### Choose Rust Implementation When: + +#### 🚀 **Performance Critical Scenarios** +- Multiple email accounts (3+ accounts) +- Large mailboxes (10,000+ emails) +- Frequent synchronization (hourly/real-time) +- High-volume email processing + +#### 🚀 **Production Environments** +- Business-critical email backups +- Need for reliable error recovery +- 24/7 operation requirements +- Professional deployment standards + +#### 🚀 **Advanced Features Required** +- Server-side IMAP filtering needed +- Complex folder filtering patterns +- Detailed logging and monitoring +- Long-term maintenance planned + +**Example Use Case**: Corporate email backup system handling 10+ accounts with complex filtering rules, running continuously in a production environment. + +--- + +## Performance Benchmarks + +### Theoretical Performance Comparison + +| Scenario | Go Implementation | Rust Implementation | Improvement | +|----------|------------------|-------------------|-------------| +| **Single small account** (1,000 emails) | 2-3 minutes | 1-2 minutes | 33-50% faster | +| **Multiple accounts** (3 accounts, 5,000 emails each) | 15-20 minutes | 8-12 minutes | 40-47% faster | +| **Large mailbox** (50,000 emails with filtering) | 45-60 minutes | 15-25 minutes | 58-67% faster | +| **Network errors** (5% packet loss) | May fail/restart | Continues with retry | Much more reliable | + +*Note: These are estimated performance improvements based on architectural differences. Actual performance will vary based on network conditions, server capabilities, and email characteristics.* + +### Resource Usage + +| Metric | Go Implementation | Rust Implementation | +|--------|------------------|-------------------| +| **Memory Usage** | 20-50 MB | 15-40 MB | +| **CPU Usage** | Low (single-threaded) | Medium (multi-threaded) | +| **Network Efficiency** | Lower (downloads then filters) | Higher (filters then downloads) | +| **Disk I/O** | Sequential writes | Batched writes | + +--- + +## Migration Guide + +### From Go to Rust + +If you're currently using the Go implementation and considering migration: + +#### **When to Migrate**: +- You experience performance issues with large mailboxes +- You need better error recovery and reliability +- You want more efficient network usage +- You're planning long-term maintenance + +#### **Migration Steps**: +1. **Test in parallel**: Run both implementations with `--dry-run` to compare results +2. **Backup existing data**: Ensure your CouchDB data is backed up +3. **Update configuration**: Configuration format is identical, no changes needed +4. **Replace binary**: Simply replace the Go binary with the Rust binary +5. **Monitor performance**: Compare sync times and resource usage + +#### **Compatibility Notes**: +- ✅ Configuration files are 100% compatible +- ✅ CouchDB database format is identical +- ✅ Command-line arguments are the same +- ✅ Dry-run mode works identically + +### Staying with Go + +The Go implementation remains fully supported and is appropriate when: +- Current performance meets your needs +- Simplicity is more important than features +- Team lacks Rust expertise +- Resource usage is already optimized for your environment + +--- + +## Technical Architecture Details + +### Go Implementation Structure + +``` +go/ +├── main.go # Entry point and orchestration +├── config/ +│ └── config.go # Configuration loading and CLI parsing +├── couch/ +│ └── couch.go # CouchDB client and operations +└── mail/ + └── imap.go # IMAP client and message processing +``` + +**Key Characteristics**: +- Monolithic processing flow +- Synchronous I/O operations +- Basic error handling +- Minimal abstraction layers + +### Rust Implementation Structure + +``` +rust/src/ +├── main.rs # Entry point +├── lib.rs # Library exports +├── cli.rs # Command-line interface +├── config.rs # Configuration management +├── sync.rs # Synchronization coordinator +├── imap.rs # IMAP client with retry logic +├── couch.rs # CouchDB client with error handling +├── filters.rs # Filtering utilities +└── schemas.rs # Data structure definitions +``` + +**Key Characteristics**: +- Modular architecture with clear separation +- Asynchronous I/O with tokio runtime +- Comprehensive error handling +- Rich abstraction layers + +--- + +## Security Considerations + +Both implementations currently share the same security limitations and features: + +### Current Security Features +- ✅ TLS/SSL support for IMAP and CouchDB connections +- ✅ Configuration file validation +- ✅ Safe handling of email content + +### Shared Security Limitations +- ⚠️ Plaintext passwords in configuration files +- ⚠️ No OAuth2 support for modern email providers +- ⚠️ No credential encryption at rest + +### Future Security Improvements (Recommended for Both) +1. **Environment Variable Credentials**: Support reading passwords from environment variables +2. **OAuth2 Integration**: Support modern authentication for Gmail, Outlook, etc. +3. **Credential Encryption**: Encrypt stored credentials with system keyring integration +4. **Audit Logging**: Enhanced logging of authentication and access events + +--- + +## Deployment Considerations + +### Go Implementation Deployment + +**Binary Name**: `mail2couch-go` + +**Advantages**: +- Single binary deployment +- Minimal system dependencies +- Lower memory footprint +- Faster startup time + +**Best Practices**: +```bash +# Build for production using justfile +just build-go-release + +# Or build directly +cd go && go build -ldflags="-s -w" -o mail2couch-go . + +# Deploy with systemd service +sudo cp go/mail2couch-go /usr/local/bin/ +sudo systemctl enable mail2couch-go.service +``` + +### Rust Implementation Deployment + +**Binary Name**: `mail2couch-rs` + +**Advantages**: +- Better resource utilization under load +- Superior error recovery +- More detailed logging and monitoring +- Enhanced CLI experience + +**Best Practices**: +```bash +# Build optimized release using justfile +just build-rust-release + +# Or build directly +cd rust && cargo build --release + +# Deploy with enhanced monitoring +sudo cp rust/target/release/mail2couch-rs /usr/local/bin/ +sudo systemctl enable mail2couch-rs.service + +# Configure structured logging +export RUST_LOG=info +export MAIL2COUCH_LOG_FORMAT=json +``` + +### Universal Installation + +```bash +# Build and install both implementations (user-local) +just install +# This installs to ~/bin/mail2couch-go and ~/bin/mail2couch-rs + +# Build and install both implementations (system-wide) +sudo just system-install +# This installs to /usr/local/bin/mail2couch-go and /usr/local/bin/mail2couch-rs +``` + +--- + +## Future Development Roadmap + +### Short-term Improvements (Both Implementations) + +1. **Security Enhancements** + - Environment variable credential support + - OAuth2 authentication for major providers + - Encrypted credential storage + +2. **Usability Improvements** + - Interactive configuration wizard + - Progress indicators for long-running operations + - Enhanced error messages with solutions + +### Long-term Strategic Direction + +#### Go Implementation (Maintenance Mode) +- Bug fixes and security updates +- Maintain compatibility with Rust version +- Focus on simplicity and stability +- Target: Personal and small-scale deployments + +#### Rust Implementation (Active Development) +- Performance optimizations +- Advanced features (web interface, monitoring APIs) +- Enterprise features (clustering, high availability) +- Target: Production and large-scale deployments + +### Recommended Development Focus + +1. **Primary Development**: Focus on Rust implementation for new features +2. **Compatibility Maintenance**: Ensure Go version remains compatible +3. **Migration Path**: Provide clear migration guidance and tooling +4. **Documentation**: Maintain comprehensive documentation for both + +--- + +## Conclusion + +Both implementations represent excellent software engineering practices and serve different market segments effectively: + +- **Go Implementation**: Ideal for users who prioritize simplicity, fast deployment, and ease of understanding. Perfect for personal use and small-scale deployments. + +- **Rust Implementation**: Superior choice for users who need performance, reliability, and advanced features. Recommended for production environments and large-scale email processing. + +### Final Recommendation + +**For new deployments**: Start with the Rust implementation unless simplicity is your primary concern. The performance benefits and reliability features provide significant value. + +**For existing Go users**: Consider migration if you experience performance limitations or need better error recovery. The migration path is straightforward due to configuration compatibility. + +**For development contributions**: Focus on the Rust implementation for new features, while maintaining the Go version for bug fixes and compatibility. + +The project demonstrates that having two implementations can serve different user needs effectively, with each leveraging the strengths of its respective programming language and ecosystem. \ No newline at end of file diff --git a/README.md b/README.md index 8267464..f6eb104 100644 --- a/README.md +++ b/README.md @@ -43,26 +43,6 @@ A powerful email backup utility that synchronizes mail from IMAP accounts to Cou - **Comprehensive Logging**: Detailed output for monitoring and troubleshooting - **Error Resilience**: Graceful handling of network issues and server problems -## Project Status - -**Production Ready** (August 2025): Both Go and Rust implementations are fully functional with identical feature sets and database output. - -- ✅ **Complete Feature Parity**: Server-side filtering, binary attachments, incremental sync -- ✅ **Comprehensive Testing**: Verified identical CouchDB output between implementations -- ✅ **SystemD Integration**: Automated scheduling with timer units -- ✅ **Production Quality**: Robust error handling, retry logic, dry-run mode - -## 📚 Documentation - -Comprehensive documentation is available in the [`docs/`](docs/) directory: - -- **[docs/README.md](docs/README.md)** - Documentation overview and quick start -- **[docs/ANALYSIS.md](docs/ANALYSIS.md)** - Technical analysis and current status -- **[docs/IMPLEMENTATION_COMPARISON.md](docs/IMPLEMENTATION_COMPARISON.md)** - Go vs Rust comparison -- **[docs/FOLDER_PATTERNS.md](docs/FOLDER_PATTERNS.md)** - Folder filtering guide -- **[docs/couchdb-schemas.md](docs/couchdb-schemas.md)** - Database schema documentation -- **[docs/TODO.md](docs/TODO.md)** - Development roadmap and future plans - ## Quick Start ### Installation diff --git a/TODO.md b/TODO.md new file mode 100644 index 0000000..e004c00 --- /dev/null +++ b/TODO.md @@ -0,0 +1,47 @@ +# mail2couch TODO and Feature Requests + +## Planned Features + +### Keyword Filtering for Messages + +Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration. + +**Proposed Configuration Extension:** + +```json +{ + "messageFilter": { + "since": "2024-01-01", + "subjectKeywords": ["urgent", "important", "meeting"], + "senderKeywords": ["@company.com", "notifications"], + "recipientKeywords": ["team@company.com", "all@"] + } +} +``` + +**Implementation Details:** + +- `subjectKeywords`: Array of keywords to match in email subject lines +- `senderKeywords`: Array of keywords to match in sender email addresses or names +- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names +- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns +- Case-insensitive matching by default +- Support for simple wildcards or regex patterns + +**Use Cases:** + +1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords +2. **Project-based Archiving**: Filter emails related to specific projects or clients +3. **Notification Management**: Exclude or include automated notifications based on sender patterns +4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders + +**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails. + +## Other Planned Improvements + +1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content +2. **Message Body Extraction**: Support for HTML/plain text and multipart messages +3. **Attachment Handling**: Optional support for email attachments +4. **Batch Operations**: Improve CouchDB insertion performance +5. **Error Recovery**: Retry logic and partial sync recovery +6. **Testing**: Comprehensive unit test coverage \ No newline at end of file diff --git a/docs/couchdb-schemas.md b/couchdb-schemas.md similarity index 100% rename from docs/couchdb-schemas.md rename to couchdb-schemas.md diff --git a/docs/ANALYSIS.md b/docs/ANALYSIS.md deleted file mode 100644 index e87c67a..0000000 --- a/docs/ANALYSIS.md +++ /dev/null @@ -1,134 +0,0 @@ -# Comprehensive Analysis of `mail2couch` Implementations - -*Last Updated: August 2025* - -This document provides a comprehensive analysis of the `mail2couch` project after reaching production readiness. Both Go and Rust implementations are now fully functional, tested, and production-ready with equivalent feature sets. - ---- - -## 1. Current State (August 2025) - -The project now consists of **two production-ready implementations** of the same core tool, both achieving feature parity and production quality. - -### **The Go Implementation** -- ✅ **Production Ready**: Fully functional with comprehensive IMAP and CouchDB integration -- ✅ **Server-side Filtering**: Implements IMAP SEARCH with keyword filtering and graceful fallbacks -- ✅ **Complete Feature Set**: All core functionality implemented and tested -- ✅ **Robust Error Handling**: Proper connection management and retry logic -- ✅ **Dry-run Mode**: Comprehensive testing capabilities without data changes - -### **The Rust Implementation** -- ✅ **Production Ready**: Fully functional with advanced async architecture -- ✅ **Performance Optimized**: Asynchronous operations with superior concurrency -- ✅ **Feature Complete**: All functionality implemented with enhanced user experience -- ✅ **Enterprise Grade**: Comprehensive error handling, retry logic, and monitoring -- ✅ **Advanced CLI**: Rich logging, progress reporting, and configuration validation - ---- - -## 2. Status of Previous Issues - -All major issues identified in earlier analysis have been **resolved**: - -### ✅ **Resolved Issues** -* **`Incomplete Rust Implementation`**: **FULLY RESOLVED** - Rust implementation is production-ready -* **`Inefficient Keyword Filtering`**: **FULLY RESOLVED** - Both implementations use server-side IMAP SEARCH -* **`Performance for Large-Scale Use`**: **SIGNIFICANTLY IMPROVED** - Async Rust, optimized Go -* **`Missing Dry-Run Mode`**: **FULLY RESOLVED** - Comprehensive dry-run support in both -* **`Inconsistent Database Output`**: **FULLY RESOLVED** - Identical document schemas and behavior -* **`Limited Error Handling`**: **FULLY RESOLVED** - Robust error handling and retry logic -* **`Binary Attachment Issues`**: **FULLY RESOLVED** - Full binary attachment support verified - -### ⚠️ **Outstanding Issues** -* **`Security Model`**: Still requires plaintext passwords in configuration (environment variable support planned) -* **`Web Interface`**: Not implemented (not currently prioritized for core functionality) -* **`Interactive Setup`**: Could improve first-time user experience (low priority) - -## 3. Current Comparative Analysis: Go vs. Rust - -Both implementations now provide equivalent functionality with different architectural approaches: - -### **Go Implementation** - -#### **Strengths**: -- ✅ **Simplicity**: Sequential, straightforward code that's easy to understand and debug -- ✅ **Fast Development**: Quick compilation and simple deployment model -- ✅ **Server-side Filtering**: Full IMAP SEARCH implementation with graceful fallbacks -- ✅ **Production Stability**: Reliable operation with proper error handling -- ✅ **Comprehensive Testing**: Verified equivalent output with Rust implementation - -#### **Trade-offs**: -- ⚖️ **Sequential Processing**: Processes one mailbox at a time (adequate for most use cases) -- ⚖️ **Standard Error Handling**: Basic retry logic sufficient for typical deployments - -### **Rust Implementation** - -#### **Strengths**: -- ✅ **High Performance**: Async architecture enables concurrent operations -- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging -- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation -- ✅ **Memory Safety**: Rust's compile-time guarantees prevent entire classes of bugs -- ✅ **Advanced Architecture**: Modular design facilitates maintenance and feature additions - -#### **Trade-offs**: -- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge for maintenance -- ⚖️ **Build Time**: Longer compilation times during development - -## 4. Production Readiness Assessment - -Both implementations have achieved **production readiness** with comprehensive testing and validation: - -### **Shared Capabilities** -- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers -- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata -- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks -- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas -- ✅ **Error Resilience**: Connection retry logic, graceful degradation -- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration - -### **Verification Status** -- ✅ **Identical Output**: Both implementations produce identical CouchDB documents -- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable -- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results -- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified -- ✅ **Scale Testing**: Tested with thousands of messages and large attachments - -### **Deployment Options** -- ✅ **SystemD Services**: Timer units for automated scheduling (30min, hourly, daily) -- ✅ **Manual Execution**: Command-line tools with comprehensive help and validation -- ✅ **Configuration Management**: Automatic config file discovery, validation -- ✅ **Monitoring Integration**: Structured logging suitable for monitoring systems - -## 5. Future Enhancement Roadmap - -Based on current production status, these enhancements would further improve the project: - -### **High Priority** -- 🔐 **Enhanced Security**: Environment variable credential support to eliminate plaintext passwords -- 🚀 **Go Concurrency**: Optional goroutine-based parallel processing for multiple mailboxes -- 📊 **Progress Indicators**: Real-time progress reporting for long-running operations - -### **Medium Priority** -- 🖥️ **Interactive Setup**: Guided configuration wizard for first-time users -- 📈 **Performance Metrics**: Built-in timing and throughput reporting -- 🔄 **Advanced Sync**: Bidirectional sync capabilities and conflict resolution - -### **Low Priority** -- 🌐 **Web Interface**: Optional web UI for configuration and monitoring -- 📱 **REST API**: HTTP API for integration with other systems -- 🔌 **Plugin System**: Extensible architecture for custom filters and processors - -## 6. Recommendations - -### **For Production Deployment** -Both implementations are ready for production use. **Choose based on your requirements:** - -- **Choose Go** if you prefer simplicity, fast builds, and straightforward maintenance -- **Choose Rust** if you need maximum performance, advanced features, or plan extensive customization - -### **For Development Contributions** -- **Go implementation**: Ideal for quick fixes, simple feature additions, or learning the codebase -- **Rust implementation**: Better for performance improvements, complex features, or async operations - -### **Current Status Summary** -The mail2couch project has successfully achieved its primary goal: providing reliable, production-ready email backup solutions. Both implementations offer equivalent functionality with different architectural strengths, making the project suitable for a wide range of deployment scenarios and maintenance preferences. \ No newline at end of file diff --git a/docs/IMPLEMENTATION_COMPARISON.md b/docs/IMPLEMENTATION_COMPARISON.md deleted file mode 100644 index 9dbc6f0..0000000 --- a/docs/IMPLEMENTATION_COMPARISON.md +++ /dev/null @@ -1,154 +0,0 @@ -# Go vs Rust Implementation Comparison - -*Last Updated: August 2025* - -This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch after both have reached production readiness with equivalent functionality. - -## Executive Summary - -The mail2couch project offers **two production-ready implementations** with identical core functionality but different architectural approaches: - -- **Go Implementation**: Sequential, straightforward approach emphasizing simplicity and maintainability -- **Rust Implementation**: Asynchronous, feature-rich architecture prioritizing performance and enterprise features - -**Key Finding**: Both implementations now provide **equivalent functionality** and **identical database output**. The choice between them depends on operational requirements, team expertise, and performance needs rather than feature completeness. - -## Feature Comparison Matrix - -| Feature Category | Go Implementation | Rust Implementation | Status | -|-----------------|------------------|-------------------|---------| -| **Core Functionality** | -| IMAP/IMAPS Support | ✅ Full support | ✅ Full support | **Equivalent** | -| CouchDB Integration | ✅ Native attachments | ✅ Native attachments | **Equivalent** | -| Binary Attachments | ✅ Verified working | ✅ Verified working | **Equivalent** | -| Sync vs Archive Modes | ✅ Both modes | ✅ Both modes | **Equivalent** | -| Incremental Sync | ✅ Metadata tracking | ✅ Metadata tracking | **Equivalent** | -| **Filtering & Search** | -| Folder Filtering | ✅ IMAP LIST patterns | ✅ IMAP LIST patterns | **Equivalent** | -| Server-side Search | ✅ IMAP SEARCH keywords | ✅ IMAP SEARCH keywords | **Equivalent** | -| Keyword Filtering | ✅ Subject/sender/recipient | ✅ Subject/sender/recipient | **Equivalent** | -| Date Filtering | ✅ Since date support | ✅ Since date support | **Equivalent** | -| **Operational Features** | -| Dry-run Mode | ✅ Comprehensive | ✅ Comprehensive | **Equivalent** | -| Configuration Discovery | ✅ Multi-path search | ✅ Multi-path search | **Equivalent** | -| Command Line Interface | ✅ GNU-style flags | ✅ Modern clap-based | **Rust Advantage** | -| Progress Reporting | ✅ Basic logging | ✅ Rich structured logs | **Rust Advantage** | -| Error Handling | ✅ Retry logic | ✅ Advanced retry + async | **Rust Advantage** | -| **Performance & Architecture** | -| Concurrency Model | ⚖️ Sequential | ✅ Async/concurrent | **Rust Advantage** | -| Memory Safety | ✅ Go GC | ✅ Compile-time guarantees | **Rust Advantage** | -| Build Time | ✅ Fast (~5s) | ⚖️ Slower (~30s) | **Go Advantage** | -| Binary Size | ✅ Smaller | ⚖️ Larger | **Go Advantage** | -| Resource Usage | ✅ Low memory | ✅ Efficient async | **Equivalent** | -| **Development & Maintenance** | -| Code Complexity | ✅ Simple, readable | ⚖️ Advanced patterns | **Go Advantage** | -| Learning Curve | ✅ Easy for Go devs | ⚖️ Requires Rust knowledge | **Go Advantage** | -| Debugging | ✅ Straightforward | ⚖️ Advanced tooling needed | **Go Advantage** | -| Testing | ✅ Standard Go tests | ✅ Comprehensive test suite | **Equivalent** | -| Linting/Formatting | ✅ go fmt/vet | ✅ rustfmt/clippy | **Equivalent** | - -## Production Readiness Assessment - -Both implementations have achieved **production readiness** with comprehensive testing and validation: - -### **Shared Capabilities** -- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers -- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata -- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks -- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas -- ✅ **Error Resilience**: Connection retry logic, graceful degradation -- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration - -### **Verification Status** -- ✅ **Identical Output**: Both implementations produce identical CouchDB documents -- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable -- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results -- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified -- ✅ **Scale Testing**: Tested with thousands of messages and large attachments - -## Architectural Comparison - -### **Go Implementation: Production Simplicity** - -**Strengths:** -- ✅ **Straightforward Code**: Sequential, easy to understand and debug -- ✅ **Fast Development**: Quick compilation and simple deployment model -- ✅ **Production Stable**: Reliable operation with proper error handling -- ✅ **Low Resource**: Minimal memory usage and fast startup - -**Trade-offs:** -- ⚖️ **Sequential Processing**: One mailbox at a time (adequate for most use cases) -- ⚖️ **Basic Features**: Standard CLI and logging capabilities - -### **Rust Implementation: Enterprise Architecture** - -**Strengths:** -- ✅ **High Performance**: Async architecture enables concurrent operations -- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging -- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation -- ✅ **Memory Safety**: Compile-time guarantees prevent entire classes of bugs -- ✅ **Modular Design**: Well-structured architecture facilitates maintenance - -**Trade-offs:** -- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge -- ⚖️ **Build Time**: Longer compilation times during development - -## Use Case Recommendations - -### Choose **Go Implementation** When: - -- 🎯 **Simplicity Priority**: Easy to understand, modify, and maintain -- 🎯 **Resource Constraints**: Memory-limited environments, quick deployment -- 🎯 **Small Scale**: Personal use, few accounts, infrequent synchronization -- 🎯 **Team Familiarity**: Go expertise available, fast development cycle important - -**Example**: Personal backup of 1-2 email accounts, running daily on modest hardware. - -### Choose **Rust Implementation** When: - -- 🚀 **Performance Critical**: Multiple accounts, large mailboxes, frequent sync -- 🚀 **Production Environment**: Business-critical backups, 24/7 operation -- 🚀 **Advanced Features**: Rich logging, detailed progress reporting, complex filtering -- 🚀 **Long-term Maintenance**: Enterprise deployment with ongoing development - -**Example**: Corporate email backup handling 10+ accounts with complex filtering, running continuously. - -## Migration Compatibility - -### **100% Compatible** -- ✅ Configuration files are identical between implementations -- ✅ CouchDB database format and documents are identical -- ✅ Command-line arguments and behavior are the same -- ✅ Dry-run mode works identically -- ✅ SystemD service files available for both - -### **Migration Process** -1. Test new implementation with `--dry-run` to verify identical results -2. Stop current implementation -3. Replace binary (same config file works) -4. Start new implementation -5. Verify operation and performance - -## Development Status - -### **Current State (August 2025)** -- ✅ **Both Production Ready**: Full feature parity achieved -- ✅ **Comprehensive Testing**: Identical output verified -- ✅ **Complete Documentation**: Usage guides and examples -- ✅ **SystemD Integration**: Automated scheduling support -- ✅ **Build System**: Unified justfile for both implementations - -### **Future Enhancement Priorities** -1. **Security**: Environment variable credential support -2. **Go Concurrency**: Optional parallel processing -3. **Progress Indicators**: Real-time progress reporting -4. **Interactive Setup**: Guided configuration wizard - -## Conclusion - -Both implementations represent production-quality solutions with different strengths: - -- **Go Implementation**: Ideal for users prioritizing simplicity, maintainability, and straightforward operation -- **Rust Implementation**: Superior for users needing performance, advanced features, and enterprise-grade reliability - -**Recommendation**: Choose based on your operational requirements and team expertise. Both provide identical functionality and data output, making migration straightforward when needs change. \ No newline at end of file diff --git a/docs/README.md b/docs/README.md deleted file mode 100644 index ba9647b..0000000 --- a/docs/README.md +++ /dev/null @@ -1,94 +0,0 @@ -# mail2couch Documentation - -This directory contains comprehensive documentation for the mail2couch project, which provides two production-ready implementations for backing up mail from IMAP servers to CouchDB. - -## 📚 Documentation Index - -### Core Documentation -- **[ANALYSIS.md](ANALYSIS.md)** - Detailed technical analysis of both implementations -- **[IMPLEMENTATION_COMPARISON.md](IMPLEMENTATION_COMPARISON.md)** - Side-by-side comparison of Go vs Rust implementations -- **[couchdb-schemas.md](couchdb-schemas.md)** - CouchDB document schemas and database structure -- **[TODO.md](TODO.md)** - Development roadmap and outstanding tasks - -### Configuration & Setup -- **[FOLDER_PATTERNS.md](FOLDER_PATTERNS.md)** - Guide to folder filtering patterns and wildcards -- **[test-config-comparison.md](test-config-comparison.md)** - Configuration examples and testing scenarios - -### Examples -- **[examples/](examples/)** - Sample CouchDB documents and configuration files - - `sample-mail-document.json` - Complete email document with attachments - - `sample-sync-metadata.json` - Sync state tracking document - - `simple-mail-document.json` - Basic email document structure - -## 🚀 Quick Start - -Both implementations are production-ready with identical feature sets: - -### Go Implementation -```bash -cd go && go build -o mail2couch-go . -./mail2couch-go --config ../config.json --dry-run -``` - -### Rust Implementation -```bash -cd rust && cargo build --release -./target/release/mail2couch-rs --config ../config.json --dry-run -``` - -## ✅ Current Status (August 2025) - -Both implementations are **production-ready** with: - -- ✅ **Full IMAP support** with TLS/SSL connections -- ✅ **Server-side folder filtering** using IMAP LIST patterns -- ✅ **Server-side message filtering** using IMAP SEARCH with keyword support -- ✅ **Binary attachment handling** with CouchDB native attachments -- ✅ **Incremental synchronization** with metadata tracking -- ✅ **Sync vs Archive modes** for different backup strategies -- ✅ **Dry-run mode** for safe testing -- ✅ **Comprehensive error handling** with graceful fallbacks -- ✅ **SystemD integration** with timer units for automated scheduling -- ✅ **Build system integration** with justfile for unified project management - -## 🔧 Key Features - -### Filtering & Search -- **Folder Filtering**: Wildcard patterns (`*`, `?`, `[abc]`) with include/exclude lists -- **Message Filtering**: Subject, sender, and recipient keyword filtering -- **IMAP SEARCH**: Server-side filtering when supported, client-side fallback -- **Date Filtering**: Incremental sync based on last sync time or configured since date - -### Data Storage -- **CouchDB Integration**: Native attachment storage, per-account databases -- **Document Structure**: Standardized schema with full email metadata -- **Sync Metadata**: State tracking for efficient incremental updates -- **Duplicate Prevention**: UID-based deduplication across syncs - -### Operations -- **Mode Selection**: Archive (append-only) or Sync (mirror) modes -- **Connection Handling**: Automatic retry logic, graceful error recovery -- **Progress Reporting**: Detailed logging with message counts and timing -- **Resource Management**: Configurable message limits, connection pooling - -## 📊 Performance & Compatibility - -Both implementations have been tested with: -- **IMAP Servers**: Gmail, Office365, Dovecot, GreenMail -- **CouchDB Versions**: 3.x with native attachment support -- **Message Volumes**: Tested with thousands of messages and large attachments -- **Network Conditions**: Automatic retry and reconnection handling - -## 🛠️ Development - -See individual implementation directories for development setup: -- **Go**: `/go/` - Standard Go toolchain with modules -- **Rust**: `/rust/` - Cargo-based build system with comprehensive testing - -For unified development commands, use the project justfile: -```bash -just build # Build both implementations -just test # Run all tests -just check # Run linting and formatting -just install # Install systemd services -``` \ No newline at end of file diff --git a/docs/TODO.md b/docs/TODO.md deleted file mode 100644 index 1df605f..0000000 --- a/docs/TODO.md +++ /dev/null @@ -1,145 +0,0 @@ -# mail2couch Development Roadmap - -*Last Updated: August 2025* - -This document outlines the development roadmap for mail2couch, with both Go and Rust implementations now in production-ready status. - -## ✅ Completed Major Milestones - -### Production Readiness (August 2025) -- ✅ **Full Feature Parity**: Both implementations provide identical functionality -- ✅ **Server-side IMAP SEARCH**: Keyword filtering implemented in both Go and Rust -- ✅ **Binary Attachment Support**: Verified working with CouchDB native attachments -- ✅ **Incremental Sync**: Cross-implementation compatibility verified -- ✅ **Dry-run Mode**: Comprehensive testing capabilities in both implementations -- ✅ **Error Handling**: Robust retry logic and graceful fallbacks -- ✅ **SystemD Integration**: Timer units for automated scheduling -- ✅ **Build System**: Unified justfile for both implementations -- ✅ **Documentation**: Comprehensive guides and comparisons -- ✅ **Code Quality**: All linting and formatting standards met - -### Architecture & Testing -- ✅ **Database Output Equivalence**: Both implementations produce identical CouchDB documents -- ✅ **Filtering Accuracy**: Server-side IMAP LIST and SEARCH with client-side fallbacks -- ✅ **Connection Handling**: TLS support, automatic retry, graceful error recovery -- ✅ **Configuration Management**: Automatic file discovery, validation, GNU-style CLI - -### Originally Planned Features (Now Complete) -- ✅ **Keyword Filtering for Messages**: Subject, sender, and recipient keyword filtering implemented -- ✅ **Real IMAP Message Parsing**: Full message content extraction with go-message and mail-parser -- ✅ **Message Body Extraction**: HTML/plain text and multipart message support -- ✅ **Attachment Handling**: Complete binary attachment support with CouchDB native storage -- ✅ **Error Recovery**: Comprehensive retry logic and partial sync recovery -- ✅ **Performance**: Batch operations and efficient CouchDB insertion - -## 🚧 Current Development Priorities - -### High Priority -1. **🔐 Enhanced Security Model** - - Environment variable credential support (`MAIL2COUCH_IMAP_PASSWORD`, etc.) - - Eliminate plaintext passwords from configuration files - - System keyring integration for credential storage - -### Medium Priority -2. **🚀 Go Implementation Concurrency** - - Optional goroutine-based parallel mailbox processing - - Maintain simplicity while improving performance for multiple accounts - - Configurable concurrency levels - -3. **📊 Progress Indicators** - - Real-time progress reporting for long-running operations - - ETA calculations for large mailbox synchronization - - Progress bars for terminal output - -4. **🖥️ Interactive Setup** - - Guided configuration wizard (`mail2couch setup`) - - Interactive validation of IMAP and CouchDB connectivity - - Generate sample configurations for common providers - -### Low Priority -5. **📈 Performance Metrics** - - Built-in timing and throughput reporting - - Memory usage monitoring - - Network efficiency statistics - -6. **🔄 Advanced Sync Features** - - Bidirectional sync capabilities - - Conflict resolution strategies - - Message modification detection - -7. **🌐 Web Interface** - - Optional web UI for configuration and monitoring - - CouchDB view integration for email browsing - - Search interface for archived emails - -8. **📱 API Integration** - - REST API for external system integration - - Webhook support for sync completion notifications - - Monitoring system integration - -## 📋 Technical Debt & Improvements - -### Code Quality -- **Unit Test Coverage**: Expand test coverage for both implementations -- **Integration Testing**: Automated testing with various IMAP servers -- **Performance Benchmarking**: Standardized performance comparison tools - -### User Experience -- **Error Messages**: More descriptive error messages with suggested solutions -- **Configuration Validation**: Enhanced validation with helpful error descriptions -- **Logging**: Structured logging with different verbosity levels - -### Security -- **OAuth2 Support**: Modern authentication for Gmail, Outlook, etc. -- **Credential Encryption**: Encrypt stored credentials at rest -- **Audit Logging**: Enhanced logging of authentication and access events - -## 🎯 Release Planning - -### Next Minor Release (v1.1) -- Environment variable credential support -- Interactive setup command -- Enhanced error messages - -### Next Major Release (v2.0) -- OAuth2 authentication support -- Web interface (optional) -- Go implementation concurrency improvements - -## 📊 Implementation Status - -| Feature Category | Go Implementation | Rust Implementation | Priority | -|-----------------|------------------|-------------------|----------| -| **Core Features** | ✅ Complete | ✅ Complete | - | -| **Security Model** | ⚠️ Basic | ⚠️ Basic | High | -| **Concurrency** | ⚠️ Sequential | ✅ Async | Medium | -| **Progress Reporting** | ⚠️ Basic | ⚠️ Basic | Medium | -| **Interactive Setup** | ❌ Missing | ❌ Missing | Medium | -| **Web Interface** | ❌ Missing | ❌ Missing | Low | - -## 🤝 Contributing - -### Areas Needing Contribution -1. **Security Features**: OAuth2 implementation, credential encryption -2. **User Experience**: Interactive setup, progress indicators -3. **Testing**: Unit tests, integration tests, performance benchmarks -4. **Documentation**: Usage examples, troubleshooting guides - -### Development Guidelines -- Maintain feature parity between Go and Rust implementations -- Follow established code quality standards (linting, formatting) -- Include comprehensive testing for new features -- Update documentation with new functionality - -## 📝 Notes - -### Design Decisions -- **Two Implementations**: Maintain both Go (simplicity) and Rust (performance) versions -- **Configuration Compatibility**: Ensure identical configuration formats -- **Database Compatibility**: Both implementations must produce identical CouchDB output - -### Long-term Vision -- Position Go implementation for personal/small-scale use -- Position Rust implementation for enterprise/large-scale use -- Maintain migration path between implementations -- Focus on reliability and data integrity above all else \ No newline at end of file diff --git a/docs/examples/sample-mail-document.json b/examples/sample-mail-document.json similarity index 100% rename from docs/examples/sample-mail-document.json rename to examples/sample-mail-document.json diff --git a/docs/examples/sample-sync-metadata.json b/examples/sample-sync-metadata.json similarity index 100% rename from docs/examples/sample-sync-metadata.json rename to examples/sample-sync-metadata.json diff --git a/docs/examples/simple-mail-document.json b/examples/simple-mail-document.json similarity index 100% rename from docs/examples/simple-mail-document.json rename to examples/simple-mail-document.json diff --git a/go/config/config.go b/go/config/config.go index b7bf6ca..713e73d 100644 --- a/go/config/config.go +++ b/go/config/config.go @@ -45,12 +45,6 @@ type MessageFilter struct { RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses } -// HasKeywordFilters checks if this filter has any keyword-based filters that can use IMAP SEARCH -func (mf *MessageFilter) HasKeywordFilters() bool { - return len(mf.SubjectKeywords) > 0 || len(mf.SenderKeywords) > 0 - // Note: RecipientKeywords not included as IMAP SEARCH doesn't have a reliable TO field search -} - func LoadConfig(path string) (*Config, error) { configFile, err := os.Open(path) if err != nil { diff --git a/go/mail/imap.go b/go/mail/imap.go index 5894a12..712a0eb 100644 --- a/go/mail/imap.go +++ b/go/mail/imap.go @@ -85,12 +85,12 @@ func (c *ImapClient) ListMailboxes() ([]string, error) { // ListFilteredMailboxes lists mailboxes matching the given folder filters using IMAP LIST func (c *ImapClient) ListFilteredMailboxes(filter *config.FolderFilter) ([]string, error) { var allMailboxes []string - + // If no include patterns, get all mailboxes if len(filter.Include) == 0 { return c.ListMailboxes() } - + // Use IMAP LIST with each include pattern to let the server filter seen := make(map[string]bool) for _, pattern := range filter.Include { @@ -100,7 +100,7 @@ func (c *ImapClient) ListFilteredMailboxes(filter *config.FolderFilter) ([]strin log.Printf("Failed to list mailboxes with pattern '%s': %v", pattern, err) continue } - + for _, info := range infos { if !seen[info.Mailbox] { allMailboxes = append(allMailboxes, info.Mailbox) @@ -108,12 +108,12 @@ func (c *ImapClient) ListFilteredMailboxes(filter *config.FolderFilter) ([]strin } } } - + // Apply exclude filters client-side (IMAP LIST doesn't support exclusion) if len(filter.Exclude) == 0 { return allMailboxes, nil } - + var filteredMailboxes []string for _, mailbox := range allMailboxes { excluded := false @@ -127,7 +127,7 @@ func (c *ImapClient) ListFilteredMailboxes(filter *config.FolderFilter) ([]strin filteredMailboxes = append(filteredMailboxes, mailbox) } } - + return filteredMailboxes, nil } @@ -137,25 +137,25 @@ func (c *ImapClient) matchesImapPattern(pattern, name string) bool { if pattern == name { return true } - + // Handle simple prefix wildcard: "Work*" should match "Work/Projects" if strings.HasSuffix(pattern, "*") && !strings.Contains(pattern[:len(pattern)-1], "*") { prefix := strings.TrimSuffix(pattern, "*") return strings.HasPrefix(name, prefix) } - - // Handle simple suffix wildcard: "*Temp" should match "Work/Temp" + + // Handle simple suffix wildcard: "*Temp" should match "Work/Temp" if strings.HasPrefix(pattern, "*") && !strings.Contains(pattern[1:], "*") { suffix := strings.TrimPrefix(pattern, "*") return strings.HasSuffix(name, suffix) } - + // Handle contains wildcard: "*Temp*" should match "Work/Temp/Archive" if strings.HasPrefix(pattern, "*") && strings.HasSuffix(pattern, "*") { middle := strings.Trim(pattern, "*") return strings.Contains(name, middle) } - + // For other patterns, fall back to basic string comparison return false } @@ -201,63 +201,15 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i } uidCmd.Close() - // Determine which messages to fetch based on filtering criteria + // Determine which messages to fetch based on since date var seqSet imap.SeqSet - // Use advanced search with keyword filtering when available - if messageFilter != nil && messageFilter.HasKeywordFilters() { - log.Printf("Using IMAP SEARCH with keyword filters") - uids, err := c.searchMessagesAdvanced(since, messageFilter) - if err != nil { - log.Printf("Advanced IMAP SEARCH failed, falling back to simple search: %v", err) - // Fall back to simple date-based search or fetch all - if since != nil { - searchCriteria := &imap.SearchCriteria{Since: *since} - searchCmd := c.Search(searchCriteria, nil) - searchResults, err := searchCmd.Wait() - if err != nil { - log.Printf("Simple IMAP SEARCH also failed, fetching recent messages: %v", err) - numToFetch := mbox.NumMessages - if maxMessages > 0 && int(numToFetch) > maxMessages { - numToFetch = uint32(maxMessages) - } - seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) - } else { - searchSeqNums := searchResults.AllSeqNums() - if len(searchSeqNums) == 0 { - return []*Message{}, currentUIDs, nil - } - for _, seqNum := range searchSeqNums { - seqSet.AddNum(seqNum) - } - } - } else { - numToFetch := mbox.NumMessages - if maxMessages > 0 && int(numToFetch) > maxMessages { - numToFetch = uint32(maxMessages) - } - if numToFetch > 0 { - seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages) - } - } - } else { - // Use results from advanced search - if len(uids) == 0 { - return []*Message{}, currentUIDs, nil - } - - // Limit results if maxMessages is specified - if maxMessages > 0 && len(uids) > maxMessages { - uids = uids[len(uids)-maxMessages:] - } - - for _, uid := range uids { - seqSet.AddNum(uid) - } + if since != nil { + // Use IMAP SEARCH to find messages since the specified date + searchCriteria := &imap.SearchCriteria{ + Since: *since, } - } else if since != nil { - // Use simple IMAP SEARCH for date filtering only - searchCriteria := &imap.SearchCriteria{Since: *since} + searchCmd := c.Search(searchCriteria, nil) searchResults, err := searchCmd.Wait() if err != nil { @@ -285,7 +237,7 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i } } } else { - // No filtering - fetch recent messages up to maxMessages + // No since date specified, fetch recent messages up to maxMessages numToFetch := mbox.NumMessages if maxMessages > 0 && int(numToFetch) > maxMessages { numToFetch = uint32(maxMessages) @@ -322,8 +274,8 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i continue } - // Apply message-level keyword filtering (only for keywords not handled by IMAP SEARCH) - if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter, messageFilter.HasKeywordFilters()) { + // Apply message-level keyword filtering + if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) { continue // Skip this message due to keyword filter } @@ -337,137 +289,6 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i return messages, currentUIDs, nil } -// buildOrChain creates a nested OR chain for multiple keywords -// Example: ["A", "B", "C"] becomes: A OR (B OR C) -func buildOrChain(keywords []string, headerKey string) *imap.SearchCriteria { - if len(keywords) == 0 { - return &imap.SearchCriteria{} - } - - if len(keywords) == 1 { - return &imap.SearchCriteria{ - Header: []imap.SearchCriteriaHeaderField{{ - Key: headerKey, - Value: keywords[0], - }}, - } - } - - // For multiple keywords, build nested OR structure - // Start with the last keyword - result := &imap.SearchCriteria{ - Header: []imap.SearchCriteriaHeaderField{{ - Key: headerKey, - Value: keywords[len(keywords)-1], - }}, - } - - // Build the chain backwards: each previous keyword becomes "keyword OR result" - for i := len(keywords) - 2; i >= 0; i-- { - keyword := keywords[i] - keywordCriteria := &imap.SearchCriteria{ - Header: []imap.SearchCriteriaHeaderField{{ - Key: headerKey, - Value: keyword, - }}, - } - - result = &imap.SearchCriteria{ - Or: [][2]imap.SearchCriteria{{ - *keywordCriteria, - *result, - }}, - } - } - - return result -} - -// searchMessagesAdvanced performs IMAP SEARCH with keyword filtering -// Returns sequence numbers of messages matching the search criteria -func (c *ImapClient) searchMessagesAdvanced(since *time.Time, messageFilter *config.MessageFilter) ([]uint32, error) { - // Build search criteria using structured approach - searchCriteria := &imap.SearchCriteria{} - - // Add date filter - if since != nil { - searchCriteria.Since = *since - } - - // Add subject keyword filters (use OR logic for multiple subject keywords) - if len(messageFilter.SubjectKeywords) > 0 { - if len(messageFilter.SubjectKeywords) == 1 { - // Single subject keyword - add to main criteria - searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{ - Key: "Subject", - Value: messageFilter.SubjectKeywords[0], - }) - } else { - // Multiple subject keywords - need to create a chain of OR conditions - // Build a nested OR structure: (A OR (B OR (C OR D))) - subjectCriteria := buildOrChain(messageFilter.SubjectKeywords, "Subject") - if len(searchCriteria.Header) > 0 || !searchCriteria.Since.IsZero() { - // Combine with existing criteria - searchCriteria.And(subjectCriteria) - } else { - *searchCriteria = *subjectCriteria - } - } - } - - // Add sender keyword filters (use OR logic for multiple sender keywords) - if len(messageFilter.SenderKeywords) > 0 { - if len(messageFilter.SenderKeywords) == 1 { - // Single sender keyword - add to main criteria - searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{ - Key: "From", - Value: messageFilter.SenderKeywords[0], - }) - } else { - // Multiple sender keywords - need to create a chain of OR conditions - senderCriteria := buildOrChain(messageFilter.SenderKeywords, "From") - // Always use AND to combine with existing criteria - searchCriteria.And(senderCriteria) - } - } - - // Add recipient keyword filters (use OR logic for multiple recipient keywords) - if len(messageFilter.RecipientKeywords) > 0 { - if len(messageFilter.RecipientKeywords) == 1 { - // Single recipient keyword - add to main criteria - searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{ - Key: "To", - Value: messageFilter.RecipientKeywords[0], - }) - } else { - // Multiple recipient keywords - need to create a chain of OR conditions - recipientCriteria := buildOrChain(messageFilter.RecipientKeywords, "To") - // Always use AND to combine with existing criteria - searchCriteria.And(recipientCriteria) - } - } - - log.Printf("Using IMAP SEARCH with keyword filters (subject: %v, sender: %v, recipient: %v)", - messageFilter.SubjectKeywords, messageFilter.SenderKeywords, messageFilter.RecipientKeywords) - - // Execute search - searchCmd := c.Search(searchCriteria, nil) - searchResults, err := searchCmd.Wait() - if err != nil { - return nil, fmt.Errorf("advanced search failed: %w", err) - } - - // Convert results to sequence numbers - seqNums := searchResults.AllSeqNums() - var uids []uint32 - for _, seqNum := range seqNums { - uids = append(uids, seqNum) - } - - log.Printf("Found %d messages matching advanced search criteria", len(uids)) - return uids, nil -} - // parseMessage parses an IMAP fetch response into our Message struct func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) { msg := &Message{ @@ -637,30 +458,27 @@ func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) erro return nil } + // ShouldProcessMessage checks if a message should be processed based on keyword filters -// serverSideFiltered indicates if subject/sender keywords were already filtered server-side via IMAP SEARCH -func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter, serverSideFiltered bool) bool { - // Skip subject and sender keyword checks if already filtered server-side - if !serverSideFiltered { - // Check subject keywords - if len(filter.SubjectKeywords) > 0 { - if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) { - return false +func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool { + // Check subject keywords + if len(filter.SubjectKeywords) > 0 { + if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) { + return false + } + } + + // Check sender keywords + if len(filter.SenderKeywords) > 0 { + senderMatch := false + for _, sender := range msg.From { + if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) { + senderMatch = true + break } } - - // Check sender keywords - if len(filter.SenderKeywords) > 0 { - senderMatch := false - for _, sender := range msg.From { - if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) { - senderMatch = true - break - } - } - if !senderMatch { - return false - } + if !senderMatch { + return false } } diff --git a/go/main.go b/go/main.go index e96ff56..fa04250 100644 --- a/go/main.go +++ b/go/main.go @@ -152,7 +152,7 @@ func processImapSource(source *config.MailSource, couchClient *couch.Client, dbN continue } } else { - fmt.Printf(" DRY-RUN: Would sync mailbox %s with %d current UIDs (mode: %s)\n", + fmt.Printf(" DRY-RUN: Would sync mailbox %s with %d current UIDs (mode: %s)\n", mailbox, len(currentUIDs), source.Mode) } @@ -200,7 +200,7 @@ func processImapSource(source *config.MailSource, couchClient *couch.Client, dbN fmt.Printf(" DRY-RUN: Would store %d messages from %s\n", len(messages), mailbox) // Show sample of what would be stored if len(docs) > 0 { - fmt.Printf(" DRY-RUN: Sample message ID: %s (Subject: %s)\n", + fmt.Printf(" DRY-RUN: Sample message ID: %s (Subject: %s)\n", docs[0].ID, docs[0].Subject) } } diff --git a/rust/src/cli.rs b/rust/src/cli.rs index fea0afa..abb9159 100644 --- a/rust/src/cli.rs +++ b/rust/src/cli.rs @@ -1,5 +1,5 @@ //! Command line interface for mail2couch -//! +//! //! This module handles command line argument parsing and bash completion generation, //! matching the behavior of the Go implementation. @@ -47,7 +47,7 @@ pub fn parse_command_line() -> CommandLineArgs { } CommandLineArgs { - config_path: matches.get_one::("config").cloned(), + config_path: matches.get_one::("config").map(|s| s.clone()), max_messages: matches.get_one::("max-messages").copied(), dry_run: matches.get_flag("dry-run"), generate_bash_completion: matches.get_flag("generate-bash-completion"), @@ -57,23 +57,20 @@ pub fn parse_command_line() -> CommandLineArgs { /// Generate bash completion script for mail2couch pub fn generate_bash_completion() { - let app_name = env::args() - .next() + let app_name = env::args().next() .map(|path| { - Path::new(&path) - .file_name() + Path::new(&path).file_name() .and_then(|name| name.to_str()) .unwrap_or("mail2couch") .to_string() }) .unwrap_or_else(|| "mail2couch".to_string()); - let script = format!( - r#"#!/bin/bash -# Bash completion script for {app_name} -# Generated automatically by {app_name} --generate-bash-completion + let script = format!(r#"#!/bin/bash +# Bash completion script for {} +# Generated automatically by {} --generate-bash-completion -_{app_name}_completions() {{ +_{}_completions() {{ local cur prev words cword _init_completion || return @@ -101,16 +98,15 @@ _{app_name}_completions() {{ }} # Register the completion function -complete -F _{app_name}_completions {app_name} +complete -F _{}_completions {} # Enable completion for common variations of the command name -if [[ "$({app_name} --help 2>/dev/null)" =~ "mail2couch" ]]; then - complete -F _{app_name}_completions mail2couch +if [[ "$({} --help 2>/dev/null)" =~ "mail2couch" ]]; then + complete -F _{}_completions mail2couch fi -"# - ); +"#, app_name, app_name, app_name, app_name, app_name, app_name, app_name); - print!("{script}"); + print!("{}", script); } #[cfg(test)] @@ -126,4 +122,4 @@ mod tests { }); // Just verify it doesn't panic, we can't easily test the output without capturing stdout } -} +} \ No newline at end of file diff --git a/rust/src/config.rs b/rust/src/config.rs index 49d243a..38a6f38 100644 --- a/rust/src/config.rs +++ b/rust/src/config.rs @@ -1,5 +1,5 @@ //! Configuration loading and management for mail2couch -//! +//! //! This module handles loading configuration from JSON files with automatic //! file discovery, matching the behavior of the Go implementation. @@ -117,7 +117,7 @@ impl Config { pub fn load_from_path(path: &str) -> Result { let content = fs::read_to_string(path)?; let mut config: Config = serde_json::from_str(&content)?; - + // Validate and set defaults for mail sources for source in &mut config.mail_sources { if source.mode.is_empty() { @@ -130,7 +130,7 @@ impl Config { }); } } - + Ok(config) } @@ -161,12 +161,7 @@ impl Config { // Add user directory paths if let Some(home_dir) = dirs::home_dir() { - candidates.push( - home_dir - .join(".config") - .join("mail2couch") - .join("config.json"), - ); + candidates.push(home_dir.join(".config").join("mail2couch").join("config.json")); candidates.push(home_dir.join(".mail2couch.json")); } @@ -177,7 +172,9 @@ impl Config { } } - Err(ConfigError::NotFound { paths: candidates }) + Err(ConfigError::NotFound { + paths: candidates, + }) } /// Load configuration with automatic file discovery @@ -227,10 +224,10 @@ mod tests { "#; let config: Config = serde_json::from_str(config_json).unwrap(); - + assert_eq!(config.couch_db.url, "http://localhost:5984"); assert_eq!(config.mail_sources.len(), 1); - + let source = &config.mail_sources[0]; assert_eq!(source.name, "Test Account"); assert_eq!(source.mode, "archive"); @@ -270,20 +267,20 @@ mod tests { fn test_config_file_discovery() { let temp_dir = tempdir().unwrap(); let config_path = temp_dir.path().join("config.json"); - + let config_content = r#" { "couchDb": {"url": "http://localhost:5984", "user": "admin", "password": "password"}, "mailSources": [] } "#; - + fs::write(&config_path, config_content).unwrap(); - + // Change to temp directory for relative path test let original_dir = std::env::current_dir().unwrap(); std::env::set_current_dir(&temp_dir).unwrap(); - + let args = CommandLineArgs { config_path: None, max_messages: None, @@ -291,11 +288,11 @@ mod tests { generate_bash_completion: false, help: false, }; - + let found_path = Config::find_config_file(&args).unwrap(); assert_eq!(found_path, PathBuf::from("config.json")); - + // Restore original directory std::env::set_current_dir(original_dir).unwrap(); } -} +} \ No newline at end of file diff --git a/rust/src/couch.rs b/rust/src/couch.rs index ac1e22c..881dc99 100644 --- a/rust/src/couch.rs +++ b/rust/src/couch.rs @@ -1,5 +1,5 @@ //! CouchDB client integration for mail2couch -//! +//! //! This module provides a CouchDB client that handles database operations //! for storing email messages and sync metadata. @@ -58,14 +58,14 @@ impl CouchClient { match operation().await { Ok(result) => { if attempt > 1 { - log::debug!("✅ CouchDB {operation_name} successful on attempt {attempt}"); + log::debug!("✅ CouchDB {} successful on attempt {}", operation_name, attempt); } return Ok(result); } Err(e) => { // Check if this is a retryable error let is_retryable = match &e.downcast_ref::() { - Some(CouchError::Http(_)) => true, // Network errors are retryable + Some(CouchError::Http(_)) => true, // Network errors are retryable Some(CouchError::CouchDb { status, .. }) => { // Retry on server errors (5xx) but not client errors (4xx) *status >= 500 @@ -74,7 +74,7 @@ impl CouchClient { }; last_error = Some(e); - + if is_retryable && attempt < MAX_RETRIES { log::warn!( "🔄 CouchDB {} attempt {} failed, retrying in {}ms: {}", @@ -120,13 +120,13 @@ impl CouchClient { pub async fn test_connection(&self) -> Result<()> { let url = format!("{}/", self.base_url); let mut request = self.client.get(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + if response.status().is_success() { Ok(()) } else { @@ -143,23 +143,18 @@ impl CouchClient { let url = format!("{}/{}", self.base_url, db_name); let mut request = self.client.put(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::CREATED | StatusCode::ACCEPTED => Ok(()), status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to create database {}: {} - {}", - db_name, - status, - error_text - )) + Err(anyhow!("Failed to create database {}: {} - {}", db_name, status, error_text)) } } } @@ -168,7 +163,7 @@ impl CouchClient { pub async fn database_exists(&self, db_name: &str) -> Result { let url = format!("{}/{}", self.base_url, db_name); let mut request = self.client.head(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } @@ -178,18 +173,14 @@ impl CouchClient { } /// Store a mail document in CouchDB with optional attachments and retry logic - pub async fn store_mail_document( - &self, - db_name: &str, - mut document: MailDocument, - ) -> Result { + pub async fn store_mail_document(&self, db_name: &str, mut document: MailDocument) -> Result { // Set the document ID if not already set if document.id.is_none() { document.set_id(); } let doc_id = document.id.as_ref().unwrap().clone(); - + // Check if document already exists to avoid duplicates if self.document_exists(db_name, &doc_id).await? { return Ok(doc_id); @@ -199,33 +190,30 @@ impl CouchClient { let encoded_doc_id = urlencoding::encode(&doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.put(&url).json(&document); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } - let response = request.send().await.map_err(CouchError::Http)?; - + let response = request.send().await + .map_err(|e| CouchError::Http(e))?; + match response.status() { StatusCode::CREATED | StatusCode::ACCEPTED => { - let couch_response: CouchResponse = - response.json().await.map_err(CouchError::Http)?; + let couch_response: CouchResponse = response.json().await + .map_err(|e| CouchError::Http(e))?; Ok(couch_response.id.unwrap_or_else(|| doc_id.clone())) } status => { - let error_text = response - .text() - .await + let error_text = response.text().await .unwrap_or_else(|_| "Failed to read error response".to_string()); Err(CouchError::CouchDb { status: status.as_u16(), message: error_text, - } - .into()) + }.into()) } } - }) - .await + }).await } /// Store an attachment for a document in CouchDB @@ -244,35 +232,26 @@ impl CouchClient { // Upload the attachment let encoded_doc_id = urlencoding::encode(doc_id); let encoded_attachment_name = urlencoding::encode(attachment_name); - let url = format!( - "{}/{}/{}/{}?rev={}", - self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev - ); - let mut request = self - .client + let url = format!("{}/{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev); + let mut request = self.client .put(&url) .header("Content-Type", content_type) .body(data.to_vec()); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::CREATED | StatusCode::ACCEPTED => { let couch_response: CouchResponse = response.json().await?; - Ok(couch_response.rev.unwrap_or(rev)) + Ok(couch_response.rev.unwrap_or_else(|| rev)) } status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to store attachment {}: {} - {}", - attachment_name, - status, - error_text - )) + Err(anyhow!("Failed to store attachment {}: {} - {}", attachment_name, status, error_text)) } } } @@ -282,13 +261,13 @@ impl CouchClient { let encoded_doc_id = urlencoding::encode(doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.get(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::OK => { let doc: Value = response.json().await?; @@ -297,24 +276,15 @@ impl CouchClient { StatusCode::NOT_FOUND => Ok(None), status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to get document {}: {} - {}", - doc_id, - status, - error_text - )) + Err(anyhow!("Failed to get document {}: {} - {}", doc_id, status, error_text)) } } } /// Store sync metadata in CouchDB - pub async fn store_sync_metadata( - &self, - db_name: &str, - metadata: &SyncMetadata, - ) -> Result { + pub async fn store_sync_metadata(&self, db_name: &str, metadata: &SyncMetadata) -> Result { let doc_id = metadata.id.as_ref().unwrap(); - + // Try to get existing document first to get the revision let mut metadata_to_store = metadata.clone(); if let Ok(existing) = self.get_sync_metadata(db_name, &metadata.mailbox).await { @@ -324,13 +294,13 @@ impl CouchClient { let encoded_doc_id = urlencoding::encode(doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.put(&url).json(&metadata_to_store); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::CREATED | StatusCode::ACCEPTED => { let couch_response: CouchResponse = response.json().await?; @@ -338,43 +308,35 @@ impl CouchClient { } status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to store sync metadata {}: {} - {}", - doc_id, - status, - error_text - )) + Err(anyhow!("Failed to store sync metadata {}: {} - {}", doc_id, status, error_text)) } } } /// Get sync metadata for a mailbox pub async fn get_sync_metadata(&self, db_name: &str, mailbox: &str) -> Result { - let doc_id = format!("sync_metadata_{mailbox}"); + let doc_id = format!("sync_metadata_{}", mailbox); let encoded_doc_id = urlencoding::encode(&doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.get(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::OK => { let metadata: SyncMetadata = response.json().await?; Ok(metadata) } - StatusCode::NOT_FOUND => Err(CouchError::NotFound { id: doc_id }.into()), + StatusCode::NOT_FOUND => { + Err(CouchError::NotFound { id: doc_id }.into()) + } status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to get sync metadata {}: {} - {}", - doc_id, - status, - error_text - )) + Err(anyhow!("Failed to get sync metadata {}: {} - {}", doc_id, status, error_text)) } } } @@ -384,7 +346,7 @@ impl CouchClient { let encoded_doc_id = urlencoding::encode(doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.head(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } @@ -397,13 +359,13 @@ impl CouchClient { pub async fn get_database_info(&self, db_name: &str) -> Result { let url = format!("{}/{}", self.base_url, db_name); let mut request = self.client.get(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + match response.status() { StatusCode::OK => { let info: Value = response.json().await?; @@ -411,12 +373,7 @@ impl CouchClient { } status => { let error_text = response.text().await?; - Err(anyhow!( - "Failed to get database info for {}: {} - {}", - db_name, - status, - error_text - )) + Err(anyhow!("Failed to get database info for {}: {} - {}", db_name, status, error_text)) } } } @@ -425,24 +382,21 @@ impl CouchClient { pub async fn get_mailbox_uids(&self, db_name: &str, mailbox: &str) -> Result> { let url = format!("{}/{}/_all_docs", self.base_url, db_name); let query_params = [ - ("startkey", format!("\"{mailbox}\"")), - ("endkey", format!("\"{mailbox}\\ufff0\"")), // High Unicode character for range end + ("startkey", format!("\"{}\"", mailbox)), + ("endkey", format!("\"{}\\ufff0\"", mailbox)), // High Unicode character for range end ("include_docs", "false".to_string()), ]; let mut request = self.client.get(&url).query(&query_params); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + if !response.status().is_success() { - return Err(anyhow!( - "Failed to query stored messages: {}", - response.status() - )); + return Err(anyhow!("Failed to query stored messages: {}", response.status())); } let result: serde_json::Value = response.json().await?; @@ -452,7 +406,7 @@ impl CouchClient { for row in rows { if let Some(id) = row["id"].as_str() { // Parse UID from document ID format: {mailbox}_{uid} - if let Some(uid_str) = id.strip_prefix(&format!("{mailbox}_")) { + if let Some(uid_str) = id.strip_prefix(&format!("{}_", mailbox)) { if let Ok(uid) = uid_str.parse::() { uids.push(uid); } @@ -470,45 +424,36 @@ impl CouchClient { let encoded_doc_id = urlencoding::encode(doc_id); let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id); let mut request = self.client.get(&url); - + if let Some((username, password)) = &self.auth { request = request.basic_auth(username, Some(password)); } let response = request.send().await?; - + if response.status() == StatusCode::NOT_FOUND { return Ok(()); // Document already doesn't exist } let doc: Value = response.json().await?; - let rev = doc["_rev"] - .as_str() + let rev = doc["_rev"].as_str() .ok_or_else(|| anyhow!("Document {} has no _rev field", doc_id))?; - // Now delete the document - let delete_url = format!( - "{}/{}/{}?rev={}", - self.base_url, db_name, encoded_doc_id, rev - ); + // Now delete the document + let delete_url = format!("{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, rev); let mut delete_request = self.client.delete(&delete_url); - + if let Some((username, password)) = &self.auth { delete_request = delete_request.basic_auth(username, Some(password)); } let delete_response = delete_request.send().await?; - + match delete_response.status() { StatusCode::OK | StatusCode::ACCEPTED => Ok(()), status => { let error_text = delete_response.text().await?; - Err(anyhow!( - "Failed to delete document {}: {} - {}", - doc_id, - status, - error_text - )) + Err(anyhow!("Failed to delete document {}: {} - {}", doc_id, status, error_text)) } } } @@ -536,4 +481,4 @@ mod tests { // Note: Additional integration tests would require a running CouchDB instance // These would be similar to the Go implementation tests -} +} \ No newline at end of file diff --git a/rust/src/filters.rs b/rust/src/filters.rs index 59f011c..b445cac 100644 --- a/rust/src/filters.rs +++ b/rust/src/filters.rs @@ -1,5 +1,5 @@ //! Folder and message filtering functionality for mail2couch -//! +//! //! This module provides filtering logic for determining which folders and messages //! should be processed, matching the behavior of the Go implementation. @@ -14,18 +14,12 @@ pub fn should_process_folder(folder_name: &str, filter: &FolderFilter) -> bool { let included = if filter.include.is_empty() { true } else { - filter - .include - .iter() - .any(|pattern| matches_pattern(folder_name, pattern)) + filter.include.iter().any(|pattern| matches_pattern(folder_name, pattern)) }; // If included, check if it's excluded if included { - !filter - .exclude - .iter() - .any(|pattern| matches_pattern(folder_name, pattern)) + !filter.exclude.iter().any(|pattern| matches_pattern(folder_name, pattern)) } else { false } @@ -51,8 +45,7 @@ fn matches_pattern(folder_name: &str, pattern: &str) -> bool { /// Apply folder filters to a list of folders and return the filtered list pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec { - folders - .iter() + folders.iter() .filter(|folder| should_process_folder(folder, filter)) .cloned() .collect() @@ -60,23 +53,19 @@ pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec /// Expand wildcard patterns to match actual folder names /// This is useful for validating patterns against available folders -pub fn expand_patterns( - patterns: &[String], - available_folders: &[String], -) -> Result> { +pub fn expand_patterns(patterns: &[String], available_folders: &[String]) -> Result> { let mut expanded = HashSet::new(); for pattern in patterns { - let matches: Vec<_> = available_folders - .iter() + let matches: Vec<_> = available_folders.iter() .filter(|folder| matches_pattern(folder, pattern)) .cloned() .collect(); if matches.is_empty() { - log::warn!("Pattern '{pattern}' matches no folders"); + log::warn!("Pattern '{}' matches no folders", pattern); } else { - log::debug!("Pattern '{pattern}' matches: {matches:?}"); + log::debug!("Pattern '{}' matches: {:?}", pattern, matches); expanded.extend(matches); } } @@ -86,31 +75,26 @@ pub fn expand_patterns( /// Validate folder filter patterns against available folders /// Returns warnings for patterns that don't match any folders -pub fn validate_folder_patterns( - filter: &FolderFilter, - available_folders: &[String], -) -> Vec { +pub fn validate_folder_patterns(filter: &FolderFilter, available_folders: &[String]) -> Vec { let mut warnings = Vec::new(); // Check include patterns for pattern in &filter.include { - let matches = available_folders - .iter() + let matches = available_folders.iter() .any(|folder| matches_pattern(folder, pattern)); - + if !matches { - warnings.push(format!("Include pattern '{pattern}' matches no folders")); + warnings.push(format!("Include pattern '{}' matches no folders", pattern)); } } // Check exclude patterns for pattern in &filter.exclude { - let matches = available_folders - .iter() + let matches = available_folders.iter() .any(|folder| matches_pattern(folder, pattern)); - + if !matches { - warnings.push(format!("Exclude pattern '{pattern}' matches no folders")); + warnings.push(format!("Exclude pattern '{}' matches no folders", pattern)); } } @@ -119,16 +103,17 @@ pub fn validate_folder_patterns( /// Get a summary of folder filtering results pub fn get_filter_summary( - all_folders: &[String], - filtered_folders: &[String], - filter: &FolderFilter, + all_folders: &[String], + filtered_folders: &[String], + filter: &FolderFilter ) -> String { let total_count = all_folders.len(); let filtered_count = filtered_folders.len(); let excluded_count = total_count - filtered_count; let mut summary = format!( - "Folder filtering: {total_count} total, {filtered_count} selected, {excluded_count} excluded" + "Folder filtering: {} total, {} selected, {} excluded", + total_count, filtered_count, excluded_count ); if !filter.include.is_empty() { @@ -170,7 +155,7 @@ mod tests { assert!(matches_pattern("Work/Projects", "*/Projects")); assert!(matches_pattern("Work/Archive", "Work/A*")); assert!(matches_pattern("Sent", "?ent")); - + assert!(!matches_pattern("INBOX", "Sent")); assert!(!matches_pattern("Work/Projects", "Personal/*")); assert!(!matches_pattern("INBOX", "??")); @@ -185,7 +170,7 @@ mod tests { }; let filtered = filter_folders(&folders, &filter); - + assert!(filtered.contains(&"INBOX".to_string())); assert!(filtered.contains(&"Work/Projects".to_string())); assert!(!filtered.contains(&"Trash".to_string())); @@ -196,16 +181,12 @@ mod tests { fn test_folder_filtering_specific() { let folders = create_test_folders(); let filter = FolderFilter { - include: vec![ - "INBOX".to_string(), - "Sent".to_string(), - "Work/*".to_string(), - ], + include: vec!["INBOX".to_string(), "Sent".to_string(), "Work/*".to_string()], exclude: vec!["*Temp*".to_string()], }; let filtered = filter_folders(&folders, &filter); - + assert!(filtered.contains(&"INBOX".to_string())); assert!(filtered.contains(&"Sent".to_string())); assert!(filtered.contains(&"Work/Projects".to_string())); @@ -224,7 +205,7 @@ mod tests { }; let filtered = filter_folders(&folders, &filter); - + // Should include everything except excluded assert!(filtered.contains(&"INBOX".to_string())); assert!(filtered.contains(&"Work/Projects".to_string())); @@ -237,9 +218,9 @@ mod tests { fn test_pattern_expansion() { let folders = create_test_folders(); let patterns = vec!["Work/*".to_string(), "Personal/*".to_string()]; - + let expanded = expand_patterns(&patterns, &folders).unwrap(); - + assert!(expanded.contains("Work/Projects")); assert!(expanded.contains("Work/Archive")); assert!(expanded.contains("Work/Temp")); @@ -258,7 +239,7 @@ mod tests { }; let warnings = validate_folder_patterns(&filter, &folders); - + assert_eq!(warnings.len(), 2); assert!(warnings.iter().any(|w| w.contains("NonExistent/*"))); assert!(warnings.iter().any(|w| w.contains("AnotherNonExistent"))); @@ -274,9 +255,9 @@ mod tests { let filtered = filter_folders(&folders, &filter); let summary = get_filter_summary(&folders, &filtered, &filter); - + assert!(summary.contains(&format!("{} total", folders.len()))); assert!(summary.contains(&format!("{} selected", filtered.len()))); assert!(summary.contains("exclude:")); } -} +} \ No newline at end of file diff --git a/rust/src/imap.rs b/rust/src/imap.rs index 9a2e813..0b24ed7 100644 --- a/rust/src/imap.rs +++ b/rust/src/imap.rs @@ -1,10 +1,10 @@ //! IMAP client functionality for mail2couch -//! +//! //! This module provides IMAP client operations for connecting to mail servers, //! listing mailboxes, and retrieving messages. use crate::config::{MailSource, MessageFilter}; -use crate::schemas::{AttachmentStub, MailDocument}; +use crate::schemas::{MailDocument, AttachmentStub}; use anyhow::{anyhow, Result}; use async_imap::types::Fetch; use async_imap::{Client, Session}; @@ -112,7 +112,7 @@ impl ImapClient { session: None, source, }; - + client.establish_connection_with_retry().await?; Ok(client) } @@ -128,7 +128,7 @@ impl ImapClient { match self.establish_connection().await { Ok(()) => { if attempt > 1 { - log::info!("✅ IMAP connection successful on attempt {attempt}"); + log::info!("✅ IMAP connection successful on attempt {}", attempt); } return Ok(()); } @@ -154,22 +154,20 @@ impl ImapClient { )) } + /// Establish connection to IMAP server async fn establish_connection(&mut self) -> Result<()> { // Connect to the IMAP server let addr = format!("{}:{}", self.source.host, self.source.port); - let tcp_stream = TcpStream::connect(&addr) - .await - .map_err(|e| ImapError::Connection(format!("Failed to connect to {addr}: {e}")))?; + let tcp_stream = TcpStream::connect(&addr).await + .map_err(|e| ImapError::Connection(format!("Failed to connect to {}: {}", addr, e)))?; // Determine if we should use TLS based on port let imap_stream = if self.should_use_tls() { // Use TLS for secure connection (typically port 993) let tls_connector = TlsConnector::new(); - let tls_stream = tls_connector - .connect(&self.source.host, tcp_stream) - .await - .map_err(|e| ImapError::Connection(format!("TLS connection failed: {e}")))?; + let tls_stream = tls_connector.connect(&self.source.host, tcp_stream).await + .map_err(|e| ImapError::Connection(format!("TLS connection failed: {}", e)))?; ImapStream::Tls(tls_stream) } else { // Use plain connection (typically port 143 or test environments) @@ -177,11 +175,7 @@ impl ImapClient { }; // Log connection type for debugging - let connection_type = if self.should_use_tls() { - "TLS" - } else { - "Plain" - }; + let connection_type = if self.should_use_tls() { "TLS" } else { "Plain" }; log::debug!( "Connecting to {}:{} using {} connection", self.source.host, @@ -196,7 +190,7 @@ impl ImapClient { let session = client .login(&self.source.user, &self.source.password) .await - .map_err(|e| ImapError::Authentication(format!("Login failed: {e:?}")))?; + .map_err(|e| ImapError::Authentication(format!("Login failed: {:?}", e)))?; self.session = Some(session); Ok(()) @@ -225,25 +219,17 @@ impl ImapClient { /// List all mailboxes pub async fn list_mailboxes(&mut self) -> Result> { - let session = self - .session - .as_mut() + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; - let mut mailboxes = session - .list(Some(""), Some("*")) - .await - .map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {e:?}")))?; + let mut mailboxes = session.list(Some(""), Some("*")).await + .map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {:?}", e)))?; let mut mailbox_names = Vec::new(); while let Some(mailbox_result) = mailboxes.next().await { match mailbox_result { Ok(mailbox) => mailbox_names.push(mailbox.name().to_string()), - Err(e) => { - return Err( - ImapError::Operation(format!("Error processing mailbox: {e:?}")).into(), - ) - } + Err(e) => return Err(ImapError::Operation(format!("Error processing mailbox: {:?}", e)).into()), } } @@ -251,13 +237,8 @@ impl ImapClient { } /// List mailboxes using IMAP LIST with server-side pattern filtering - pub async fn list_filtered_mailboxes( - &mut self, - filter: &crate::config::FolderFilter, - ) -> Result> { - let session = self - .session - .as_mut() + pub async fn list_filtered_mailboxes(&mut self, filter: &crate::config::FolderFilter) -> Result> { + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; let mut all_mailboxes = Vec::new(); @@ -270,14 +251,13 @@ impl ImapClient { // Use IMAP LIST with each include pattern for server-side filtering for pattern in &filter.include { - log::debug!("Listing mailboxes with pattern: {pattern}"); - - let mut mailboxes = session.list(Some(""), Some(pattern)).await.map_err(|e| { - log::warn!("Failed to list mailboxes with pattern '{pattern}': {e:?}"); - ImapError::Operation(format!( - "Failed to list mailboxes with pattern '{pattern}': {e:?}" - )) - })?; + log::debug!("Listing mailboxes with pattern: {}", pattern); + + let mut mailboxes = session.list(Some(""), Some(pattern)).await + .map_err(|e| { + log::warn!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e); + ImapError::Operation(format!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e)) + })?; while let Some(mailbox_result) = mailboxes.next().await { match mailbox_result { @@ -288,7 +268,7 @@ impl ImapClient { } } Err(e) => { - log::warn!("Error processing mailbox with pattern '{pattern}': {e:?}"); + log::warn!("Error processing mailbox with pattern '{}': {:?}", pattern, e); continue; } } @@ -303,10 +283,9 @@ impl ImapClient { let filtered_mailboxes: Vec = all_mailboxes .into_iter() .filter(|mailbox| { - !filter - .exclude - .iter() - .any(|exclude_pattern| self.matches_imap_pattern(exclude_pattern, mailbox)) + !filter.exclude.iter().any(|exclude_pattern| { + self.matches_imap_pattern(exclude_pattern, mailbox) + }) }) .collect(); @@ -321,8 +300,8 @@ impl ImapClient { } // Handle simple prefix wildcard: "Work*" should match "Work/Projects" - if pattern.ends_with('*') && !pattern[..pattern.len() - 1].contains('*') { - let prefix = &pattern[..pattern.len() - 1]; + if pattern.ends_with('*') && !pattern[..pattern.len()-1].contains('*') { + let prefix = &pattern[..pattern.len()-1]; return name.starts_with(prefix); } @@ -334,7 +313,7 @@ impl ImapClient { // Handle contains wildcard: "*Temp*" should match "Work/Temp/Archive" if pattern.starts_with('*') && pattern.ends_with('*') { - let middle = &pattern[1..pattern.len() - 1]; + let middle = &pattern[1..pattern.len()-1]; return name.contains(middle); } @@ -344,14 +323,11 @@ impl ImapClient { /// Select a mailbox pub async fn select_mailbox(&mut self, mailbox: &str) -> Result { - let session = self - .session - .as_mut() + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; - let mailbox_data = session.select(mailbox).await.map_err(|e| { - ImapError::Operation(format!("Failed to select mailbox {mailbox}: {e:?}")) - })?; + let mailbox_data = session.select(mailbox).await + .map_err(|e| ImapError::Operation(format!("Failed to select mailbox {}: {:?}", mailbox, e)))?; Ok(MailboxInfo { name: mailbox.to_string(), @@ -364,22 +340,19 @@ impl ImapClient { /// Search for messages using IMAP SEARCH command with retry logic /// Returns UIDs of matching messages - pub async fn search_messages( - &mut self, - since_date: Option<&DateTime>, - ) -> Result> { + pub async fn search_messages(&mut self, since_date: Option<&DateTime>) -> Result> { const MAX_RETRIES: u32 = 3; const RETRY_DELAY_MS: u64 = 500; - + let mut last_error = None; for attempt in 1..=MAX_RETRIES { let result = self.search_messages_internal(since_date).await; - + match result { Ok(uids) => { if attempt > 1 { - log::debug!("✅ IMAP search successful on attempt {attempt}"); + log::debug!("✅ IMAP search successful on attempt {}", attempt); } return Ok(uids); } @@ -406,35 +379,29 @@ impl ImapClient { } /// Internal search implementation without retry logic - async fn search_messages_internal( - &mut self, - since_date: Option<&DateTime>, - ) -> Result> { - let session = self - .session - .as_mut() + async fn search_messages_internal(&mut self, since_date: Option<&DateTime>) -> Result> { + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; let search_query = if let Some(since) = since_date { // Format date as required by IMAP (DD-MMM-YYYY) // IMAP months are 3-letter abbreviations in English let formatted_date = since.format("%d-%b-%Y").to_string(); - log::debug!("Searching for messages since: {formatted_date}"); - format!("SINCE {formatted_date}") + log::debug!("Searching for messages since: {}", formatted_date); + format!("SINCE {}", formatted_date) } else { log::debug!("Searching for all messages"); "ALL".to_string() }; - log::debug!("IMAP search query: {search_query}"); - - let uids = session.uid_search(&search_query).await.map_err(|e| { - ImapError::Operation(format!("Search failed with query '{search_query}': {e:?}")) - })?; + log::debug!("IMAP search query: {}", search_query); + + let uids = session.uid_search(&search_query).await + .map_err(|e| ImapError::Operation(format!("Search failed with query '{}': {:?}", search_query, e)))?; let uid_vec: Vec = uids.into_iter().collect(); log::debug!("Found {} messages matching search criteria", uid_vec.len()); - + Ok(uid_vec) } @@ -447,9 +414,7 @@ impl ImapClient { subject_keywords: Option<&[String]>, from_keywords: Option<&[String]>, ) -> Result> { - let session = self - .session - .as_mut() + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; let mut search_parts = Vec::new(); @@ -457,12 +422,12 @@ impl ImapClient { // Add date filters if let Some(since) = since_date { let formatted_date = since.format("%d-%b-%Y").to_string(); - search_parts.push(format!("SINCE {formatted_date}")); + search_parts.push(format!("SINCE {}", formatted_date)); } if let Some(before) = before_date { let formatted_date = before.format("%d-%b-%Y").to_string(); - search_parts.push(format!("BEFORE {formatted_date}")); + search_parts.push(format!("BEFORE {}", formatted_date)); } // Add subject keyword filters @@ -486,39 +451,25 @@ impl ImapClient { search_parts.join(" ") }; - log::debug!("Advanced IMAP search query: {search_query}"); - - let uids = session.uid_search(&search_query).await.map_err(|e| { - ImapError::Operation(format!( - "Advanced search failed with query '{search_query}': {e:?}" - )) - })?; + log::debug!("Advanced IMAP search query: {}", search_query); + + let uids = session.uid_search(&search_query).await + .map_err(|e| ImapError::Operation(format!("Advanced search failed with query '{}': {:?}", search_query, e)))?; let uid_vec: Vec = uids.into_iter().collect(); - log::debug!( - "Found {} messages matching advanced search criteria", - uid_vec.len() - ); - + log::debug!("Found {} messages matching advanced search criteria", uid_vec.len()); + Ok(uid_vec) } /// Fetch message by UID with attachment data - pub async fn fetch_message( - &mut self, - uid: u32, - mailbox: &str, - ) -> Result<(MailDocument, Vec<(String, String, Vec)>)> { - let session = self - .session - .as_mut() + pub async fn fetch_message(&mut self, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec)>)> { + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; // Fetch message headers and body - let mut messages = session - .uid_fetch(format!("{uid}"), "RFC822") - .await - .map_err(|e| ImapError::Operation(format!("Failed to fetch message {uid}: {e:?}")))?; + let mut messages = session.uid_fetch(format!("{}", uid), "RFC822").await + .map_err(|e| ImapError::Operation(format!("Failed to fetch message {}: {:?}", uid, e)))?; // Collect the first message if let Some(message_result) = messages.next().await { @@ -528,10 +479,7 @@ impl ImapClient { drop(messages); self.parse_message(&message, uid, mailbox).await } - Err(e) => Err(ImapError::Operation(format!( - "Failed to process message {uid}: {e:?}" - )) - .into()), + Err(e) => Err(ImapError::Operation(format!("Failed to process message {}: {:?}", uid, e)).into()), } } else { Err(anyhow!("Message {} not found", uid)) @@ -539,19 +487,12 @@ impl ImapClient { } /// Fetch multiple messages by UIDs with attachment data - pub async fn fetch_messages( - &mut self, - uids: &[u32], - max_count: Option, - mailbox: &str, - ) -> Result)>)>> { + pub async fn fetch_messages(&mut self, uids: &[u32], max_count: Option, mailbox: &str) -> Result)>)>> { if uids.is_empty() { return Ok(Vec::new()); } - let session = self - .session - .as_mut() + let session = self.session.as_mut() .ok_or_else(|| anyhow!("Not connected to IMAP server"))?; // Limit the number of messages if specified @@ -566,27 +507,24 @@ impl ImapClient { }; // Create UID sequence - let uid_sequence = uids_to_fetch - .iter() + let uid_sequence = uids_to_fetch.iter() .map(|uid| uid.to_string()) .collect::>() .join(","); // Fetch messages - let mut messages = session - .uid_fetch(&uid_sequence, "RFC822") - .await - .map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {e:?}")))?; + let mut messages = session.uid_fetch(&uid_sequence, "RFC822").await + .map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {:?}", e)))?; // Collect all messages first to avoid borrowing issues let mut fetched_messages = Vec::new(); while let Some(message_result) = messages.next().await { match message_result { Ok(message) => fetched_messages.push(message), - Err(e) => log::warn!("Failed to fetch message: {e:?}"), + Err(e) => log::warn!("Failed to fetch message: {:?}", e), } } - + // Drop the messages stream to release the session borrow drop(messages); @@ -596,7 +534,7 @@ impl ImapClient { match self.parse_message(message, uid, mailbox).await { Ok((doc, attachments)) => mail_documents.push((doc, attachments)), Err(e) => { - log::warn!("Failed to parse message {uid}: {e}"); + log::warn!("Failed to parse message {}: {}", uid, e); } } } @@ -606,14 +544,8 @@ impl ImapClient { } /// Parse a raw IMAP message into a MailDocument with attachment data - async fn parse_message( - &self, - message: &Fetch, - uid: u32, - mailbox: &str, - ) -> Result<(MailDocument, Vec<(String, String, Vec)>)> { - let body = message - .body() + async fn parse_message(&self, message: &Fetch, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec)>)> { + let body = message.body() .ok_or_else(|| ImapError::Parsing("No message body found".to_string()))?; // Parse the email using mail-parser library @@ -622,7 +554,7 @@ impl ImapClient { // Extract sender addresses let from = self.extract_addresses(&parsed_message, "From"); - + // Extract recipient addresses let to = self.extract_addresses(&parsed_message, "To"); @@ -634,7 +566,7 @@ impl ImapClient { // Extract date let date = if let Some(date_time) = parsed_message.get_date() { - DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(Utc::now) + DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(|| Utc::now()) } else { Utc::now() }; @@ -646,8 +578,7 @@ impl ImapClient { let headers = self.extract_headers(&parsed_message); // Extract attachments and their data - let (has_attachments, _attachment_stubs, attachment_data) = - self.extract_attachments_with_data(&parsed_message); + let (has_attachments, _attachment_stubs, attachment_data) = self.extract_attachments_with_data(&parsed_message); let mail_doc = MailDocument::new( uid.to_string(), @@ -667,11 +598,7 @@ impl ImapClient { // Log attachment information if !attachment_data.is_empty() { - log::info!( - "Found {} attachments for message {}", - attachment_data.len(), - uid - ); + log::info!("Found {} attachments for message {}", attachment_data.len(), uid); } Ok((mail_doc, attachment_data)) @@ -683,7 +610,7 @@ impl ImapClient { // For address headers, use as_text() and parse manually // mail-parser doesn't provide a direct address parsing method let header_text = header.as_text_ref().unwrap_or(""); - + // Simple address extraction - split by comma and clean up header_text .split(',') @@ -746,55 +673,45 @@ impl ImapClient { /// Extract all headers from a parsed message fn extract_headers(&self, message: &Message) -> HashMap> { let mut headers = HashMap::new(); - + for header in message.get_headers() { let name = header.name().to_lowercase(); let value = match header.value().as_text_ref() { Some(text) => text.to_string(), None => format!("{:?}", header.value()), // Fallback for non-text values }; - - headers.entry(name).or_insert_with(Vec::new).push(value); + + headers.entry(name) + .or_insert_with(Vec::new) + .push(value); } - + headers } /// Extract attachments from a parsed message with binary data /// Returns (has_attachments, attachment_stubs, attachment_data) - #[allow(clippy::type_complexity)] - fn extract_attachments_with_data( - &self, - message: &Message, - ) -> ( - bool, - HashMap, - Vec<(String, String, Vec)>, - ) { + fn extract_attachments_with_data(&self, message: &Message) -> (bool, HashMap, Vec<(String, String, Vec)>) { let mut attachment_stubs = HashMap::new(); let mut attachment_data = Vec::new(); - + // Iterate through all message parts looking for attachments for (index, part) in message.parts.iter().enumerate() { // Check if this part is an attachment if let Some(content_type) = part.get_content_type() { - let is_attachment = self.is_attachment_part(part, content_type); - + let is_attachment = self.is_attachment_part(part, &content_type); + if is_attachment { // Generate a filename for the attachment let filename = self.get_attachment_filename(part, index); - + // Get the binary content data using the proper mail-parser API // This works for both text and binary attachments (images, PDFs, etc.) let body_data = part.get_contents().to_vec(); - log::debug!( - "Found attachment content: {} bytes (content-type: {})", - body_data.len(), - content_type.c_type - ); - + log::debug!("Found attachment content: {} bytes (content-type: {})", body_data.len(), content_type.c_type); + let content_type_str = content_type.c_type.to_string(); - + // Create attachment stub - get_contents() always returns the full data if !body_data.is_empty() { let attachment_stub = AttachmentStub { @@ -802,39 +719,36 @@ impl ImapClient { length: Some(body_data.len() as u64), stub: None, // Will be stored as actual attachment data }; - + attachment_stubs.insert(filename.clone(), attachment_stub); attachment_data.push((filename, content_type_str, body_data)); } } } } - + let has_attachments = !attachment_stubs.is_empty(); (has_attachments, attachment_stubs, attachment_data) } + /// Determine if a message part is an attachment - fn is_attachment_part( - &self, - part: &mail_parser::MessagePart, - content_type: &mail_parser::ContentType, - ) -> bool { + fn is_attachment_part(&self, part: &mail_parser::MessagePart, content_type: &mail_parser::ContentType) -> bool { // Check Content-Disposition header first if let Some(disposition) = part.get_content_disposition() { return disposition.c_type.to_lowercase() == "attachment"; } - + // If no explicit disposition, check content type // Consider non-text types as potential attachments let main_type = content_type.c_type.split('/').next().unwrap_or(""); match main_type { - "text" => false, // Text parts are usually body content + "text" => false, // Text parts are usually body content "multipart" => false, // Multipart containers are not attachments - _ => true, // Images, applications, etc. are likely attachments + _ => true, // Images, applications, etc. are likely attachments } } - + /// Generate a filename for an attachment fn get_attachment_filename(&self, part: &mail_parser::MessagePart, index: usize) -> String { // Try to get filename from Content-Disposition @@ -848,7 +762,7 @@ impl ImapClient { } } } - + // Try to get filename from Content-Type if let Some(content_type) = part.get_content_type() { // Find name in attributes vector @@ -860,16 +774,16 @@ impl ImapClient { } } } - + // Generate a default filename based on content type and index if let Some(content_type) = part.get_content_type() { let extension = self.get_extension_from_content_type(&content_type.c_type); - format!("attachment_{index}{extension}") + format!("attachment_{}{}", index, extension) } else { - format!("attachment_{index}.bin") + format!("attachment_{}.bin", index) } } - + /// Get file extension from MIME content type fn get_extension_from_content_type(&self, content_type: &str) -> &'static str { match content_type { @@ -891,23 +805,22 @@ impl ImapClient { /// Close the IMAP connection pub async fn close(self) -> Result<()> { if let Some(mut session) = self.session { - session - .logout() - .await - .map_err(|e| ImapError::Operation(format!("Logout failed: {e:?}")))?; + session.logout().await + .map_err(|e| ImapError::Operation(format!("Logout failed: {:?}", e)))?; } Ok(()) } } /// Apply message filters to determine if a message should be processed -pub fn should_process_message(mail_doc: &MailDocument, filter: &MessageFilter) -> bool { +pub fn should_process_message( + mail_doc: &MailDocument, + filter: &MessageFilter, +) -> bool { // Check subject keywords if !filter.subject_keywords.is_empty() { let subject_lower = mail_doc.subject.to_lowercase(); - let has_subject_keyword = filter - .subject_keywords - .iter() + let has_subject_keyword = filter.subject_keywords.iter() .any(|keyword| subject_lower.contains(&keyword.to_lowercase())); if !has_subject_keyword { return false; @@ -916,13 +829,12 @@ pub fn should_process_message(mail_doc: &MailDocument, filter: &MessageFilter) - // Check sender keywords if !filter.sender_keywords.is_empty() { - let has_sender_keyword = mail_doc.from.iter().any(|from_addr| { - let from_lower = from_addr.to_lowercase(); - filter - .sender_keywords - .iter() - .any(|keyword| from_lower.contains(&keyword.to_lowercase())) - }); + let has_sender_keyword = mail_doc.from.iter() + .any(|from_addr| { + let from_lower = from_addr.to_lowercase(); + filter.sender_keywords.iter() + .any(|keyword| from_lower.contains(&keyword.to_lowercase())) + }); if !has_sender_keyword { return false; } @@ -930,13 +842,12 @@ pub fn should_process_message(mail_doc: &MailDocument, filter: &MessageFilter) - // Check recipient keywords if !filter.recipient_keywords.is_empty() { - let has_recipient_keyword = mail_doc.to.iter().any(|to_addr| { - let to_lower = to_addr.to_lowercase(); - filter - .recipient_keywords - .iter() - .any(|keyword| to_lower.contains(&keyword.to_lowercase())) - }); + let has_recipient_keyword = mail_doc.to.iter() + .any(|to_addr| { + let to_lower = to_addr.to_lowercase(); + filter.recipient_keywords.iter() + .any(|keyword| to_lower.contains(&keyword.to_lowercase())) + }); if !has_recipient_keyword { return false; } @@ -987,7 +898,7 @@ mod tests { #[test] fn test_rfc822_parsing() { - let _client = ImapClient { + let client = ImapClient { session: None, source: MailSource { name: "test".to_string(), @@ -1007,4 +918,4 @@ mod tests { // This test needs to be updated to use actual message parsing // For now, we'll skip the detailed test since it requires a full email message } -} +} \ No newline at end of file diff --git a/rust/src/lib.rs b/rust/src/lib.rs index 8f58cac..e1417f8 100644 --- a/rust/src/lib.rs +++ b/rust/src/lib.rs @@ -14,16 +14,16 @@ //! The library uses well-defined CouchDB document schemas that are compatible //! with the Go implementation. See the `schemas` module for details. -pub mod cli; -pub mod config; -pub mod couch; -pub mod filters; -pub mod imap; pub mod schemas; +pub mod config; +pub mod cli; +pub mod couch; +pub mod imap; +pub mod filters; pub mod sync; // Re-export main types for convenience -pub use config::{CommandLineArgs, Config, MailSource}; +pub use schemas::{MailDocument, SyncMetadata, AttachmentStub, generate_database_name}; +pub use config::{Config, MailSource, CommandLineArgs}; pub use couch::CouchClient; -pub use imap::ImapClient; -pub use schemas::{generate_database_name, AttachmentStub, MailDocument, SyncMetadata}; +pub use imap::ImapClient; \ No newline at end of file diff --git a/rust/src/main.rs b/rust/src/main.rs index 8d4b2c6..ed2c83a 100644 --- a/rust/src/main.rs +++ b/rust/src/main.rs @@ -1,7 +1,11 @@ use anyhow::Result; use env_logger::Env; use log::{error, info}; -use mail2couch::{cli::parse_command_line, config::Config, sync::SyncCoordinator}; +use mail2couch::{ + cli::parse_command_line, + config::Config, + sync::SyncCoordinator, +}; use std::process; #[tokio::main] @@ -14,7 +18,7 @@ async fn main() { // Run the main application if let Err(e) = run(args).await { - error!("❌ Application failed: {e}"); + error!("❌ Application failed: {}", e); process::exit(1); } } @@ -27,7 +31,7 @@ async fn run(args: mail2couch::config::CommandLineArgs) -> Result<()> { info!("Using configuration file: {}", config_path.display()); if let Some(max) = args.max_messages { - info!("Maximum messages per mailbox: {max}"); + info!("Maximum messages per mailbox: {}", max); } else { info!("Maximum messages per mailbox: unlimited"); } @@ -63,11 +67,7 @@ fn print_config_summary(config: &mail2couch::config::Config) { info!(" Mail sources: {}", config.mail_sources.len()); for (i, source) in config.mail_sources.iter().enumerate() { - let status = if source.enabled { - "enabled" - } else { - "disabled" - }; + let status = if source.enabled { "enabled" } else { "disabled" }; info!( " {}: {} ({}) - {} ({})", i + 1, @@ -88,11 +88,8 @@ fn print_config_summary(config: &mail2couch::config::Config) { info!(" Since: {:?}", source.message_filter.since); } if !source.message_filter.subject_keywords.is_empty() { - info!( - " Subject keywords: {:?}", - source.message_filter.subject_keywords - ); + info!(" Subject keywords: {:?}", source.message_filter.subject_keywords); } } } -} +} \ No newline at end of file diff --git a/rust/src/schemas.rs b/rust/src/schemas.rs index 3431b5d..5f75145 100644 --- a/rust/src/schemas.rs +++ b/rust/src/schemas.rs @@ -121,7 +121,6 @@ pub struct SyncMetadata { impl MailDocument { /// Create a new MailDocument with required fields - #[allow(clippy::too_many_arguments)] pub fn new( source_uid: String, mailbox: String, @@ -135,7 +134,7 @@ impl MailDocument { ) -> Self { let now = Utc::now(); Self { - id: None, // Will be set when storing to CouchDB + id: None, // Will be set when storing to CouchDB rev: None, // Managed by CouchDB attachments: None, source_uid, @@ -173,7 +172,7 @@ impl SyncMetadata { ) -> Self { let now = Utc::now(); Self { - id: Some(format!("sync_metadata_{mailbox}")), + id: Some(format!("sync_metadata_{}", mailbox)), rev: None, // Managed by CouchDB doc_type: "sync_metadata".to_string(), mailbox, @@ -200,15 +199,7 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String { .to_lowercase() .chars() .map(|c| { - if c.is_ascii_alphanumeric() - || c == '_' - || c == '$' - || c == '(' - || c == ')' - || c == '+' - || c == '-' - || c == '/' - { + if c.is_ascii_alphanumeric() || c == '_' || c == '$' || c == '(' || c == ')' || c == '+' || c == '-' || c == '/' { c } else { '_' @@ -218,9 +209,9 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String { // Ensure starts with a letter if valid_name.is_empty() || !valid_name.chars().next().unwrap().is_ascii_lowercase() { - valid_name = format!("m2c_mail_{valid_name}"); + valid_name = format!("m2c_mail_{}", valid_name); } else { - valid_name = format!("m2c_{valid_name}"); + valid_name = format!("m2c_{}", valid_name); } valid_name @@ -232,14 +223,8 @@ mod tests { #[test] fn test_generate_database_name() { - assert_eq!( - generate_database_name("Personal Gmail", ""), - "m2c_personal_gmail" - ); - assert_eq!( - generate_database_name("", "user@example.com"), - "m2c_user_example_com" - ); + assert_eq!(generate_database_name("Personal Gmail", ""), "m2c_personal_gmail"); + assert_eq!(generate_database_name("", "user@example.com"), "m2c_user_example_com"); assert_eq!(generate_database_name("123work", ""), "m2c_mail_123work"); } @@ -258,14 +243,19 @@ mod tests { ); assert_eq!(doc.generate_id(), "INBOX_123"); - + doc.set_id(); assert_eq!(doc.id, Some("INBOX_123".to_string())); } #[test] fn test_sync_metadata_creation() { - let metadata = SyncMetadata::new("INBOX".to_string(), Utc::now(), 456, 100); + let metadata = SyncMetadata::new( + "INBOX".to_string(), + Utc::now(), + 456, + 100, + ); assert_eq!(metadata.id, Some("sync_metadata_INBOX".to_string())); assert_eq!(metadata.doc_type, "sync_metadata"); @@ -273,4 +263,4 @@ mod tests { assert_eq!(metadata.last_message_uid, 456); assert_eq!(metadata.message_count, 100); } -} +} \ No newline at end of file diff --git a/rust/src/sync.rs b/rust/src/sync.rs index 11f9b53..29e9e95 100644 --- a/rust/src/sync.rs +++ b/rust/src/sync.rs @@ -1,16 +1,16 @@ //! Synchronization logic for mail2couch -//! +//! //! This module coordinates the synchronization process between IMAP servers and CouchDB, //! implementing incremental sync with metadata tracking. -use crate::config::{CommandLineArgs, Config, MailSource}; +use crate::config::{Config, MailSource, CommandLineArgs}; use crate::couch::CouchClient; use crate::filters::{get_filter_summary, validate_folder_patterns}; -use crate::imap::{should_process_message, ImapClient}; -use crate::schemas::{generate_database_name, SyncMetadata}; +use crate::imap::{ImapClient, should_process_message}; +use crate::schemas::{SyncMetadata, generate_database_name}; use anyhow::{anyhow, Result}; use chrono::{DateTime, Utc}; -use log::{debug, error, info, warn}; +use log::{info, warn, error, debug}; /// Main synchronization coordinator pub struct SyncCoordinator { @@ -46,7 +46,7 @@ impl SyncCoordinator { /// Create a new sync coordinator pub fn new(config: Config, args: CommandLineArgs) -> Result { let couch_client = CouchClient::new(&config.couch_db)?; - + Ok(SyncCoordinator { config, couch_client, @@ -57,9 +57,7 @@ impl SyncCoordinator { /// Test connections to all services pub async fn test_connections(&self) -> Result<()> { info!("Testing CouchDB connection..."); - self.couch_client - .test_connection() - .await + self.couch_client.test_connection().await .map_err(|e| anyhow!("CouchDB connection failed: {}", e))?; info!("✅ CouchDB connection successful"); @@ -70,10 +68,9 @@ impl SyncCoordinator { } info!("Testing IMAP connection to {}...", source.name); - let imap_client = ImapClient::connect(source.clone()) - .await + let imap_client = ImapClient::connect(source.clone()).await .map_err(|e| anyhow!("IMAP connection to {} failed: {}", source.name, e))?; - + imap_client.close().await?; info!("✅ IMAP connection to {} successful", source.name); } @@ -98,7 +95,9 @@ impl SyncCoordinator { Ok(result) => { info!( "✅ Completed sync for {}: {} messages across {} mailboxes", - result.source_name, result.total_messages, result.mailboxes_processed + result.source_name, + result.total_messages, + result.mailboxes_processed ); results.push(result); } @@ -115,52 +114,41 @@ impl SyncCoordinator { /// Synchronize a single mail source async fn sync_source(&mut self, source: &MailSource) -> Result { let start_time = Utc::now(); - + // Generate database name let db_name = generate_database_name(&source.name, &source.user); - info!("Using database: {db_name}"); + info!("Using database: {}", db_name); // Create database if it doesn't exist (skip in dry-run mode) if !self.args.dry_run { self.couch_client.create_database(&db_name).await?; } else { - info!("🔍 DRY-RUN: Would create database {db_name}"); + info!("🔍 DRY-RUN: Would create database {}", db_name); } // Connect to IMAP server let mut imap_client = ImapClient::connect(source.clone()).await?; // Use IMAP LIST with patterns for server-side filtering - let filtered_mailboxes = imap_client - .list_filtered_mailboxes(&source.folder_filter) - .await?; - info!( - "Found {} matching mailboxes after server-side filtering", - filtered_mailboxes.len() - ); + let filtered_mailboxes = imap_client.list_filtered_mailboxes(&source.folder_filter).await?; + info!("Found {} matching mailboxes after server-side filtering", filtered_mailboxes.len()); // For validation and summary, we still need the full list - let all_mailboxes = if !source.folder_filter.include.is_empty() - || !source.folder_filter.exclude.is_empty() - { + let all_mailboxes = if !source.folder_filter.include.is_empty() || !source.folder_filter.exclude.is_empty() { // Only fetch all mailboxes if we have filters (for logging/validation) - imap_client - .list_mailboxes() - .await - .unwrap_or_else(|_| Vec::new()) + imap_client.list_mailboxes().await.unwrap_or_else(|_| Vec::new()) } else { filtered_mailboxes.clone() }; if !all_mailboxes.is_empty() { - let filter_summary = - get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter); - info!("{filter_summary}"); + let filter_summary = get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter); + info!("{}", filter_summary); // Validate folder patterns and show warnings let warnings = validate_folder_patterns(&source.folder_filter, &all_mailboxes); for warning in warnings { - warn!("{warning}"); + warn!("{}", warning); } } @@ -169,12 +157,9 @@ impl SyncCoordinator { let mut total_messages = 0; for mailbox in &filtered_mailboxes { - info!("Syncing mailbox: {mailbox}"); - - match self - .sync_mailbox(&mut imap_client, &db_name, mailbox, source) - .await - { + info!("Syncing mailbox: {}", mailbox); + + match self.sync_mailbox(&mut imap_client, &db_name, mailbox, source).await { Ok(result) => { if result.messages_deleted > 0 { info!( @@ -198,7 +183,7 @@ impl SyncCoordinator { mailbox_results.push(result); } Err(e) => { - error!(" ❌ Failed to sync mailbox {mailbox}: {e}"); + error!(" ❌ Failed to sync mailbox {}: {}", mailbox, e); // Continue with other mailboxes } } @@ -229,45 +214,35 @@ impl SyncCoordinator { // Select the mailbox let mailbox_info = imap_client.select_mailbox(mailbox).await?; - debug!( - "Selected mailbox {}: {} messages", - mailbox, mailbox_info.exists - ); + debug!("Selected mailbox {}: {} messages", mailbox, mailbox_info.exists); // Get last sync metadata (skip in dry-run mode) let since_date = if !self.args.dry_run { match self.couch_client.get_sync_metadata(db_name, mailbox).await { Ok(metadata) => { - info!( - " Found sync metadata, last sync: {}", - metadata.last_sync_time - ); + info!(" Found sync metadata, last sync: {}", metadata.last_sync_time); Some(metadata.last_sync_time) } Err(_) => { info!(" No sync metadata found, performing full sync"); // Parse since date from message filter if provided - source.message_filter.since.as_ref().and_then(|since_str| { - DateTime::parse_from_str( - &format!("{since_str} 00:00:00 +0000"), - "%Y-%m-%d %H:%M:%S %z", - ) - .map(|dt| dt.with_timezone(&Utc)) - .ok() - }) + source.message_filter.since.as_ref() + .and_then(|since_str| { + DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z") + .map(|dt| dt.with_timezone(&Utc)) + .ok() + }) } } } else { info!(" 🔍 DRY-RUN: Would check for sync metadata"); // In dry-run mode, use config since date if available - source.message_filter.since.as_ref().and_then(|since_str| { - DateTime::parse_from_str( - &format!("{since_str} 00:00:00 +0000"), - "%Y-%m-%d %H:%M:%S %z", - ) - .map(|dt| dt.with_timezone(&Utc)) - .ok() - }) + source.message_filter.since.as_ref() + .and_then(|since_str| { + DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z") + .map(|dt| dt.with_timezone(&Utc)) + .ok() + }) }; // Search for messages using server-side IMAP SEARCH with keyword filtering when possible @@ -283,41 +258,32 @@ impl SyncCoordinator { } else { Some(source.message_filter.sender_keywords.as_slice()) }; - + info!(" Using IMAP SEARCH with keyword filters"); - imap_client - .search_messages_advanced( - since_date.as_ref(), - None, // before_date - subject_keywords, - from_keywords, - ) - .await? + imap_client.search_messages_advanced( + since_date.as_ref(), + None, // before_date + subject_keywords, + from_keywords, + ).await? } else { // Use simple date-based search imap_client.search_messages(since_date.as_ref()).await? }; - info!( - " Found {} messages matching search criteria", - message_uids.len() - ); + info!(" Found {} messages matching search criteria", message_uids.len()); // Handle sync mode - check for deleted messages (skip in dry-run mode) let mut messages_deleted = 0; if source.mode == "sync" { if !self.args.dry_run { - messages_deleted = self - .handle_deleted_messages(db_name, mailbox, &message_uids) - .await + messages_deleted = self.handle_deleted_messages(db_name, mailbox, &message_uids).await .unwrap_or_else(|e| { - warn!(" Failed to handle deleted messages: {e}"); + warn!(" Failed to handle deleted messages: {}", e); 0 }); - + if messages_deleted > 0 { - info!( - " 🗑️ Deleted {messages_deleted} messages that no longer exist on server" - ); + info!(" 🗑️ Deleted {} messages that no longer exist on server", messages_deleted); } } else { info!(" 🔍 DRY-RUN: Would check for deleted messages in sync mode"); @@ -339,7 +305,7 @@ impl SyncCoordinator { // Apply max message limit if specified let uids_to_process = if let Some(max) = self.args.max_messages { if message_uids.len() > max as usize { - info!(" Limiting to {max} messages due to --max-messages flag"); + info!(" Limiting to {} messages due to --max-messages flag", max); &message_uids[..max as usize] } else { &message_uids @@ -349,10 +315,8 @@ impl SyncCoordinator { }; // Fetch and process messages - let messages = imap_client - .fetch_messages(uids_to_process, self.args.max_messages, mailbox) - .await?; - + let messages = imap_client.fetch_messages(uids_to_process, self.args.max_messages, mailbox).await?; + let mut messages_stored = 0; let mut messages_skipped = 0; let mut last_uid = None; @@ -369,63 +333,50 @@ impl SyncCoordinator { // Store the message document first (skip in dry-run mode) if !self.args.dry_run { - match self - .couch_client - .store_mail_document(db_name, mail_doc) - .await - { + match self.couch_client.store_mail_document(db_name, mail_doc).await { Ok(doc_id) => { messages_stored += 1; - + // Store attachments if any exist if !attachments.is_empty() { for (filename, content_type, data) in attachments { - match self - .couch_client - .store_attachment( - db_name, - &doc_id, - &filename, - &content_type, - &data, - ) - .await - { + match self.couch_client.store_attachment( + db_name, + &doc_id, + &filename, + &content_type, + &data, + ).await { Ok(_) => { - debug!(" Stored attachment: {filename}"); + debug!(" Stored attachment: {}", filename); } Err(e) => { - warn!(" Failed to store attachment {filename}: {e}"); + warn!(" Failed to store attachment {}: {}", filename, e); } } } } - + // Parse UID from source_uid if let Ok(uid) = uid_str.parse::() { last_uid = Some(last_uid.map_or(uid, |prev: u32| prev.max(uid))); } } Err(e) => { - warn!(" Failed to store message {uid_str}: {e}"); + warn!(" Failed to store message {}: {}", uid_str, e); messages_skipped += 1; } } } else { // In dry-run mode, simulate successful storage messages_stored += 1; - debug!( - " 🔍 DRY-RUN: Would store message {} (Subject: {})", - uid_str, mail_doc.subject - ); - + debug!(" 🔍 DRY-RUN: Would store message {} (Subject: {})", + uid_str, mail_doc.subject); + if !attachments.is_empty() { - debug!( - " 🔍 DRY-RUN: Would store {} attachments", - attachments.len() - ); + debug!(" 🔍 DRY-RUN: Would store {} attachments", attachments.len()); } - + // Parse UID from source_uid if let Ok(uid) = uid_str.parse::() { last_uid = Some(last_uid.map_or(uid, |prev: u32| prev.max(uid))); @@ -436,20 +387,18 @@ impl SyncCoordinator { // Update sync metadata (skip in dry-run mode) if let Some(uid) = last_uid { if !self.args.dry_run { - let sync_metadata = - SyncMetadata::new(mailbox.to_string(), start_time, uid, messages_stored); + let sync_metadata = SyncMetadata::new( + mailbox.to_string(), + start_time, + uid, + messages_stored, + ); - if let Err(e) = self - .couch_client - .store_sync_metadata(db_name, &sync_metadata) - .await - { - warn!(" Failed to store sync metadata: {e}"); + if let Err(e) = self.couch_client.store_sync_metadata(db_name, &sync_metadata).await { + warn!(" Failed to store sync metadata: {}", e); } } else { - info!( - " 🔍 DRY-RUN: Would update sync metadata (last UID: {uid}, {messages_stored} messages)" - ); + info!(" 🔍 DRY-RUN: Would update sync metadata (last UID: {}, {} messages)", uid, messages_stored); } } @@ -475,28 +424,27 @@ impl SyncCoordinator { ) -> Result { // Get all stored message UIDs for this mailbox from CouchDB let stored_uids = self.get_stored_message_uids(db_name, mailbox).await?; - + if stored_uids.is_empty() { return Ok(0); // No stored messages to delete } // Find UIDs that exist in CouchDB but not on the server - let server_uid_set: std::collections::HashSet = - current_server_uids.iter().cloned().collect(); + let server_uid_set: std::collections::HashSet = current_server_uids.iter().cloned().collect(); let mut deleted_count = 0; for stored_uid in stored_uids { if !server_uid_set.contains(&stored_uid) { // This message was deleted from the server, remove it from CouchDB - let doc_id = format!("{mailbox}_{stored_uid}"); - + let doc_id = format!("{}_{}", mailbox, stored_uid); + match self.couch_client.delete_document(db_name, &doc_id).await { Ok(_) => { - debug!(" Deleted document: {doc_id}"); + debug!(" Deleted document: {}", doc_id); deleted_count += 1; } Err(e) => { - warn!(" Failed to delete document {doc_id}: {e}"); + warn!(" Failed to delete document {}: {}", doc_id, e); } } } @@ -551,16 +499,18 @@ impl SyncCoordinator { info!("{}", "=".repeat(50)); if self.args.dry_run { info!( - "📊 DRY-RUN Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages found" + "📊 DRY-RUN Total: {} sources, {} mailboxes, {} messages found", + total_sources, total_mailboxes, total_messages ); } else { info!( - "📊 Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages" + "📊 Total: {} sources, {} mailboxes, {} messages", + total_sources, total_mailboxes, total_messages ); } if let Some(max) = self.args.max_messages { - info!("⚠️ Message limit was applied: {max} per mailbox"); + info!("⚠️ Message limit was applied: {} per mailbox", max); } if self.args.dry_run { @@ -581,21 +531,23 @@ mod tests { user: "admin".to_string(), password: "password".to_string(), }, - mail_sources: vec![MailSource { - name: "Test Account".to_string(), - enabled: true, - protocol: "imap".to_string(), - host: "localhost".to_string(), - port: 3143, - user: "testuser".to_string(), - password: "testpass".to_string(), - mode: "archive".to_string(), - folder_filter: FolderFilter { - include: vec!["*".to_string()], - exclude: vec!["Trash".to_string()], - }, - message_filter: MessageFilter::default(), - }], + mail_sources: vec![ + MailSource { + name: "Test Account".to_string(), + enabled: true, + protocol: "imap".to_string(), + host: "localhost".to_string(), + port: 3143, + user: "testuser".to_string(), + password: "testpass".to_string(), + mode: "archive".to_string(), + folder_filter: FolderFilter { + include: vec!["*".to_string()], + exclude: vec!["Trash".to_string()], + }, + message_filter: MessageFilter::default(), + } + ], } } @@ -616,4 +568,4 @@ mod tests { } // Additional integration tests would require running services -} +} \ No newline at end of file diff --git a/test-both-implementations.sh b/test-both-implementations.sh index 74cccf0..35c8e05 100755 --- a/test-both-implementations.sh +++ b/test-both-implementations.sh @@ -73,13 +73,13 @@ check_results() { echo -e "${BLUE}🔍 Checking results...${NC}" echo -e "${BLUE} Listing all databases:${NC}" - curl -s -u admin:password http://localhost:5984/_all_dbs | python3 -m json.tool + curl -s http://localhost:5984/_all_dbs | python3 -m json.tool echo -e "\n${BLUE} Go implementation databases:${NC}" for db in go_wildcard_all_folders_test go_work_pattern_test go_specific_folders_only; do db_name="m2c_${db}" - if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then - doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))") + if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then + doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))") echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}" else echo -e "${RED} ❌ ${db_name}: not found${NC}" @@ -89,8 +89,8 @@ check_results() { echo -e "\n${BLUE} Rust implementation databases:${NC}" for db in rust_wildcard_all_folders_test rust_work_pattern_test rust_specific_folders_only; do db_name="m2c_${db}" - if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then - doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))") + if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then + doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))") echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}" else echo -e "${RED} ❌ ${db_name}: not found${NC}" diff --git a/docs/test-config-comparison.md b/test-config-comparison.md similarity index 100% rename from docs/test-config-comparison.md rename to test-config-comparison.md