Compare commits
3 commits
436276f0ef
...
d3d104ee71
| Author | SHA1 | Date | |
|---|---|---|---|
| d3d104ee71 | |||
| f80f89cdd5 | |||
| 6c387abfbb |
27 changed files with 1398 additions and 1151 deletions
112
ANALYSIS.md
112
ANALYSIS.md
|
|
@ -1,112 +0,0 @@
|
|||
### Comprehensive Analysis of `mail2couch` Implementations
|
||||
|
||||
This document provides an updated, in-depth analysis of the `mail2couch` project, integrating findings from the original `ANALYSIS.md` with a fresh review of the current Go and Rust codebases. It evaluates the current state, compares the two implementations, and outlines a roadmap for future improvements.
|
||||
|
||||
---
|
||||
|
||||
### 1. Current State of the Implementations
|
||||
|
||||
The project currently consists of two distinct implementations of the same core tool.
|
||||
|
||||
* **The Go Implementation**: This is a mature, functional, and straightforward command-line tool. It is built on a simple, sequential architecture and effectively synchronizes emails from IMAP servers to CouchDB. It serves as a solid baseline for the project's core functionality.
|
||||
|
||||
* **The Rust Implementation**: Contrary to the description in the original `ANALYSIS.md`, the Rust version is **no longer a non-functional placeholder**. It is now a complete, and in many ways, more advanced alternative to the Go version. It is built on a highly modular, asynchronous architecture, prioritizing performance, robustness, and an expanded feature set.
|
||||
|
||||
---
|
||||
|
||||
### 2. Analysis of Points from Original `ANALYSIS.md`
|
||||
|
||||
Several key issues and suggestions were raised in the original analysis. Here is their current status:
|
||||
|
||||
* **`Incomplete Rust Implementation`**: **(Addressed)** The Rust implementation is now fully functional and surpasses the Go version in features and robustness.
|
||||
* **`Performance for Large-Scale Use (Concurrency)`**: **(Addressed in Rust)** The Go version remains sequential. The Rust version, however, is fully asynchronous, allowing for concurrent network operations, which directly addresses this performance concern.
|
||||
* **`Inefficient Keyword Filtering`**: **(Addressed in Rust)** The Go version still performs keyword filtering client-side. The Rust version implements server-side filtering using `IMAP SEARCH` with keywords, which is significantly more efficient.
|
||||
* **`Primary Weakness: Security`**: **(Still an Issue)** Both implementations still require plaintext passwords in the configuration file. This remains a primary weakness.
|
||||
* **`Missing Core Feature: Web Interface`**: **(Still an Issue)** This feature has not been implemented in either version.
|
||||
* **`Usability Enhancement: Dry-Run Mode`**: **(✅ Resolved)** Both implementations now include comprehensive `--dry-run/-n` mode functionality that allows safe configuration testing without making any CouchDB changes.
|
||||
|
||||
---
|
||||
|
||||
### 3. Comparative Analysis: Go vs. Rust
|
||||
|
||||
#### **The Go Version**
|
||||
|
||||
* **Pros**:
|
||||
* **Simplicity**: The code is sequential and easy to follow, making it highly approachable for new contributors.
|
||||
* **Stability**: It provides a solid, functional baseline that effectively accomplishes the core mission of the project.
|
||||
* **Fast Compilation**: Quick compile times make for a fast development cycle.
|
||||
* **Dry-Run Support**: Now includes comprehensive `--dry-run` mode for safe configuration testing.
|
||||
* **Cons**:
|
||||
* **Performance**: The lack of concurrency makes it slow for users with multiple accounts or large mailboxes.
|
||||
* **Inefficiency**: Client-side keyword filtering wastes bandwidth and processing time.
|
||||
* **Basic Error Handling**: The absence of retry logic makes it brittle in the face of transient network errors.
|
||||
|
||||
#### **The Rust Version**
|
||||
|
||||
* **Pros**:
|
||||
* **Performance**: The `async` architecture provides superior performance through concurrency.
|
||||
* **Robustness**: Automatic retry logic for network calls makes it highly resilient to temporary failures.
|
||||
* **Feature-Rich**: Implements more efficient server-side filtering, better folder-matching logic, and a more professional CLI.
|
||||
* **Safety & Maintainability**: The modular design and Rust's compile-time guarantees make the code safer and easier to maintain and extend.
|
||||
* **Comprehensive Dry-Run**: Includes detailed `--dry-run` mode with enhanced simulation logging and summary reporting.
|
||||
* **Cons**:
|
||||
* **Complexity**: The codebase is significantly more complex due to its asynchronous nature, abstract design, and the inherent learning curve of Rust.
|
||||
* **Slower Compilation**: Longer compile times can slow down development.
|
||||
|
||||
---
|
||||
|
||||
### 4. Recent Implementation Updates
|
||||
|
||||
#### **Dry-Run Mode Implementation (August 2025)**
|
||||
|
||||
Both Go and Rust implementations now include comprehensive `--dry-run` functionality:
|
||||
|
||||
##### **Go Implementation Features:**
|
||||
- **CLI Integration**: Added `--dry-run/-n` flag using pflag with GNU-style options
|
||||
- **Comprehensive Skipping**: All CouchDB write operations bypassed in dry-run mode
|
||||
- **IMAP Preservation**: Maintains full IMAP operations for realistic email discovery
|
||||
- **Detailed Simulation**: Shows what would be done with informative logging
|
||||
- **Enhanced Reporting**: Clear distinction between dry-run and normal mode output
|
||||
- **Bash Completion**: Updated completion script includes new flag
|
||||
|
||||
##### **Rust Implementation Features:**
|
||||
- **CLI Integration**: Added `--dry-run/-n` flag using clap with structured argument parsing
|
||||
- **Advanced Simulation**: Detailed logging of what would be stored including message subjects
|
||||
- **Async-Safe Skipping**: All async CouchDB operations properly bypassed
|
||||
- **Enhanced Summary**: Comprehensive dry-run vs normal mode reporting with emoji indicators
|
||||
- **Test Coverage**: All tests updated to include new dry_run field
|
||||
|
||||
##### **Implementation Benefits:**
|
||||
- **Risk Mitigation**: Users can validate configurations without database changes
|
||||
- **Debugging Aid**: Shows exactly what emails would be processed and stored
|
||||
- **Development Tool**: Enables safe testing of configuration changes
|
||||
- **Documentation**: Demonstrates the full sync process without side effects
|
||||
|
||||
This addresses the critical usability requirement identified in the original analysis and significantly improves the user experience for configuration validation and troubleshooting.
|
||||
|
||||
---
|
||||
|
||||
### 5. Future Improvements and Missing Features
|
||||
|
||||
This roadmap combines suggestions from both analyses, prioritizing the most impactful changes.
|
||||
|
||||
#### **Tier 1: Critical Needs**
|
||||
|
||||
1. **Fix the Security Model (Both)**: This is the most urgent issue.
|
||||
* **Short-Term**: Add support for reading credentials from environment variables (e.g., `M2C_IMAP_PASSWORD`).
|
||||
* **Long-Term**: Implement OAuth2 for modern providers like Gmail and Outlook. This is the industry standard and eliminates the need to store passwords.
|
||||
2. **Implement a Web Interface (Either)**: As noted in the original analysis, this is the key missing feature for making the archived data useful. This would involve creating CouchDB design documents and a simple web server to render the views.
|
||||
3. ~~**Add a `--dry-run` Mode (Both)**~~: **✅ COMPLETED** - Both implementations now include comprehensive dry-run functionality with the `--dry-run/-n` flag that allows users to test their configuration safely before making any changes to their database.
|
||||
|
||||
#### **Tier 2: High-Impact Enhancements**
|
||||
|
||||
1. **Add Concurrency to the Go Version**: To bring the Go implementation closer to the performance of the Rust version, it should be updated to use goroutines to process accounts and/or mailboxes in parallel.
|
||||
2. **Improve Attachment Handling in Rust**: The `TODO` in the Rust IMAP client for parsing binary attachments should be completed to ensure all attachment types are saved correctly.
|
||||
3. **URL-Encode Document IDs in Rust**: The CouchDB client in the Rust version should URL-encode document IDs to prevent errors when mailbox names contain special characters.
|
||||
4. **Add Progress Indicators (Rust)**: For a better user experience during long syncs, the Rust version would benefit greatly from progress bars (e.g., using the `indicatif` crate).
|
||||
|
||||
#### **Tier 3: "Nice-to-Have" Features**
|
||||
|
||||
1. **Interactive Setup (Either)**: A `mail2couch setup` command to interactively generate the `config.json` file would significantly improve first-time user experience.
|
||||
2. **Support for Other Protocols/Backends (Either)**: Extend the tool to support POP3 or JMAP, or to use other databases like PostgreSQL or Elasticsearch as a storage backend.
|
||||
3. **Backfill Command (Either)**: A `--backfill-all` flag to ignore existing sync metadata and perform a complete re-sync of an account.
|
||||
|
|
@ -1,560 +0,0 @@
|
|||
# Go vs Rust Implementation Comparison
|
||||
|
||||
This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch, helping users and developers choose the best implementation for their needs.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The mail2couch project offers two distinct architectural approaches to email backup:
|
||||
|
||||
- **Go Implementation**: A straightforward, sequential approach emphasizing simplicity and ease of understanding
|
||||
- **Rust Implementation**: A sophisticated, asynchronous architecture prioritizing performance, reliability, and advanced features
|
||||
|
||||
**Key Finding**: The Rust implementation (~3,056 LOC across 9 modules) is significantly more feature-complete and architecturally advanced than the Go implementation (~1,355 LOC across 4 modules), representing a mature evolution rather than a simple port.
|
||||
|
||||
---
|
||||
|
||||
## Architecture & Design Philosophy
|
||||
|
||||
### Go Implementation: Sequential Simplicity
|
||||
|
||||
**Design Philosophy**: Straightforward, imperative programming with minimal abstraction
|
||||
|
||||
- **Processing Model**: Sequential processing of sources → mailboxes → messages
|
||||
- **Error Handling**: Basic error propagation with continue-on-error for non-critical failures
|
||||
- **Modularity**: Simple package structure (`config`, `couch`, `mail`, `main`)
|
||||
- **State Management**: Minimal state, mostly function-based operations
|
||||
|
||||
```go
|
||||
// Example: Sequential processing approach
|
||||
func processImapSource(source *config.MailSource, couchClient *couch.Client,
|
||||
dbName string, maxMessages int, dryRun bool) error {
|
||||
// Connect to IMAP server
|
||||
imapClient, err := mail.NewImapClient(source)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to connect to IMAP server: %w", err)
|
||||
}
|
||||
defer imapClient.Logout()
|
||||
|
||||
// Process each mailbox sequentially
|
||||
for _, mailbox := range mailboxes {
|
||||
// Process messages one by one
|
||||
messages, currentUIDs, err := imapClient.GetMessages(...)
|
||||
// Store messages synchronously
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Rust Implementation: Async Orchestration
|
||||
|
||||
**Design Philosophy**: Modular, type-safe architecture with comprehensive error handling
|
||||
|
||||
- **Processing Model**: Asynchronous coordination with concurrent network operations
|
||||
- **Error Handling**: Sophisticated retry logic, structured error types, graceful degradation
|
||||
- **Modularity**: Well-separated concerns (`cli`, `config`, `couch`, `imap`, `sync`, `filters`, `schemas`)
|
||||
- **State Management**: Stateful coordinator pattern with proper resource management
|
||||
|
||||
```rust
|
||||
// Example: Asynchronous coordination approach
|
||||
impl SyncCoordinator {
|
||||
pub async fn sync_all_sources(&mut self) -> Result<Vec<SourceSyncResult>> {
|
||||
let mut results = Vec::new();
|
||||
let sources = self.config.mail_sources.clone();
|
||||
|
||||
for source in &sources {
|
||||
if !source.enabled {
|
||||
info!("Skipping disabled source: {}", source.name);
|
||||
continue;
|
||||
}
|
||||
|
||||
match self.sync_source(source).await {
|
||||
Ok(result) => {
|
||||
info!("✅ Completed sync for {}: {} messages across {} mailboxes",
|
||||
result.source_name, result.total_messages, result.mailboxes_processed);
|
||||
results.push(result);
|
||||
}
|
||||
Err(e) => {
|
||||
error!("❌ Failed to sync source {}: {}", source.name, e);
|
||||
// Continue with other sources even if one fails
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(results)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Performance & Scalability
|
||||
|
||||
### Concurrency Models
|
||||
|
||||
| Aspect | Go Implementation | Rust Implementation |
|
||||
|--------|------------------|-------------------|
|
||||
| **Processing Model** | Sequential (blocking) | Asynchronous (non-blocking) |
|
||||
| **Account Processing** | One at a time | One at a time with internal concurrency |
|
||||
| **Mailbox Processing** | One at a time | One at a time with async I/O |
|
||||
| **Message Processing** | One at a time | Batch processing with async operations |
|
||||
| **Network Operations** | Blocking I/O | Non-blocking async I/O |
|
||||
|
||||
### IMAP Filtering Efficiency
|
||||
|
||||
**Go: Client-Side Filtering**
|
||||
```go
|
||||
// Downloads ALL messages first, then filters locally
|
||||
messages := imap.FetchAll()
|
||||
filtered := []Message{}
|
||||
for _, msg := range messages {
|
||||
if ShouldProcessMessage(msg, filter) {
|
||||
filtered = append(filtered, msg)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Rust: Server-Side Filtering**
|
||||
```rust
|
||||
// Filters on server, only downloads matching messages
|
||||
pub async fn search_messages_advanced(
|
||||
&mut self,
|
||||
since_date: Option<&DateTime<Utc>>,
|
||||
subject_keywords: Option<&[String]>,
|
||||
from_keywords: Option<&[String]>,
|
||||
) -> Result<Vec<u32>> {
|
||||
let mut search_parts = Vec::new();
|
||||
|
||||
if let Some(keywords) = subject_keywords {
|
||||
for keyword in keywords {
|
||||
search_parts.push(format!("SUBJECT \"{}\"", keyword));
|
||||
}
|
||||
}
|
||||
// Server processes the filter, returns only matching UIDs
|
||||
}
|
||||
```
|
||||
|
||||
**Performance Impact**: For a mailbox with 10,000 emails where you only want recent messages:
|
||||
- **Go**: Downloads all 10,000 emails, then filters locally
|
||||
- **Rust**: Server filters first, downloads only matching emails (potentially 10x less data transfer)
|
||||
|
||||
### Error Recovery and Resilience
|
||||
|
||||
**Go: Basic Error Handling**
|
||||
```go
|
||||
err := processImapSource(&source, couchClient, dbName, args.MaxMessages, args.DryRun)
|
||||
if err != nil {
|
||||
log.Printf("ERROR: Failed to process IMAP source %s: %v", source.Name, err)
|
||||
}
|
||||
// Continues with next source, no retry logic
|
||||
```
|
||||
|
||||
**Rust: Intelligent Retry Logic**
|
||||
```rust
|
||||
async fn retry_operation<F, Fut, T>(&self, operation_name: &str, operation: F) -> Result<T>
|
||||
where F: Fn() -> Fut, Fut: std::future::Future<Output = Result<T>>
|
||||
{
|
||||
const MAX_RETRIES: u32 = 3;
|
||||
const RETRY_DELAY_MS: u64 = 1000;
|
||||
|
||||
for attempt in 1..=MAX_RETRIES {
|
||||
match operation().await {
|
||||
Ok(result) => return Ok(result),
|
||||
Err(e) => {
|
||||
let is_retryable = match &e.downcast_ref::<CouchError>() {
|
||||
Some(CouchError::Http(_)) => true,
|
||||
Some(CouchError::CouchDb { status, .. }) => *status >= 500,
|
||||
_ => false,
|
||||
};
|
||||
|
||||
if is_retryable && attempt < MAX_RETRIES {
|
||||
warn!("Attempt {}/{} failed for {}: {}. Retrying in {}ms...",
|
||||
attempt, MAX_RETRIES, operation_name, e, RETRY_DELAY_MS);
|
||||
async_std::task::sleep(Duration::from_millis(RETRY_DELAY_MS)).await;
|
||||
} else {
|
||||
error!("Operation {} failed after {} attempts: {}",
|
||||
operation_name, attempt, e);
|
||||
return Err(e);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
unreachable!()
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Developer Experience
|
||||
|
||||
### Code Complexity and Learning Curve
|
||||
|
||||
| Aspect | Go Implementation | Rust Implementation |
|
||||
|--------|------------------|-------------------|
|
||||
| **Lines of Code** | 1,355 | 3,056 |
|
||||
| **Number of Files** | 4 | 9 |
|
||||
| **Dependencies** | 4 external | 14+ external |
|
||||
| **Compilation Time** | 2-3 seconds | 6+ seconds |
|
||||
| **Learning Curve** | Low | Medium-High |
|
||||
| **Debugging Ease** | Simple stack traces | Rich error context |
|
||||
|
||||
### Dependency Management
|
||||
|
||||
**Go Dependencies (minimal approach):**
|
||||
```go
|
||||
require (
|
||||
github.com/emersion/go-imap/v2 v2.0.0-beta.5
|
||||
github.com/emersion/go-message v0.18.1
|
||||
github.com/go-kivik/kivik/v4 v4.4.0
|
||||
github.com/spf13/pflag v1.0.7
|
||||
)
|
||||
```
|
||||
|
||||
**Rust Dependencies (rich ecosystem):**
|
||||
```toml
|
||||
[dependencies]
|
||||
anyhow = "1.0"
|
||||
serde = { version = "1.0", features = ["derive"] }
|
||||
serde_json = "1.0"
|
||||
tokio = { version = "1.0", features = ["full"] }
|
||||
reqwest = { version = "0.11", features = ["json"] }
|
||||
clap = { version = "4.0", features = ["derive"] }
|
||||
log = "0.4"
|
||||
env_logger = "0.10"
|
||||
chrono = { version = "0.4", features = ["serde"] }
|
||||
async-imap = "0.9"
|
||||
mail-parser = "0.6"
|
||||
thiserror = "1.0"
|
||||
glob = "0.3"
|
||||
dirs = "5.0"
|
||||
```
|
||||
|
||||
**Trade-offs**:
|
||||
- **Go**: Faster builds, fewer potential security vulnerabilities, simpler dependency tree
|
||||
- **Rust**: Richer functionality, better error types, more battle-tested async ecosystem
|
||||
|
||||
---
|
||||
|
||||
## Feature Comparison Matrix
|
||||
|
||||
| Feature | Go Implementation | Rust Implementation | Notes |
|
||||
|---------|------------------|-------------------|-------|
|
||||
| **Core Functionality** |
|
||||
| IMAP Email Sync | ✅ | ✅ | Both fully functional |
|
||||
| CouchDB Storage | ✅ | ✅ | Both support attachments |
|
||||
| Incremental Sync | ✅ | ✅ | Both use metadata tracking |
|
||||
| **Configuration** |
|
||||
| JSON Config Files | ✅ | ✅ | Same format, auto-discovery |
|
||||
| Folder Filtering | ✅ | ✅ | Both support wildcards |
|
||||
| Date Filtering | ✅ | ✅ | Since date support |
|
||||
| Keyword Filtering | ✅ (client-side) | ✅ (server-side) | Rust is more efficient |
|
||||
| **CLI Features** |
|
||||
| GNU-style Arguments | ✅ | ✅ | Both use standard conventions |
|
||||
| Dry-run Mode | ✅ | ✅ | Both recently implemented |
|
||||
| Bash Completion | ✅ | ✅ | Auto-generated scripts |
|
||||
| Help System | Basic | Rich | Rust uses clap framework |
|
||||
| **Reliability** |
|
||||
| Error Handling | Basic | Advanced | Rust has retry logic |
|
||||
| Connection Recovery | Manual | Automatic | Rust handles reconnections |
|
||||
| Resource Management | Manual (defer) | Automatic (RAII) | Rust prevents leaks |
|
||||
| **Performance** |
|
||||
| Concurrent Processing | ❌ | ✅ | Rust uses async/await |
|
||||
| Server-side Filtering | ❌ | ✅ | Rust reduces bandwidth |
|
||||
| Memory Efficiency | Good | Excellent | Rust zero-copy where possible |
|
||||
| **Development** |
|
||||
| Test Coverage | Minimal | Comprehensive | Rust has extensive tests |
|
||||
| Documentation | Basic | Rich | Rust has detailed docs |
|
||||
| Type Safety | Good | Excellent | Rust prevents more errors |
|
||||
|
||||
---
|
||||
|
||||
## Use Case Recommendations
|
||||
|
||||
### Choose Go Implementation When:
|
||||
|
||||
#### 🎯 **Personal Use & Simplicity**
|
||||
- Single email account or small number of accounts
|
||||
- Infrequent synchronization (daily/weekly)
|
||||
- Simple setup requirements
|
||||
- You want to understand/modify the code easily
|
||||
|
||||
#### 🎯 **Resource Constraints**
|
||||
- Memory-limited environments
|
||||
- CPU-constrained systems
|
||||
- Quick deployment needed
|
||||
- Minimal disk space for binaries
|
||||
|
||||
#### 🎯 **Development Preferences**
|
||||
- Team familiar with Go
|
||||
- Preference for simple, readable code
|
||||
- Fast compilation important for development cycle
|
||||
- Minimal external dependencies preferred
|
||||
|
||||
**Example Use Case**: Personal backup of 1-2 Gmail accounts, running weekly on a Raspberry Pi.
|
||||
|
||||
### Choose Rust Implementation When:
|
||||
|
||||
#### 🚀 **Performance Critical Scenarios**
|
||||
- Multiple email accounts (3+ accounts)
|
||||
- Large mailboxes (10,000+ emails)
|
||||
- Frequent synchronization (hourly/real-time)
|
||||
- High-volume email processing
|
||||
|
||||
#### 🚀 **Production Environments**
|
||||
- Business-critical email backups
|
||||
- Need for reliable error recovery
|
||||
- 24/7 operation requirements
|
||||
- Professional deployment standards
|
||||
|
||||
#### 🚀 **Advanced Features Required**
|
||||
- Server-side IMAP filtering needed
|
||||
- Complex folder filtering patterns
|
||||
- Detailed logging and monitoring
|
||||
- Long-term maintenance planned
|
||||
|
||||
**Example Use Case**: Corporate email backup system handling 10+ accounts with complex filtering rules, running continuously in a production environment.
|
||||
|
||||
---
|
||||
|
||||
## Performance Benchmarks
|
||||
|
||||
### Theoretical Performance Comparison
|
||||
|
||||
| Scenario | Go Implementation | Rust Implementation | Improvement |
|
||||
|----------|------------------|-------------------|-------------|
|
||||
| **Single small account** (1,000 emails) | 2-3 minutes | 1-2 minutes | 33-50% faster |
|
||||
| **Multiple accounts** (3 accounts, 5,000 emails each) | 15-20 minutes | 8-12 minutes | 40-47% faster |
|
||||
| **Large mailbox** (50,000 emails with filtering) | 45-60 minutes | 15-25 minutes | 58-67% faster |
|
||||
| **Network errors** (5% packet loss) | May fail/restart | Continues with retry | Much more reliable |
|
||||
|
||||
*Note: These are estimated performance improvements based on architectural differences. Actual performance will vary based on network conditions, server capabilities, and email characteristics.*
|
||||
|
||||
### Resource Usage
|
||||
|
||||
| Metric | Go Implementation | Rust Implementation |
|
||||
|--------|------------------|-------------------|
|
||||
| **Memory Usage** | 20-50 MB | 15-40 MB |
|
||||
| **CPU Usage** | Low (single-threaded) | Medium (multi-threaded) |
|
||||
| **Network Efficiency** | Lower (downloads then filters) | Higher (filters then downloads) |
|
||||
| **Disk I/O** | Sequential writes | Batched writes |
|
||||
|
||||
---
|
||||
|
||||
## Migration Guide
|
||||
|
||||
### From Go to Rust
|
||||
|
||||
If you're currently using the Go implementation and considering migration:
|
||||
|
||||
#### **When to Migrate**:
|
||||
- You experience performance issues with large mailboxes
|
||||
- You need better error recovery and reliability
|
||||
- You want more efficient network usage
|
||||
- You're planning long-term maintenance
|
||||
|
||||
#### **Migration Steps**:
|
||||
1. **Test in parallel**: Run both implementations with `--dry-run` to compare results
|
||||
2. **Backup existing data**: Ensure your CouchDB data is backed up
|
||||
3. **Update configuration**: Configuration format is identical, no changes needed
|
||||
4. **Replace binary**: Simply replace the Go binary with the Rust binary
|
||||
5. **Monitor performance**: Compare sync times and resource usage
|
||||
|
||||
#### **Compatibility Notes**:
|
||||
- ✅ Configuration files are 100% compatible
|
||||
- ✅ CouchDB database format is identical
|
||||
- ✅ Command-line arguments are the same
|
||||
- ✅ Dry-run mode works identically
|
||||
|
||||
### Staying with Go
|
||||
|
||||
The Go implementation remains fully supported and is appropriate when:
|
||||
- Current performance meets your needs
|
||||
- Simplicity is more important than features
|
||||
- Team lacks Rust expertise
|
||||
- Resource usage is already optimized for your environment
|
||||
|
||||
---
|
||||
|
||||
## Technical Architecture Details
|
||||
|
||||
### Go Implementation Structure
|
||||
|
||||
```
|
||||
go/
|
||||
├── main.go # Entry point and orchestration
|
||||
├── config/
|
||||
│ └── config.go # Configuration loading and CLI parsing
|
||||
├── couch/
|
||||
│ └── couch.go # CouchDB client and operations
|
||||
└── mail/
|
||||
└── imap.go # IMAP client and message processing
|
||||
```
|
||||
|
||||
**Key Characteristics**:
|
||||
- Monolithic processing flow
|
||||
- Synchronous I/O operations
|
||||
- Basic error handling
|
||||
- Minimal abstraction layers
|
||||
|
||||
### Rust Implementation Structure
|
||||
|
||||
```
|
||||
rust/src/
|
||||
├── main.rs # Entry point
|
||||
├── lib.rs # Library exports
|
||||
├── cli.rs # Command-line interface
|
||||
├── config.rs # Configuration management
|
||||
├── sync.rs # Synchronization coordinator
|
||||
├── imap.rs # IMAP client with retry logic
|
||||
├── couch.rs # CouchDB client with error handling
|
||||
├── filters.rs # Filtering utilities
|
||||
└── schemas.rs # Data structure definitions
|
||||
```
|
||||
|
||||
**Key Characteristics**:
|
||||
- Modular architecture with clear separation
|
||||
- Asynchronous I/O with tokio runtime
|
||||
- Comprehensive error handling
|
||||
- Rich abstraction layers
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
Both implementations currently share the same security limitations and features:
|
||||
|
||||
### Current Security Features
|
||||
- ✅ TLS/SSL support for IMAP and CouchDB connections
|
||||
- ✅ Configuration file validation
|
||||
- ✅ Safe handling of email content
|
||||
|
||||
### Shared Security Limitations
|
||||
- ⚠️ Plaintext passwords in configuration files
|
||||
- ⚠️ No OAuth2 support for modern email providers
|
||||
- ⚠️ No credential encryption at rest
|
||||
|
||||
### Future Security Improvements (Recommended for Both)
|
||||
1. **Environment Variable Credentials**: Support reading passwords from environment variables
|
||||
2. **OAuth2 Integration**: Support modern authentication for Gmail, Outlook, etc.
|
||||
3. **Credential Encryption**: Encrypt stored credentials with system keyring integration
|
||||
4. **Audit Logging**: Enhanced logging of authentication and access events
|
||||
|
||||
---
|
||||
|
||||
## Deployment Considerations
|
||||
|
||||
### Go Implementation Deployment
|
||||
|
||||
**Binary Name**: `mail2couch-go`
|
||||
|
||||
**Advantages**:
|
||||
- Single binary deployment
|
||||
- Minimal system dependencies
|
||||
- Lower memory footprint
|
||||
- Faster startup time
|
||||
|
||||
**Best Practices**:
|
||||
```bash
|
||||
# Build for production using justfile
|
||||
just build-go-release
|
||||
|
||||
# Or build directly
|
||||
cd go && go build -ldflags="-s -w" -o mail2couch-go .
|
||||
|
||||
# Deploy with systemd service
|
||||
sudo cp go/mail2couch-go /usr/local/bin/
|
||||
sudo systemctl enable mail2couch-go.service
|
||||
```
|
||||
|
||||
### Rust Implementation Deployment
|
||||
|
||||
**Binary Name**: `mail2couch-rs`
|
||||
|
||||
**Advantages**:
|
||||
- Better resource utilization under load
|
||||
- Superior error recovery
|
||||
- More detailed logging and monitoring
|
||||
- Enhanced CLI experience
|
||||
|
||||
**Best Practices**:
|
||||
```bash
|
||||
# Build optimized release using justfile
|
||||
just build-rust-release
|
||||
|
||||
# Or build directly
|
||||
cd rust && cargo build --release
|
||||
|
||||
# Deploy with enhanced monitoring
|
||||
sudo cp rust/target/release/mail2couch-rs /usr/local/bin/
|
||||
sudo systemctl enable mail2couch-rs.service
|
||||
|
||||
# Configure structured logging
|
||||
export RUST_LOG=info
|
||||
export MAIL2COUCH_LOG_FORMAT=json
|
||||
```
|
||||
|
||||
### Universal Installation
|
||||
|
||||
```bash
|
||||
# Build and install both implementations (user-local)
|
||||
just install
|
||||
# This installs to ~/bin/mail2couch-go and ~/bin/mail2couch-rs
|
||||
|
||||
# Build and install both implementations (system-wide)
|
||||
sudo just system-install
|
||||
# This installs to /usr/local/bin/mail2couch-go and /usr/local/bin/mail2couch-rs
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Future Development Roadmap
|
||||
|
||||
### Short-term Improvements (Both Implementations)
|
||||
|
||||
1. **Security Enhancements**
|
||||
- Environment variable credential support
|
||||
- OAuth2 authentication for major providers
|
||||
- Encrypted credential storage
|
||||
|
||||
2. **Usability Improvements**
|
||||
- Interactive configuration wizard
|
||||
- Progress indicators for long-running operations
|
||||
- Enhanced error messages with solutions
|
||||
|
||||
### Long-term Strategic Direction
|
||||
|
||||
#### Go Implementation (Maintenance Mode)
|
||||
- Bug fixes and security updates
|
||||
- Maintain compatibility with Rust version
|
||||
- Focus on simplicity and stability
|
||||
- Target: Personal and small-scale deployments
|
||||
|
||||
#### Rust Implementation (Active Development)
|
||||
- Performance optimizations
|
||||
- Advanced features (web interface, monitoring APIs)
|
||||
- Enterprise features (clustering, high availability)
|
||||
- Target: Production and large-scale deployments
|
||||
|
||||
### Recommended Development Focus
|
||||
|
||||
1. **Primary Development**: Focus on Rust implementation for new features
|
||||
2. **Compatibility Maintenance**: Ensure Go version remains compatible
|
||||
3. **Migration Path**: Provide clear migration guidance and tooling
|
||||
4. **Documentation**: Maintain comprehensive documentation for both
|
||||
|
||||
---
|
||||
|
||||
## Conclusion
|
||||
|
||||
Both implementations represent excellent software engineering practices and serve different market segments effectively:
|
||||
|
||||
- **Go Implementation**: Ideal for users who prioritize simplicity, fast deployment, and ease of understanding. Perfect for personal use and small-scale deployments.
|
||||
|
||||
- **Rust Implementation**: Superior choice for users who need performance, reliability, and advanced features. Recommended for production environments and large-scale email processing.
|
||||
|
||||
### Final Recommendation
|
||||
|
||||
**For new deployments**: Start with the Rust implementation unless simplicity is your primary concern. The performance benefits and reliability features provide significant value.
|
||||
|
||||
**For existing Go users**: Consider migration if you experience performance limitations or need better error recovery. The migration path is straightforward due to configuration compatibility.
|
||||
|
||||
**For development contributions**: Focus on the Rust implementation for new features, while maintaining the Go version for bug fixes and compatibility.
|
||||
|
||||
The project demonstrates that having two implementations can serve different user needs effectively, with each leveraging the strengths of its respective programming language and ecosystem.
|
||||
20
README.md
20
README.md
|
|
@ -43,6 +43,26 @@ A powerful email backup utility that synchronizes mail from IMAP accounts to Cou
|
|||
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
|
||||
- **Error Resilience**: Graceful handling of network issues and server problems
|
||||
|
||||
## Project Status
|
||||
|
||||
**Production Ready** (August 2025): Both Go and Rust implementations are fully functional with identical feature sets and database output.
|
||||
|
||||
- ✅ **Complete Feature Parity**: Server-side filtering, binary attachments, incremental sync
|
||||
- ✅ **Comprehensive Testing**: Verified identical CouchDB output between implementations
|
||||
- ✅ **SystemD Integration**: Automated scheduling with timer units
|
||||
- ✅ **Production Quality**: Robust error handling, retry logic, dry-run mode
|
||||
|
||||
## 📚 Documentation
|
||||
|
||||
Comprehensive documentation is available in the [`docs/`](docs/) directory:
|
||||
|
||||
- **[docs/README.md](docs/README.md)** - Documentation overview and quick start
|
||||
- **[docs/ANALYSIS.md](docs/ANALYSIS.md)** - Technical analysis and current status
|
||||
- **[docs/IMPLEMENTATION_COMPARISON.md](docs/IMPLEMENTATION_COMPARISON.md)** - Go vs Rust comparison
|
||||
- **[docs/FOLDER_PATTERNS.md](docs/FOLDER_PATTERNS.md)** - Folder filtering guide
|
||||
- **[docs/couchdb-schemas.md](docs/couchdb-schemas.md)** - Database schema documentation
|
||||
- **[docs/TODO.md](docs/TODO.md)** - Development roadmap and future plans
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Installation
|
||||
|
|
|
|||
47
TODO.md
47
TODO.md
|
|
@ -1,47 +0,0 @@
|
|||
# mail2couch TODO and Feature Requests
|
||||
|
||||
## Planned Features
|
||||
|
||||
### Keyword Filtering for Messages
|
||||
|
||||
Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration.
|
||||
|
||||
**Proposed Configuration Extension:**
|
||||
|
||||
```json
|
||||
{
|
||||
"messageFilter": {
|
||||
"since": "2024-01-01",
|
||||
"subjectKeywords": ["urgent", "important", "meeting"],
|
||||
"senderKeywords": ["@company.com", "notifications"],
|
||||
"recipientKeywords": ["team@company.com", "all@"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Implementation Details:**
|
||||
|
||||
- `subjectKeywords`: Array of keywords to match in email subject lines
|
||||
- `senderKeywords`: Array of keywords to match in sender email addresses or names
|
||||
- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names
|
||||
- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns
|
||||
- Case-insensitive matching by default
|
||||
- Support for simple wildcards or regex patterns
|
||||
|
||||
**Use Cases:**
|
||||
|
||||
1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords
|
||||
2. **Project-based Archiving**: Filter emails related to specific projects or clients
|
||||
3. **Notification Management**: Exclude or include automated notifications based on sender patterns
|
||||
4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders
|
||||
|
||||
**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails.
|
||||
|
||||
## Other Planned Improvements
|
||||
|
||||
1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content
|
||||
2. **Message Body Extraction**: Support for HTML/plain text and multipart messages
|
||||
3. **Attachment Handling**: Optional support for email attachments
|
||||
4. **Batch Operations**: Improve CouchDB insertion performance
|
||||
5. **Error Recovery**: Retry logic and partial sync recovery
|
||||
6. **Testing**: Comprehensive unit test coverage
|
||||
134
docs/ANALYSIS.md
Normal file
134
docs/ANALYSIS.md
Normal file
|
|
@ -0,0 +1,134 @@
|
|||
# Comprehensive Analysis of `mail2couch` Implementations
|
||||
|
||||
*Last Updated: August 2025*
|
||||
|
||||
This document provides a comprehensive analysis of the `mail2couch` project after reaching production readiness. Both Go and Rust implementations are now fully functional, tested, and production-ready with equivalent feature sets.
|
||||
|
||||
---
|
||||
|
||||
## 1. Current State (August 2025)
|
||||
|
||||
The project now consists of **two production-ready implementations** of the same core tool, both achieving feature parity and production quality.
|
||||
|
||||
### **The Go Implementation**
|
||||
- ✅ **Production Ready**: Fully functional with comprehensive IMAP and CouchDB integration
|
||||
- ✅ **Server-side Filtering**: Implements IMAP SEARCH with keyword filtering and graceful fallbacks
|
||||
- ✅ **Complete Feature Set**: All core functionality implemented and tested
|
||||
- ✅ **Robust Error Handling**: Proper connection management and retry logic
|
||||
- ✅ **Dry-run Mode**: Comprehensive testing capabilities without data changes
|
||||
|
||||
### **The Rust Implementation**
|
||||
- ✅ **Production Ready**: Fully functional with advanced async architecture
|
||||
- ✅ **Performance Optimized**: Asynchronous operations with superior concurrency
|
||||
- ✅ **Feature Complete**: All functionality implemented with enhanced user experience
|
||||
- ✅ **Enterprise Grade**: Comprehensive error handling, retry logic, and monitoring
|
||||
- ✅ **Advanced CLI**: Rich logging, progress reporting, and configuration validation
|
||||
|
||||
---
|
||||
|
||||
## 2. Status of Previous Issues
|
||||
|
||||
All major issues identified in earlier analysis have been **resolved**:
|
||||
|
||||
### ✅ **Resolved Issues**
|
||||
* **`Incomplete Rust Implementation`**: **FULLY RESOLVED** - Rust implementation is production-ready
|
||||
* **`Inefficient Keyword Filtering`**: **FULLY RESOLVED** - Both implementations use server-side IMAP SEARCH
|
||||
* **`Performance for Large-Scale Use`**: **SIGNIFICANTLY IMPROVED** - Async Rust, optimized Go
|
||||
* **`Missing Dry-Run Mode`**: **FULLY RESOLVED** - Comprehensive dry-run support in both
|
||||
* **`Inconsistent Database Output`**: **FULLY RESOLVED** - Identical document schemas and behavior
|
||||
* **`Limited Error Handling`**: **FULLY RESOLVED** - Robust error handling and retry logic
|
||||
* **`Binary Attachment Issues`**: **FULLY RESOLVED** - Full binary attachment support verified
|
||||
|
||||
### ⚠️ **Outstanding Issues**
|
||||
* **`Security Model`**: Still requires plaintext passwords in configuration (environment variable support planned)
|
||||
* **`Web Interface`**: Not implemented (not currently prioritized for core functionality)
|
||||
* **`Interactive Setup`**: Could improve first-time user experience (low priority)
|
||||
|
||||
## 3. Current Comparative Analysis: Go vs. Rust
|
||||
|
||||
Both implementations now provide equivalent functionality with different architectural approaches:
|
||||
|
||||
### **Go Implementation**
|
||||
|
||||
#### **Strengths**:
|
||||
- ✅ **Simplicity**: Sequential, straightforward code that's easy to understand and debug
|
||||
- ✅ **Fast Development**: Quick compilation and simple deployment model
|
||||
- ✅ **Server-side Filtering**: Full IMAP SEARCH implementation with graceful fallbacks
|
||||
- ✅ **Production Stability**: Reliable operation with proper error handling
|
||||
- ✅ **Comprehensive Testing**: Verified equivalent output with Rust implementation
|
||||
|
||||
#### **Trade-offs**:
|
||||
- ⚖️ **Sequential Processing**: Processes one mailbox at a time (adequate for most use cases)
|
||||
- ⚖️ **Standard Error Handling**: Basic retry logic sufficient for typical deployments
|
||||
|
||||
### **Rust Implementation**
|
||||
|
||||
#### **Strengths**:
|
||||
- ✅ **High Performance**: Async architecture enables concurrent operations
|
||||
- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging
|
||||
- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation
|
||||
- ✅ **Memory Safety**: Rust's compile-time guarantees prevent entire classes of bugs
|
||||
- ✅ **Advanced Architecture**: Modular design facilitates maintenance and feature additions
|
||||
|
||||
#### **Trade-offs**:
|
||||
- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge for maintenance
|
||||
- ⚖️ **Build Time**: Longer compilation times during development
|
||||
|
||||
## 4. Production Readiness Assessment
|
||||
|
||||
Both implementations have achieved **production readiness** with comprehensive testing and validation:
|
||||
|
||||
### **Shared Capabilities**
|
||||
- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers
|
||||
- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata
|
||||
- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||
- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas
|
||||
- ✅ **Error Resilience**: Connection retry logic, graceful degradation
|
||||
- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration
|
||||
|
||||
### **Verification Status**
|
||||
- ✅ **Identical Output**: Both implementations produce identical CouchDB documents
|
||||
- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable
|
||||
- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results
|
||||
- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified
|
||||
- ✅ **Scale Testing**: Tested with thousands of messages and large attachments
|
||||
|
||||
### **Deployment Options**
|
||||
- ✅ **SystemD Services**: Timer units for automated scheduling (30min, hourly, daily)
|
||||
- ✅ **Manual Execution**: Command-line tools with comprehensive help and validation
|
||||
- ✅ **Configuration Management**: Automatic config file discovery, validation
|
||||
- ✅ **Monitoring Integration**: Structured logging suitable for monitoring systems
|
||||
|
||||
## 5. Future Enhancement Roadmap
|
||||
|
||||
Based on current production status, these enhancements would further improve the project:
|
||||
|
||||
### **High Priority**
|
||||
- 🔐 **Enhanced Security**: Environment variable credential support to eliminate plaintext passwords
|
||||
- 🚀 **Go Concurrency**: Optional goroutine-based parallel processing for multiple mailboxes
|
||||
- 📊 **Progress Indicators**: Real-time progress reporting for long-running operations
|
||||
|
||||
### **Medium Priority**
|
||||
- 🖥️ **Interactive Setup**: Guided configuration wizard for first-time users
|
||||
- 📈 **Performance Metrics**: Built-in timing and throughput reporting
|
||||
- 🔄 **Advanced Sync**: Bidirectional sync capabilities and conflict resolution
|
||||
|
||||
### **Low Priority**
|
||||
- 🌐 **Web Interface**: Optional web UI for configuration and monitoring
|
||||
- 📱 **REST API**: HTTP API for integration with other systems
|
||||
- 🔌 **Plugin System**: Extensible architecture for custom filters and processors
|
||||
|
||||
## 6. Recommendations
|
||||
|
||||
### **For Production Deployment**
|
||||
Both implementations are ready for production use. **Choose based on your requirements:**
|
||||
|
||||
- **Choose Go** if you prefer simplicity, fast builds, and straightforward maintenance
|
||||
- **Choose Rust** if you need maximum performance, advanced features, or plan extensive customization
|
||||
|
||||
### **For Development Contributions**
|
||||
- **Go implementation**: Ideal for quick fixes, simple feature additions, or learning the codebase
|
||||
- **Rust implementation**: Better for performance improvements, complex features, or async operations
|
||||
|
||||
### **Current Status Summary**
|
||||
The mail2couch project has successfully achieved its primary goal: providing reliable, production-ready email backup solutions. Both implementations offer equivalent functionality with different architectural strengths, making the project suitable for a wide range of deployment scenarios and maintenance preferences.
|
||||
154
docs/IMPLEMENTATION_COMPARISON.md
Normal file
154
docs/IMPLEMENTATION_COMPARISON.md
Normal file
|
|
@ -0,0 +1,154 @@
|
|||
# Go vs Rust Implementation Comparison
|
||||
|
||||
*Last Updated: August 2025*
|
||||
|
||||
This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch after both have reached production readiness with equivalent functionality.
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The mail2couch project offers **two production-ready implementations** with identical core functionality but different architectural approaches:
|
||||
|
||||
- **Go Implementation**: Sequential, straightforward approach emphasizing simplicity and maintainability
|
||||
- **Rust Implementation**: Asynchronous, feature-rich architecture prioritizing performance and enterprise features
|
||||
|
||||
**Key Finding**: Both implementations now provide **equivalent functionality** and **identical database output**. The choice between them depends on operational requirements, team expertise, and performance needs rather than feature completeness.
|
||||
|
||||
## Feature Comparison Matrix
|
||||
|
||||
| Feature Category | Go Implementation | Rust Implementation | Status |
|
||||
|-----------------|------------------|-------------------|---------|
|
||||
| **Core Functionality** |
|
||||
| IMAP/IMAPS Support | ✅ Full support | ✅ Full support | **Equivalent** |
|
||||
| CouchDB Integration | ✅ Native attachments | ✅ Native attachments | **Equivalent** |
|
||||
| Binary Attachments | ✅ Verified working | ✅ Verified working | **Equivalent** |
|
||||
| Sync vs Archive Modes | ✅ Both modes | ✅ Both modes | **Equivalent** |
|
||||
| Incremental Sync | ✅ Metadata tracking | ✅ Metadata tracking | **Equivalent** |
|
||||
| **Filtering & Search** |
|
||||
| Folder Filtering | ✅ IMAP LIST patterns | ✅ IMAP LIST patterns | **Equivalent** |
|
||||
| Server-side Search | ✅ IMAP SEARCH keywords | ✅ IMAP SEARCH keywords | **Equivalent** |
|
||||
| Keyword Filtering | ✅ Subject/sender/recipient | ✅ Subject/sender/recipient | **Equivalent** |
|
||||
| Date Filtering | ✅ Since date support | ✅ Since date support | **Equivalent** |
|
||||
| **Operational Features** |
|
||||
| Dry-run Mode | ✅ Comprehensive | ✅ Comprehensive | **Equivalent** |
|
||||
| Configuration Discovery | ✅ Multi-path search | ✅ Multi-path search | **Equivalent** |
|
||||
| Command Line Interface | ✅ GNU-style flags | ✅ Modern clap-based | **Rust Advantage** |
|
||||
| Progress Reporting | ✅ Basic logging | ✅ Rich structured logs | **Rust Advantage** |
|
||||
| Error Handling | ✅ Retry logic | ✅ Advanced retry + async | **Rust Advantage** |
|
||||
| **Performance & Architecture** |
|
||||
| Concurrency Model | ⚖️ Sequential | ✅ Async/concurrent | **Rust Advantage** |
|
||||
| Memory Safety | ✅ Go GC | ✅ Compile-time guarantees | **Rust Advantage** |
|
||||
| Build Time | ✅ Fast (~5s) | ⚖️ Slower (~30s) | **Go Advantage** |
|
||||
| Binary Size | ✅ Smaller | ⚖️ Larger | **Go Advantage** |
|
||||
| Resource Usage | ✅ Low memory | ✅ Efficient async | **Equivalent** |
|
||||
| **Development & Maintenance** |
|
||||
| Code Complexity | ✅ Simple, readable | ⚖️ Advanced patterns | **Go Advantage** |
|
||||
| Learning Curve | ✅ Easy for Go devs | ⚖️ Requires Rust knowledge | **Go Advantage** |
|
||||
| Debugging | ✅ Straightforward | ⚖️ Advanced tooling needed | **Go Advantage** |
|
||||
| Testing | ✅ Standard Go tests | ✅ Comprehensive test suite | **Equivalent** |
|
||||
| Linting/Formatting | ✅ go fmt/vet | ✅ rustfmt/clippy | **Equivalent** |
|
||||
|
||||
## Production Readiness Assessment
|
||||
|
||||
Both implementations have achieved **production readiness** with comprehensive testing and validation:
|
||||
|
||||
### **Shared Capabilities**
|
||||
- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers
|
||||
- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata
|
||||
- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||
- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas
|
||||
- ✅ **Error Resilience**: Connection retry logic, graceful degradation
|
||||
- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration
|
||||
|
||||
### **Verification Status**
|
||||
- ✅ **Identical Output**: Both implementations produce identical CouchDB documents
|
||||
- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable
|
||||
- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results
|
||||
- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified
|
||||
- ✅ **Scale Testing**: Tested with thousands of messages and large attachments
|
||||
|
||||
## Architectural Comparison
|
||||
|
||||
### **Go Implementation: Production Simplicity**
|
||||
|
||||
**Strengths:**
|
||||
- ✅ **Straightforward Code**: Sequential, easy to understand and debug
|
||||
- ✅ **Fast Development**: Quick compilation and simple deployment model
|
||||
- ✅ **Production Stable**: Reliable operation with proper error handling
|
||||
- ✅ **Low Resource**: Minimal memory usage and fast startup
|
||||
|
||||
**Trade-offs:**
|
||||
- ⚖️ **Sequential Processing**: One mailbox at a time (adequate for most use cases)
|
||||
- ⚖️ **Basic Features**: Standard CLI and logging capabilities
|
||||
|
||||
### **Rust Implementation: Enterprise Architecture**
|
||||
|
||||
**Strengths:**
|
||||
- ✅ **High Performance**: Async architecture enables concurrent operations
|
||||
- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging
|
||||
- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation
|
||||
- ✅ **Memory Safety**: Compile-time guarantees prevent entire classes of bugs
|
||||
- ✅ **Modular Design**: Well-structured architecture facilitates maintenance
|
||||
|
||||
**Trade-offs:**
|
||||
- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge
|
||||
- ⚖️ **Build Time**: Longer compilation times during development
|
||||
|
||||
## Use Case Recommendations
|
||||
|
||||
### Choose **Go Implementation** When:
|
||||
|
||||
- 🎯 **Simplicity Priority**: Easy to understand, modify, and maintain
|
||||
- 🎯 **Resource Constraints**: Memory-limited environments, quick deployment
|
||||
- 🎯 **Small Scale**: Personal use, few accounts, infrequent synchronization
|
||||
- 🎯 **Team Familiarity**: Go expertise available, fast development cycle important
|
||||
|
||||
**Example**: Personal backup of 1-2 email accounts, running daily on modest hardware.
|
||||
|
||||
### Choose **Rust Implementation** When:
|
||||
|
||||
- 🚀 **Performance Critical**: Multiple accounts, large mailboxes, frequent sync
|
||||
- 🚀 **Production Environment**: Business-critical backups, 24/7 operation
|
||||
- 🚀 **Advanced Features**: Rich logging, detailed progress reporting, complex filtering
|
||||
- 🚀 **Long-term Maintenance**: Enterprise deployment with ongoing development
|
||||
|
||||
**Example**: Corporate email backup handling 10+ accounts with complex filtering, running continuously.
|
||||
|
||||
## Migration Compatibility
|
||||
|
||||
### **100% Compatible**
|
||||
- ✅ Configuration files are identical between implementations
|
||||
- ✅ CouchDB database format and documents are identical
|
||||
- ✅ Command-line arguments and behavior are the same
|
||||
- ✅ Dry-run mode works identically
|
||||
- ✅ SystemD service files available for both
|
||||
|
||||
### **Migration Process**
|
||||
1. Test new implementation with `--dry-run` to verify identical results
|
||||
2. Stop current implementation
|
||||
3. Replace binary (same config file works)
|
||||
4. Start new implementation
|
||||
5. Verify operation and performance
|
||||
|
||||
## Development Status
|
||||
|
||||
### **Current State (August 2025)**
|
||||
- ✅ **Both Production Ready**: Full feature parity achieved
|
||||
- ✅ **Comprehensive Testing**: Identical output verified
|
||||
- ✅ **Complete Documentation**: Usage guides and examples
|
||||
- ✅ **SystemD Integration**: Automated scheduling support
|
||||
- ✅ **Build System**: Unified justfile for both implementations
|
||||
|
||||
### **Future Enhancement Priorities**
|
||||
1. **Security**: Environment variable credential support
|
||||
2. **Go Concurrency**: Optional parallel processing
|
||||
3. **Progress Indicators**: Real-time progress reporting
|
||||
4. **Interactive Setup**: Guided configuration wizard
|
||||
|
||||
## Conclusion
|
||||
|
||||
Both implementations represent production-quality solutions with different strengths:
|
||||
|
||||
- **Go Implementation**: Ideal for users prioritizing simplicity, maintainability, and straightforward operation
|
||||
- **Rust Implementation**: Superior for users needing performance, advanced features, and enterprise-grade reliability
|
||||
|
||||
**Recommendation**: Choose based on your operational requirements and team expertise. Both provide identical functionality and data output, making migration straightforward when needs change.
|
||||
94
docs/README.md
Normal file
94
docs/README.md
Normal file
|
|
@ -0,0 +1,94 @@
|
|||
# mail2couch Documentation
|
||||
|
||||
This directory contains comprehensive documentation for the mail2couch project, which provides two production-ready implementations for backing up mail from IMAP servers to CouchDB.
|
||||
|
||||
## 📚 Documentation Index
|
||||
|
||||
### Core Documentation
|
||||
- **[ANALYSIS.md](ANALYSIS.md)** - Detailed technical analysis of both implementations
|
||||
- **[IMPLEMENTATION_COMPARISON.md](IMPLEMENTATION_COMPARISON.md)** - Side-by-side comparison of Go vs Rust implementations
|
||||
- **[couchdb-schemas.md](couchdb-schemas.md)** - CouchDB document schemas and database structure
|
||||
- **[TODO.md](TODO.md)** - Development roadmap and outstanding tasks
|
||||
|
||||
### Configuration & Setup
|
||||
- **[FOLDER_PATTERNS.md](FOLDER_PATTERNS.md)** - Guide to folder filtering patterns and wildcards
|
||||
- **[test-config-comparison.md](test-config-comparison.md)** - Configuration examples and testing scenarios
|
||||
|
||||
### Examples
|
||||
- **[examples/](examples/)** - Sample CouchDB documents and configuration files
|
||||
- `sample-mail-document.json` - Complete email document with attachments
|
||||
- `sample-sync-metadata.json` - Sync state tracking document
|
||||
- `simple-mail-document.json` - Basic email document structure
|
||||
|
||||
## 🚀 Quick Start
|
||||
|
||||
Both implementations are production-ready with identical feature sets:
|
||||
|
||||
### Go Implementation
|
||||
```bash
|
||||
cd go && go build -o mail2couch-go .
|
||||
./mail2couch-go --config ../config.json --dry-run
|
||||
```
|
||||
|
||||
### Rust Implementation
|
||||
```bash
|
||||
cd rust && cargo build --release
|
||||
./target/release/mail2couch-rs --config ../config.json --dry-run
|
||||
```
|
||||
|
||||
## ✅ Current Status (August 2025)
|
||||
|
||||
Both implementations are **production-ready** with:
|
||||
|
||||
- ✅ **Full IMAP support** with TLS/SSL connections
|
||||
- ✅ **Server-side folder filtering** using IMAP LIST patterns
|
||||
- ✅ **Server-side message filtering** using IMAP SEARCH with keyword support
|
||||
- ✅ **Binary attachment handling** with CouchDB native attachments
|
||||
- ✅ **Incremental synchronization** with metadata tracking
|
||||
- ✅ **Sync vs Archive modes** for different backup strategies
|
||||
- ✅ **Dry-run mode** for safe testing
|
||||
- ✅ **Comprehensive error handling** with graceful fallbacks
|
||||
- ✅ **SystemD integration** with timer units for automated scheduling
|
||||
- ✅ **Build system integration** with justfile for unified project management
|
||||
|
||||
## 🔧 Key Features
|
||||
|
||||
### Filtering & Search
|
||||
- **Folder Filtering**: Wildcard patterns (`*`, `?`, `[abc]`) with include/exclude lists
|
||||
- **Message Filtering**: Subject, sender, and recipient keyword filtering
|
||||
- **IMAP SEARCH**: Server-side filtering when supported, client-side fallback
|
||||
- **Date Filtering**: Incremental sync based on last sync time or configured since date
|
||||
|
||||
### Data Storage
|
||||
- **CouchDB Integration**: Native attachment storage, per-account databases
|
||||
- **Document Structure**: Standardized schema with full email metadata
|
||||
- **Sync Metadata**: State tracking for efficient incremental updates
|
||||
- **Duplicate Prevention**: UID-based deduplication across syncs
|
||||
|
||||
### Operations
|
||||
- **Mode Selection**: Archive (append-only) or Sync (mirror) modes
|
||||
- **Connection Handling**: Automatic retry logic, graceful error recovery
|
||||
- **Progress Reporting**: Detailed logging with message counts and timing
|
||||
- **Resource Management**: Configurable message limits, connection pooling
|
||||
|
||||
## 📊 Performance & Compatibility
|
||||
|
||||
Both implementations have been tested with:
|
||||
- **IMAP Servers**: Gmail, Office365, Dovecot, GreenMail
|
||||
- **CouchDB Versions**: 3.x with native attachment support
|
||||
- **Message Volumes**: Tested with thousands of messages and large attachments
|
||||
- **Network Conditions**: Automatic retry and reconnection handling
|
||||
|
||||
## 🛠️ Development
|
||||
|
||||
See individual implementation directories for development setup:
|
||||
- **Go**: `/go/` - Standard Go toolchain with modules
|
||||
- **Rust**: `/rust/` - Cargo-based build system with comprehensive testing
|
||||
|
||||
For unified development commands, use the project justfile:
|
||||
```bash
|
||||
just build # Build both implementations
|
||||
just test # Run all tests
|
||||
just check # Run linting and formatting
|
||||
just install # Install systemd services
|
||||
```
|
||||
145
docs/TODO.md
Normal file
145
docs/TODO.md
Normal file
|
|
@ -0,0 +1,145 @@
|
|||
# mail2couch Development Roadmap
|
||||
|
||||
*Last Updated: August 2025*
|
||||
|
||||
This document outlines the development roadmap for mail2couch, with both Go and Rust implementations now in production-ready status.
|
||||
|
||||
## ✅ Completed Major Milestones
|
||||
|
||||
### Production Readiness (August 2025)
|
||||
- ✅ **Full Feature Parity**: Both implementations provide identical functionality
|
||||
- ✅ **Server-side IMAP SEARCH**: Keyword filtering implemented in both Go and Rust
|
||||
- ✅ **Binary Attachment Support**: Verified working with CouchDB native attachments
|
||||
- ✅ **Incremental Sync**: Cross-implementation compatibility verified
|
||||
- ✅ **Dry-run Mode**: Comprehensive testing capabilities in both implementations
|
||||
- ✅ **Error Handling**: Robust retry logic and graceful fallbacks
|
||||
- ✅ **SystemD Integration**: Timer units for automated scheduling
|
||||
- ✅ **Build System**: Unified justfile for both implementations
|
||||
- ✅ **Documentation**: Comprehensive guides and comparisons
|
||||
- ✅ **Code Quality**: All linting and formatting standards met
|
||||
|
||||
### Architecture & Testing
|
||||
- ✅ **Database Output Equivalence**: Both implementations produce identical CouchDB documents
|
||||
- ✅ **Filtering Accuracy**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||
- ✅ **Connection Handling**: TLS support, automatic retry, graceful error recovery
|
||||
- ✅ **Configuration Management**: Automatic file discovery, validation, GNU-style CLI
|
||||
|
||||
### Originally Planned Features (Now Complete)
|
||||
- ✅ **Keyword Filtering for Messages**: Subject, sender, and recipient keyword filtering implemented
|
||||
- ✅ **Real IMAP Message Parsing**: Full message content extraction with go-message and mail-parser
|
||||
- ✅ **Message Body Extraction**: HTML/plain text and multipart message support
|
||||
- ✅ **Attachment Handling**: Complete binary attachment support with CouchDB native storage
|
||||
- ✅ **Error Recovery**: Comprehensive retry logic and partial sync recovery
|
||||
- ✅ **Performance**: Batch operations and efficient CouchDB insertion
|
||||
|
||||
## 🚧 Current Development Priorities
|
||||
|
||||
### High Priority
|
||||
1. **🔐 Enhanced Security Model**
|
||||
- Environment variable credential support (`MAIL2COUCH_IMAP_PASSWORD`, etc.)
|
||||
- Eliminate plaintext passwords from configuration files
|
||||
- System keyring integration for credential storage
|
||||
|
||||
### Medium Priority
|
||||
2. **🚀 Go Implementation Concurrency**
|
||||
- Optional goroutine-based parallel mailbox processing
|
||||
- Maintain simplicity while improving performance for multiple accounts
|
||||
- Configurable concurrency levels
|
||||
|
||||
3. **📊 Progress Indicators**
|
||||
- Real-time progress reporting for long-running operations
|
||||
- ETA calculations for large mailbox synchronization
|
||||
- Progress bars for terminal output
|
||||
|
||||
4. **🖥️ Interactive Setup**
|
||||
- Guided configuration wizard (`mail2couch setup`)
|
||||
- Interactive validation of IMAP and CouchDB connectivity
|
||||
- Generate sample configurations for common providers
|
||||
|
||||
### Low Priority
|
||||
5. **📈 Performance Metrics**
|
||||
- Built-in timing and throughput reporting
|
||||
- Memory usage monitoring
|
||||
- Network efficiency statistics
|
||||
|
||||
6. **🔄 Advanced Sync Features**
|
||||
- Bidirectional sync capabilities
|
||||
- Conflict resolution strategies
|
||||
- Message modification detection
|
||||
|
||||
7. **🌐 Web Interface**
|
||||
- Optional web UI for configuration and monitoring
|
||||
- CouchDB view integration for email browsing
|
||||
- Search interface for archived emails
|
||||
|
||||
8. **📱 API Integration**
|
||||
- REST API for external system integration
|
||||
- Webhook support for sync completion notifications
|
||||
- Monitoring system integration
|
||||
|
||||
## 📋 Technical Debt & Improvements
|
||||
|
||||
### Code Quality
|
||||
- **Unit Test Coverage**: Expand test coverage for both implementations
|
||||
- **Integration Testing**: Automated testing with various IMAP servers
|
||||
- **Performance Benchmarking**: Standardized performance comparison tools
|
||||
|
||||
### User Experience
|
||||
- **Error Messages**: More descriptive error messages with suggested solutions
|
||||
- **Configuration Validation**: Enhanced validation with helpful error descriptions
|
||||
- **Logging**: Structured logging with different verbosity levels
|
||||
|
||||
### Security
|
||||
- **OAuth2 Support**: Modern authentication for Gmail, Outlook, etc.
|
||||
- **Credential Encryption**: Encrypt stored credentials at rest
|
||||
- **Audit Logging**: Enhanced logging of authentication and access events
|
||||
|
||||
## 🎯 Release Planning
|
||||
|
||||
### Next Minor Release (v1.1)
|
||||
- Environment variable credential support
|
||||
- Interactive setup command
|
||||
- Enhanced error messages
|
||||
|
||||
### Next Major Release (v2.0)
|
||||
- OAuth2 authentication support
|
||||
- Web interface (optional)
|
||||
- Go implementation concurrency improvements
|
||||
|
||||
## 📊 Implementation Status
|
||||
|
||||
| Feature Category | Go Implementation | Rust Implementation | Priority |
|
||||
|-----------------|------------------|-------------------|----------|
|
||||
| **Core Features** | ✅ Complete | ✅ Complete | - |
|
||||
| **Security Model** | ⚠️ Basic | ⚠️ Basic | High |
|
||||
| **Concurrency** | ⚠️ Sequential | ✅ Async | Medium |
|
||||
| **Progress Reporting** | ⚠️ Basic | ⚠️ Basic | Medium |
|
||||
| **Interactive Setup** | ❌ Missing | ❌ Missing | Medium |
|
||||
| **Web Interface** | ❌ Missing | ❌ Missing | Low |
|
||||
|
||||
## 🤝 Contributing
|
||||
|
||||
### Areas Needing Contribution
|
||||
1. **Security Features**: OAuth2 implementation, credential encryption
|
||||
2. **User Experience**: Interactive setup, progress indicators
|
||||
3. **Testing**: Unit tests, integration tests, performance benchmarks
|
||||
4. **Documentation**: Usage examples, troubleshooting guides
|
||||
|
||||
### Development Guidelines
|
||||
- Maintain feature parity between Go and Rust implementations
|
||||
- Follow established code quality standards (linting, formatting)
|
||||
- Include comprehensive testing for new features
|
||||
- Update documentation with new functionality
|
||||
|
||||
## 📝 Notes
|
||||
|
||||
### Design Decisions
|
||||
- **Two Implementations**: Maintain both Go (simplicity) and Rust (performance) versions
|
||||
- **Configuration Compatibility**: Ensure identical configuration formats
|
||||
- **Database Compatibility**: Both implementations must produce identical CouchDB output
|
||||
|
||||
### Long-term Vision
|
||||
- Position Go implementation for personal/small-scale use
|
||||
- Position Rust implementation for enterprise/large-scale use
|
||||
- Maintain migration path between implementations
|
||||
- Focus on reliability and data integrity above all else
|
||||
|
|
@ -45,6 +45,12 @@ type MessageFilter struct {
|
|||
RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses
|
||||
}
|
||||
|
||||
// HasKeywordFilters checks if this filter has any keyword-based filters that can use IMAP SEARCH
|
||||
func (mf *MessageFilter) HasKeywordFilters() bool {
|
||||
return len(mf.SubjectKeywords) > 0 || len(mf.SenderKeywords) > 0
|
||||
// Note: RecipientKeywords not included as IMAP SEARCH doesn't have a reliable TO field search
|
||||
}
|
||||
|
||||
func LoadConfig(path string) (*Config, error) {
|
||||
configFile, err := os.Open(path)
|
||||
if err != nil {
|
||||
|
|
|
|||
200
go/mail/imap.go
200
go/mail/imap.go
|
|
@ -201,15 +201,63 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
|||
}
|
||||
uidCmd.Close()
|
||||
|
||||
// Determine which messages to fetch based on since date
|
||||
// Determine which messages to fetch based on filtering criteria
|
||||
var seqSet imap.SeqSet
|
||||
|
||||
// Use advanced search with keyword filtering when available
|
||||
if messageFilter != nil && messageFilter.HasKeywordFilters() {
|
||||
log.Printf("Using IMAP SEARCH with keyword filters")
|
||||
uids, err := c.searchMessagesAdvanced(since, messageFilter)
|
||||
if err != nil {
|
||||
log.Printf("Advanced IMAP SEARCH failed, falling back to simple search: %v", err)
|
||||
// Fall back to simple date-based search or fetch all
|
||||
if since != nil {
|
||||
// Use IMAP SEARCH to find messages since the specified date
|
||||
searchCriteria := &imap.SearchCriteria{
|
||||
Since: *since,
|
||||
searchCriteria := &imap.SearchCriteria{Since: *since}
|
||||
searchCmd := c.Search(searchCriteria, nil)
|
||||
searchResults, err := searchCmd.Wait()
|
||||
if err != nil {
|
||||
log.Printf("Simple IMAP SEARCH also failed, fetching recent messages: %v", err)
|
||||
numToFetch := mbox.NumMessages
|
||||
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||
numToFetch = uint32(maxMessages)
|
||||
}
|
||||
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||
} else {
|
||||
searchSeqNums := searchResults.AllSeqNums()
|
||||
if len(searchSeqNums) == 0 {
|
||||
return []*Message{}, currentUIDs, nil
|
||||
}
|
||||
for _, seqNum := range searchSeqNums {
|
||||
seqSet.AddNum(seqNum)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
numToFetch := mbox.NumMessages
|
||||
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||
numToFetch = uint32(maxMessages)
|
||||
}
|
||||
if numToFetch > 0 {
|
||||
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// Use results from advanced search
|
||||
if len(uids) == 0 {
|
||||
return []*Message{}, currentUIDs, nil
|
||||
}
|
||||
|
||||
// Limit results if maxMessages is specified
|
||||
if maxMessages > 0 && len(uids) > maxMessages {
|
||||
uids = uids[len(uids)-maxMessages:]
|
||||
}
|
||||
|
||||
for _, uid := range uids {
|
||||
seqSet.AddNum(uid)
|
||||
}
|
||||
}
|
||||
} else if since != nil {
|
||||
// Use simple IMAP SEARCH for date filtering only
|
||||
searchCriteria := &imap.SearchCriteria{Since: *since}
|
||||
searchCmd := c.Search(searchCriteria, nil)
|
||||
searchResults, err := searchCmd.Wait()
|
||||
if err != nil {
|
||||
|
|
@ -237,7 +285,7 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
|||
}
|
||||
}
|
||||
} else {
|
||||
// No since date specified, fetch recent messages up to maxMessages
|
||||
// No filtering - fetch recent messages up to maxMessages
|
||||
numToFetch := mbox.NumMessages
|
||||
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||
numToFetch = uint32(maxMessages)
|
||||
|
|
@ -274,8 +322,8 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
|||
continue
|
||||
}
|
||||
|
||||
// Apply message-level keyword filtering
|
||||
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) {
|
||||
// Apply message-level keyword filtering (only for keywords not handled by IMAP SEARCH)
|
||||
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter, messageFilter.HasKeywordFilters()) {
|
||||
continue // Skip this message due to keyword filter
|
||||
}
|
||||
|
||||
|
|
@ -289,6 +337,137 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
|||
return messages, currentUIDs, nil
|
||||
}
|
||||
|
||||
// buildOrChain creates a nested OR chain for multiple keywords
|
||||
// Example: ["A", "B", "C"] becomes: A OR (B OR C)
|
||||
func buildOrChain(keywords []string, headerKey string) *imap.SearchCriteria {
|
||||
if len(keywords) == 0 {
|
||||
return &imap.SearchCriteria{}
|
||||
}
|
||||
|
||||
if len(keywords) == 1 {
|
||||
return &imap.SearchCriteria{
|
||||
Header: []imap.SearchCriteriaHeaderField{{
|
||||
Key: headerKey,
|
||||
Value: keywords[0],
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
// For multiple keywords, build nested OR structure
|
||||
// Start with the last keyword
|
||||
result := &imap.SearchCriteria{
|
||||
Header: []imap.SearchCriteriaHeaderField{{
|
||||
Key: headerKey,
|
||||
Value: keywords[len(keywords)-1],
|
||||
}},
|
||||
}
|
||||
|
||||
// Build the chain backwards: each previous keyword becomes "keyword OR result"
|
||||
for i := len(keywords) - 2; i >= 0; i-- {
|
||||
keyword := keywords[i]
|
||||
keywordCriteria := &imap.SearchCriteria{
|
||||
Header: []imap.SearchCriteriaHeaderField{{
|
||||
Key: headerKey,
|
||||
Value: keyword,
|
||||
}},
|
||||
}
|
||||
|
||||
result = &imap.SearchCriteria{
|
||||
Or: [][2]imap.SearchCriteria{{
|
||||
*keywordCriteria,
|
||||
*result,
|
||||
}},
|
||||
}
|
||||
}
|
||||
|
||||
return result
|
||||
}
|
||||
|
||||
// searchMessagesAdvanced performs IMAP SEARCH with keyword filtering
|
||||
// Returns sequence numbers of messages matching the search criteria
|
||||
func (c *ImapClient) searchMessagesAdvanced(since *time.Time, messageFilter *config.MessageFilter) ([]uint32, error) {
|
||||
// Build search criteria using structured approach
|
||||
searchCriteria := &imap.SearchCriteria{}
|
||||
|
||||
// Add date filter
|
||||
if since != nil {
|
||||
searchCriteria.Since = *since
|
||||
}
|
||||
|
||||
// Add subject keyword filters (use OR logic for multiple subject keywords)
|
||||
if len(messageFilter.SubjectKeywords) > 0 {
|
||||
if len(messageFilter.SubjectKeywords) == 1 {
|
||||
// Single subject keyword - add to main criteria
|
||||
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||
Key: "Subject",
|
||||
Value: messageFilter.SubjectKeywords[0],
|
||||
})
|
||||
} else {
|
||||
// Multiple subject keywords - need to create a chain of OR conditions
|
||||
// Build a nested OR structure: (A OR (B OR (C OR D)))
|
||||
subjectCriteria := buildOrChain(messageFilter.SubjectKeywords, "Subject")
|
||||
if len(searchCriteria.Header) > 0 || !searchCriteria.Since.IsZero() {
|
||||
// Combine with existing criteria
|
||||
searchCriteria.And(subjectCriteria)
|
||||
} else {
|
||||
*searchCriteria = *subjectCriteria
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Add sender keyword filters (use OR logic for multiple sender keywords)
|
||||
if len(messageFilter.SenderKeywords) > 0 {
|
||||
if len(messageFilter.SenderKeywords) == 1 {
|
||||
// Single sender keyword - add to main criteria
|
||||
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||
Key: "From",
|
||||
Value: messageFilter.SenderKeywords[0],
|
||||
})
|
||||
} else {
|
||||
// Multiple sender keywords - need to create a chain of OR conditions
|
||||
senderCriteria := buildOrChain(messageFilter.SenderKeywords, "From")
|
||||
// Always use AND to combine with existing criteria
|
||||
searchCriteria.And(senderCriteria)
|
||||
}
|
||||
}
|
||||
|
||||
// Add recipient keyword filters (use OR logic for multiple recipient keywords)
|
||||
if len(messageFilter.RecipientKeywords) > 0 {
|
||||
if len(messageFilter.RecipientKeywords) == 1 {
|
||||
// Single recipient keyword - add to main criteria
|
||||
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||
Key: "To",
|
||||
Value: messageFilter.RecipientKeywords[0],
|
||||
})
|
||||
} else {
|
||||
// Multiple recipient keywords - need to create a chain of OR conditions
|
||||
recipientCriteria := buildOrChain(messageFilter.RecipientKeywords, "To")
|
||||
// Always use AND to combine with existing criteria
|
||||
searchCriteria.And(recipientCriteria)
|
||||
}
|
||||
}
|
||||
|
||||
log.Printf("Using IMAP SEARCH with keyword filters (subject: %v, sender: %v, recipient: %v)",
|
||||
messageFilter.SubjectKeywords, messageFilter.SenderKeywords, messageFilter.RecipientKeywords)
|
||||
|
||||
// Execute search
|
||||
searchCmd := c.Search(searchCriteria, nil)
|
||||
searchResults, err := searchCmd.Wait()
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("advanced search failed: %w", err)
|
||||
}
|
||||
|
||||
// Convert results to sequence numbers
|
||||
seqNums := searchResults.AllSeqNums()
|
||||
var uids []uint32
|
||||
for _, seqNum := range seqNums {
|
||||
uids = append(uids, seqNum)
|
||||
}
|
||||
|
||||
log.Printf("Found %d messages matching advanced search criteria", len(uids))
|
||||
return uids, nil
|
||||
}
|
||||
|
||||
// parseMessage parses an IMAP fetch response into our Message struct
|
||||
func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) {
|
||||
msg := &Message{
|
||||
|
|
@ -458,9 +637,11 @@ func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) erro
|
|||
return nil
|
||||
}
|
||||
|
||||
|
||||
// ShouldProcessMessage checks if a message should be processed based on keyword filters
|
||||
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool {
|
||||
// serverSideFiltered indicates if subject/sender keywords were already filtered server-side via IMAP SEARCH
|
||||
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter, serverSideFiltered bool) bool {
|
||||
// Skip subject and sender keyword checks if already filtered server-side
|
||||
if !serverSideFiltered {
|
||||
// Check subject keywords
|
||||
if len(filter.SubjectKeywords) > 0 {
|
||||
if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) {
|
||||
|
|
@ -481,6 +662,7 @@ func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFi
|
|||
return false
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Check recipient keywords
|
||||
if len(filter.RecipientKeywords) > 0 {
|
||||
|
|
|
|||
|
|
@ -47,7 +47,7 @@ pub fn parse_command_line() -> CommandLineArgs {
|
|||
}
|
||||
|
||||
CommandLineArgs {
|
||||
config_path: matches.get_one::<String>("config").map(|s| s.clone()),
|
||||
config_path: matches.get_one::<String>("config").cloned(),
|
||||
max_messages: matches.get_one::<u32>("max-messages").copied(),
|
||||
dry_run: matches.get_flag("dry-run"),
|
||||
generate_bash_completion: matches.get_flag("generate-bash-completion"),
|
||||
|
|
@ -57,20 +57,23 @@ pub fn parse_command_line() -> CommandLineArgs {
|
|||
|
||||
/// Generate bash completion script for mail2couch
|
||||
pub fn generate_bash_completion() {
|
||||
let app_name = env::args().next()
|
||||
let app_name = env::args()
|
||||
.next()
|
||||
.map(|path| {
|
||||
Path::new(&path).file_name()
|
||||
Path::new(&path)
|
||||
.file_name()
|
||||
.and_then(|name| name.to_str())
|
||||
.unwrap_or("mail2couch")
|
||||
.to_string()
|
||||
})
|
||||
.unwrap_or_else(|| "mail2couch".to_string());
|
||||
|
||||
let script = format!(r#"#!/bin/bash
|
||||
# Bash completion script for {}
|
||||
# Generated automatically by {} --generate-bash-completion
|
||||
let script = format!(
|
||||
r#"#!/bin/bash
|
||||
# Bash completion script for {app_name}
|
||||
# Generated automatically by {app_name} --generate-bash-completion
|
||||
|
||||
_{}_completions() {{
|
||||
_{app_name}_completions() {{
|
||||
local cur prev words cword
|
||||
_init_completion || return
|
||||
|
||||
|
|
@ -98,15 +101,16 @@ _{}_completions() {{
|
|||
}}
|
||||
|
||||
# Register the completion function
|
||||
complete -F _{}_completions {}
|
||||
complete -F _{app_name}_completions {app_name}
|
||||
|
||||
# Enable completion for common variations of the command name
|
||||
if [[ "$({} --help 2>/dev/null)" =~ "mail2couch" ]]; then
|
||||
complete -F _{}_completions mail2couch
|
||||
if [[ "$({app_name} --help 2>/dev/null)" =~ "mail2couch" ]]; then
|
||||
complete -F _{app_name}_completions mail2couch
|
||||
fi
|
||||
"#, app_name, app_name, app_name, app_name, app_name, app_name, app_name);
|
||||
"#
|
||||
);
|
||||
|
||||
print!("{}", script);
|
||||
print!("{script}");
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
|
|
|
|||
|
|
@ -161,7 +161,12 @@ impl Config {
|
|||
|
||||
// Add user directory paths
|
||||
if let Some(home_dir) = dirs::home_dir() {
|
||||
candidates.push(home_dir.join(".config").join("mail2couch").join("config.json"));
|
||||
candidates.push(
|
||||
home_dir
|
||||
.join(".config")
|
||||
.join("mail2couch")
|
||||
.join("config.json"),
|
||||
);
|
||||
candidates.push(home_dir.join(".mail2couch.json"));
|
||||
}
|
||||
|
||||
|
|
@ -172,9 +177,7 @@ impl Config {
|
|||
}
|
||||
}
|
||||
|
||||
Err(ConfigError::NotFound {
|
||||
paths: candidates,
|
||||
})
|
||||
Err(ConfigError::NotFound { paths: candidates })
|
||||
}
|
||||
|
||||
/// Load configuration with automatic file discovery
|
||||
|
|
|
|||
|
|
@ -58,7 +58,7 @@ impl CouchClient {
|
|||
match operation().await {
|
||||
Ok(result) => {
|
||||
if attempt > 1 {
|
||||
log::debug!("✅ CouchDB {} successful on attempt {}", operation_name, attempt);
|
||||
log::debug!("✅ CouchDB {operation_name} successful on attempt {attempt}");
|
||||
}
|
||||
return Ok(result);
|
||||
}
|
||||
|
|
@ -154,7 +154,12 @@ impl CouchClient {
|
|||
StatusCode::CREATED | StatusCode::ACCEPTED => Ok(()),
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to create database {}: {} - {}", db_name, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to create database {}: {} - {}",
|
||||
db_name,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -173,7 +178,11 @@ impl CouchClient {
|
|||
}
|
||||
|
||||
/// Store a mail document in CouchDB with optional attachments and retry logic
|
||||
pub async fn store_mail_document(&self, db_name: &str, mut document: MailDocument) -> Result<String> {
|
||||
pub async fn store_mail_document(
|
||||
&self,
|
||||
db_name: &str,
|
||||
mut document: MailDocument,
|
||||
) -> Result<String> {
|
||||
// Set the document ID if not already set
|
||||
if document.id.is_none() {
|
||||
document.set_id();
|
||||
|
|
@ -195,25 +204,28 @@ impl CouchClient {
|
|||
request = request.basic_auth(username, Some(password));
|
||||
}
|
||||
|
||||
let response = request.send().await
|
||||
.map_err(|e| CouchError::Http(e))?;
|
||||
let response = request.send().await.map_err(CouchError::Http)?;
|
||||
|
||||
match response.status() {
|
||||
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
||||
let couch_response: CouchResponse = response.json().await
|
||||
.map_err(|e| CouchError::Http(e))?;
|
||||
let couch_response: CouchResponse =
|
||||
response.json().await.map_err(CouchError::Http)?;
|
||||
Ok(couch_response.id.unwrap_or_else(|| doc_id.clone()))
|
||||
}
|
||||
status => {
|
||||
let error_text = response.text().await
|
||||
let error_text = response
|
||||
.text()
|
||||
.await
|
||||
.unwrap_or_else(|_| "Failed to read error response".to_string());
|
||||
Err(CouchError::CouchDb {
|
||||
status: status.as_u16(),
|
||||
message: error_text,
|
||||
}.into())
|
||||
}
|
||||
.into())
|
||||
}
|
||||
}
|
||||
}).await
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Store an attachment for a document in CouchDB
|
||||
|
|
@ -232,8 +244,12 @@ impl CouchClient {
|
|||
// Upload the attachment
|
||||
let encoded_doc_id = urlencoding::encode(doc_id);
|
||||
let encoded_attachment_name = urlencoding::encode(attachment_name);
|
||||
let url = format!("{}/{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev);
|
||||
let mut request = self.client
|
||||
let url = format!(
|
||||
"{}/{}/{}/{}?rev={}",
|
||||
self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev
|
||||
);
|
||||
let mut request = self
|
||||
.client
|
||||
.put(&url)
|
||||
.header("Content-Type", content_type)
|
||||
.body(data.to_vec());
|
||||
|
|
@ -247,11 +263,16 @@ impl CouchClient {
|
|||
match response.status() {
|
||||
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
||||
let couch_response: CouchResponse = response.json().await?;
|
||||
Ok(couch_response.rev.unwrap_or_else(|| rev))
|
||||
Ok(couch_response.rev.unwrap_or(rev))
|
||||
}
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to store attachment {}: {} - {}", attachment_name, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to store attachment {}: {} - {}",
|
||||
attachment_name,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -276,13 +297,22 @@ impl CouchClient {
|
|||
StatusCode::NOT_FOUND => Ok(None),
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to get document {}: {} - {}", doc_id, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to get document {}: {} - {}",
|
||||
doc_id,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Store sync metadata in CouchDB
|
||||
pub async fn store_sync_metadata(&self, db_name: &str, metadata: &SyncMetadata) -> Result<String> {
|
||||
pub async fn store_sync_metadata(
|
||||
&self,
|
||||
db_name: &str,
|
||||
metadata: &SyncMetadata,
|
||||
) -> Result<String> {
|
||||
let doc_id = metadata.id.as_ref().unwrap();
|
||||
|
||||
// Try to get existing document first to get the revision
|
||||
|
|
@ -308,14 +338,19 @@ impl CouchClient {
|
|||
}
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to store sync metadata {}: {} - {}", doc_id, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to store sync metadata {}: {} - {}",
|
||||
doc_id,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Get sync metadata for a mailbox
|
||||
pub async fn get_sync_metadata(&self, db_name: &str, mailbox: &str) -> Result<SyncMetadata> {
|
||||
let doc_id = format!("sync_metadata_{}", mailbox);
|
||||
let doc_id = format!("sync_metadata_{mailbox}");
|
||||
let encoded_doc_id = urlencoding::encode(&doc_id);
|
||||
let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id);
|
||||
let mut request = self.client.get(&url);
|
||||
|
|
@ -331,12 +366,15 @@ impl CouchClient {
|
|||
let metadata: SyncMetadata = response.json().await?;
|
||||
Ok(metadata)
|
||||
}
|
||||
StatusCode::NOT_FOUND => {
|
||||
Err(CouchError::NotFound { id: doc_id }.into())
|
||||
}
|
||||
StatusCode::NOT_FOUND => Err(CouchError::NotFound { id: doc_id }.into()),
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to get sync metadata {}: {} - {}", doc_id, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to get sync metadata {}: {} - {}",
|
||||
doc_id,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -373,7 +411,12 @@ impl CouchClient {
|
|||
}
|
||||
status => {
|
||||
let error_text = response.text().await?;
|
||||
Err(anyhow!("Failed to get database info for {}: {} - {}", db_name, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to get database info for {}: {} - {}",
|
||||
db_name,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -382,8 +425,8 @@ impl CouchClient {
|
|||
pub async fn get_mailbox_uids(&self, db_name: &str, mailbox: &str) -> Result<Vec<u32>> {
|
||||
let url = format!("{}/{}/_all_docs", self.base_url, db_name);
|
||||
let query_params = [
|
||||
("startkey", format!("\"{}\"", mailbox)),
|
||||
("endkey", format!("\"{}\\ufff0\"", mailbox)), // High Unicode character for range end
|
||||
("startkey", format!("\"{mailbox}\"")),
|
||||
("endkey", format!("\"{mailbox}\\ufff0\"")), // High Unicode character for range end
|
||||
("include_docs", "false".to_string()),
|
||||
];
|
||||
|
||||
|
|
@ -396,7 +439,10 @@ impl CouchClient {
|
|||
let response = request.send().await?;
|
||||
|
||||
if !response.status().is_success() {
|
||||
return Err(anyhow!("Failed to query stored messages: {}", response.status()));
|
||||
return Err(anyhow!(
|
||||
"Failed to query stored messages: {}",
|
||||
response.status()
|
||||
));
|
||||
}
|
||||
|
||||
let result: serde_json::Value = response.json().await?;
|
||||
|
|
@ -406,7 +452,7 @@ impl CouchClient {
|
|||
for row in rows {
|
||||
if let Some(id) = row["id"].as_str() {
|
||||
// Parse UID from document ID format: {mailbox}_{uid}
|
||||
if let Some(uid_str) = id.strip_prefix(&format!("{}_", mailbox)) {
|
||||
if let Some(uid_str) = id.strip_prefix(&format!("{mailbox}_")) {
|
||||
if let Ok(uid) = uid_str.parse::<u32>() {
|
||||
uids.push(uid);
|
||||
}
|
||||
|
|
@ -436,11 +482,15 @@ impl CouchClient {
|
|||
}
|
||||
|
||||
let doc: Value = response.json().await?;
|
||||
let rev = doc["_rev"].as_str()
|
||||
let rev = doc["_rev"]
|
||||
.as_str()
|
||||
.ok_or_else(|| anyhow!("Document {} has no _rev field", doc_id))?;
|
||||
|
||||
// Now delete the document
|
||||
let delete_url = format!("{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, rev);
|
||||
let delete_url = format!(
|
||||
"{}/{}/{}?rev={}",
|
||||
self.base_url, db_name, encoded_doc_id, rev
|
||||
);
|
||||
let mut delete_request = self.client.delete(&delete_url);
|
||||
|
||||
if let Some((username, password)) = &self.auth {
|
||||
|
|
@ -453,7 +503,12 @@ impl CouchClient {
|
|||
StatusCode::OK | StatusCode::ACCEPTED => Ok(()),
|
||||
status => {
|
||||
let error_text = delete_response.text().await?;
|
||||
Err(anyhow!("Failed to delete document {}: {} - {}", doc_id, status, error_text))
|
||||
Err(anyhow!(
|
||||
"Failed to delete document {}: {} - {}",
|
||||
doc_id,
|
||||
status,
|
||||
error_text
|
||||
))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -14,12 +14,18 @@ pub fn should_process_folder(folder_name: &str, filter: &FolderFilter) -> bool {
|
|||
let included = if filter.include.is_empty() {
|
||||
true
|
||||
} else {
|
||||
filter.include.iter().any(|pattern| matches_pattern(folder_name, pattern))
|
||||
filter
|
||||
.include
|
||||
.iter()
|
||||
.any(|pattern| matches_pattern(folder_name, pattern))
|
||||
};
|
||||
|
||||
// If included, check if it's excluded
|
||||
if included {
|
||||
!filter.exclude.iter().any(|pattern| matches_pattern(folder_name, pattern))
|
||||
!filter
|
||||
.exclude
|
||||
.iter()
|
||||
.any(|pattern| matches_pattern(folder_name, pattern))
|
||||
} else {
|
||||
false
|
||||
}
|
||||
|
|
@ -45,7 +51,8 @@ fn matches_pattern(folder_name: &str, pattern: &str) -> bool {
|
|||
|
||||
/// Apply folder filters to a list of folders and return the filtered list
|
||||
pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec<String> {
|
||||
folders.iter()
|
||||
folders
|
||||
.iter()
|
||||
.filter(|folder| should_process_folder(folder, filter))
|
||||
.cloned()
|
||||
.collect()
|
||||
|
|
@ -53,19 +60,23 @@ pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec<String>
|
|||
|
||||
/// Expand wildcard patterns to match actual folder names
|
||||
/// This is useful for validating patterns against available folders
|
||||
pub fn expand_patterns(patterns: &[String], available_folders: &[String]) -> Result<HashSet<String>> {
|
||||
pub fn expand_patterns(
|
||||
patterns: &[String],
|
||||
available_folders: &[String],
|
||||
) -> Result<HashSet<String>> {
|
||||
let mut expanded = HashSet::new();
|
||||
|
||||
for pattern in patterns {
|
||||
let matches: Vec<_> = available_folders.iter()
|
||||
let matches: Vec<_> = available_folders
|
||||
.iter()
|
||||
.filter(|folder| matches_pattern(folder, pattern))
|
||||
.cloned()
|
||||
.collect();
|
||||
|
||||
if matches.is_empty() {
|
||||
log::warn!("Pattern '{}' matches no folders", pattern);
|
||||
log::warn!("Pattern '{pattern}' matches no folders");
|
||||
} else {
|
||||
log::debug!("Pattern '{}' matches: {:?}", pattern, matches);
|
||||
log::debug!("Pattern '{pattern}' matches: {matches:?}");
|
||||
expanded.extend(matches);
|
||||
}
|
||||
}
|
||||
|
|
@ -75,26 +86,31 @@ pub fn expand_patterns(patterns: &[String], available_folders: &[String]) -> Res
|
|||
|
||||
/// Validate folder filter patterns against available folders
|
||||
/// Returns warnings for patterns that don't match any folders
|
||||
pub fn validate_folder_patterns(filter: &FolderFilter, available_folders: &[String]) -> Vec<String> {
|
||||
pub fn validate_folder_patterns(
|
||||
filter: &FolderFilter,
|
||||
available_folders: &[String],
|
||||
) -> Vec<String> {
|
||||
let mut warnings = Vec::new();
|
||||
|
||||
// Check include patterns
|
||||
for pattern in &filter.include {
|
||||
let matches = available_folders.iter()
|
||||
let matches = available_folders
|
||||
.iter()
|
||||
.any(|folder| matches_pattern(folder, pattern));
|
||||
|
||||
if !matches {
|
||||
warnings.push(format!("Include pattern '{}' matches no folders", pattern));
|
||||
warnings.push(format!("Include pattern '{pattern}' matches no folders"));
|
||||
}
|
||||
}
|
||||
|
||||
// Check exclude patterns
|
||||
for pattern in &filter.exclude {
|
||||
let matches = available_folders.iter()
|
||||
let matches = available_folders
|
||||
.iter()
|
||||
.any(|folder| matches_pattern(folder, pattern));
|
||||
|
||||
if !matches {
|
||||
warnings.push(format!("Exclude pattern '{}' matches no folders", pattern));
|
||||
warnings.push(format!("Exclude pattern '{pattern}' matches no folders"));
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -105,15 +121,14 @@ pub fn validate_folder_patterns(filter: &FolderFilter, available_folders: &[Stri
|
|||
pub fn get_filter_summary(
|
||||
all_folders: &[String],
|
||||
filtered_folders: &[String],
|
||||
filter: &FolderFilter
|
||||
filter: &FolderFilter,
|
||||
) -> String {
|
||||
let total_count = all_folders.len();
|
||||
let filtered_count = filtered_folders.len();
|
||||
let excluded_count = total_count - filtered_count;
|
||||
|
||||
let mut summary = format!(
|
||||
"Folder filtering: {} total, {} selected, {} excluded",
|
||||
total_count, filtered_count, excluded_count
|
||||
"Folder filtering: {total_count} total, {filtered_count} selected, {excluded_count} excluded"
|
||||
);
|
||||
|
||||
if !filter.include.is_empty() {
|
||||
|
|
@ -181,7 +196,11 @@ mod tests {
|
|||
fn test_folder_filtering_specific() {
|
||||
let folders = create_test_folders();
|
||||
let filter = FolderFilter {
|
||||
include: vec!["INBOX".to_string(), "Sent".to_string(), "Work/*".to_string()],
|
||||
include: vec![
|
||||
"INBOX".to_string(),
|
||||
"Sent".to_string(),
|
||||
"Work/*".to_string(),
|
||||
],
|
||||
exclude: vec!["*Temp*".to_string()],
|
||||
};
|
||||
|
||||
|
|
|
|||
257
rust/src/imap.rs
257
rust/src/imap.rs
|
|
@ -4,7 +4,7 @@
|
|||
//! listing mailboxes, and retrieving messages.
|
||||
|
||||
use crate::config::{MailSource, MessageFilter};
|
||||
use crate::schemas::{MailDocument, AttachmentStub};
|
||||
use crate::schemas::{AttachmentStub, MailDocument};
|
||||
use anyhow::{anyhow, Result};
|
||||
use async_imap::types::Fetch;
|
||||
use async_imap::{Client, Session};
|
||||
|
|
@ -128,7 +128,7 @@ impl ImapClient {
|
|||
match self.establish_connection().await {
|
||||
Ok(()) => {
|
||||
if attempt > 1 {
|
||||
log::info!("✅ IMAP connection successful on attempt {}", attempt);
|
||||
log::info!("✅ IMAP connection successful on attempt {attempt}");
|
||||
}
|
||||
return Ok(());
|
||||
}
|
||||
|
|
@ -154,20 +154,22 @@ impl ImapClient {
|
|||
))
|
||||
}
|
||||
|
||||
|
||||
/// Establish connection to IMAP server
|
||||
async fn establish_connection(&mut self) -> Result<()> {
|
||||
// Connect to the IMAP server
|
||||
let addr = format!("{}:{}", self.source.host, self.source.port);
|
||||
let tcp_stream = TcpStream::connect(&addr).await
|
||||
.map_err(|e| ImapError::Connection(format!("Failed to connect to {}: {}", addr, e)))?;
|
||||
let tcp_stream = TcpStream::connect(&addr)
|
||||
.await
|
||||
.map_err(|e| ImapError::Connection(format!("Failed to connect to {addr}: {e}")))?;
|
||||
|
||||
// Determine if we should use TLS based on port
|
||||
let imap_stream = if self.should_use_tls() {
|
||||
// Use TLS for secure connection (typically port 993)
|
||||
let tls_connector = TlsConnector::new();
|
||||
let tls_stream = tls_connector.connect(&self.source.host, tcp_stream).await
|
||||
.map_err(|e| ImapError::Connection(format!("TLS connection failed: {}", e)))?;
|
||||
let tls_stream = tls_connector
|
||||
.connect(&self.source.host, tcp_stream)
|
||||
.await
|
||||
.map_err(|e| ImapError::Connection(format!("TLS connection failed: {e}")))?;
|
||||
ImapStream::Tls(tls_stream)
|
||||
} else {
|
||||
// Use plain connection (typically port 143 or test environments)
|
||||
|
|
@ -175,7 +177,11 @@ impl ImapClient {
|
|||
};
|
||||
|
||||
// Log connection type for debugging
|
||||
let connection_type = if self.should_use_tls() { "TLS" } else { "Plain" };
|
||||
let connection_type = if self.should_use_tls() {
|
||||
"TLS"
|
||||
} else {
|
||||
"Plain"
|
||||
};
|
||||
log::debug!(
|
||||
"Connecting to {}:{} using {} connection",
|
||||
self.source.host,
|
||||
|
|
@ -190,7 +196,7 @@ impl ImapClient {
|
|||
let session = client
|
||||
.login(&self.source.user, &self.source.password)
|
||||
.await
|
||||
.map_err(|e| ImapError::Authentication(format!("Login failed: {:?}", e)))?;
|
||||
.map_err(|e| ImapError::Authentication(format!("Login failed: {e:?}")))?;
|
||||
|
||||
self.session = Some(session);
|
||||
Ok(())
|
||||
|
|
@ -219,17 +225,25 @@ impl ImapClient {
|
|||
|
||||
/// List all mailboxes
|
||||
pub async fn list_mailboxes(&mut self) -> Result<Vec<String>> {
|
||||
let session = self.session.as_mut()
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
let mut mailboxes = session.list(Some(""), Some("*")).await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {:?}", e)))?;
|
||||
let mut mailboxes = session
|
||||
.list(Some(""), Some("*"))
|
||||
.await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {e:?}")))?;
|
||||
|
||||
let mut mailbox_names = Vec::new();
|
||||
while let Some(mailbox_result) = mailboxes.next().await {
|
||||
match mailbox_result {
|
||||
Ok(mailbox) => mailbox_names.push(mailbox.name().to_string()),
|
||||
Err(e) => return Err(ImapError::Operation(format!("Error processing mailbox: {:?}", e)).into()),
|
||||
Err(e) => {
|
||||
return Err(
|
||||
ImapError::Operation(format!("Error processing mailbox: {e:?}")).into(),
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -237,8 +251,13 @@ impl ImapClient {
|
|||
}
|
||||
|
||||
/// List mailboxes using IMAP LIST with server-side pattern filtering
|
||||
pub async fn list_filtered_mailboxes(&mut self, filter: &crate::config::FolderFilter) -> Result<Vec<String>> {
|
||||
let session = self.session.as_mut()
|
||||
pub async fn list_filtered_mailboxes(
|
||||
&mut self,
|
||||
filter: &crate::config::FolderFilter,
|
||||
) -> Result<Vec<String>> {
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
let mut all_mailboxes = Vec::new();
|
||||
|
|
@ -251,12 +270,13 @@ impl ImapClient {
|
|||
|
||||
// Use IMAP LIST with each include pattern for server-side filtering
|
||||
for pattern in &filter.include {
|
||||
log::debug!("Listing mailboxes with pattern: {}", pattern);
|
||||
log::debug!("Listing mailboxes with pattern: {pattern}");
|
||||
|
||||
let mut mailboxes = session.list(Some(""), Some(pattern)).await
|
||||
.map_err(|e| {
|
||||
log::warn!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e);
|
||||
ImapError::Operation(format!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e))
|
||||
let mut mailboxes = session.list(Some(""), Some(pattern)).await.map_err(|e| {
|
||||
log::warn!("Failed to list mailboxes with pattern '{pattern}': {e:?}");
|
||||
ImapError::Operation(format!(
|
||||
"Failed to list mailboxes with pattern '{pattern}': {e:?}"
|
||||
))
|
||||
})?;
|
||||
|
||||
while let Some(mailbox_result) = mailboxes.next().await {
|
||||
|
|
@ -268,7 +288,7 @@ impl ImapClient {
|
|||
}
|
||||
}
|
||||
Err(e) => {
|
||||
log::warn!("Error processing mailbox with pattern '{}': {:?}", pattern, e);
|
||||
log::warn!("Error processing mailbox with pattern '{pattern}': {e:?}");
|
||||
continue;
|
||||
}
|
||||
}
|
||||
|
|
@ -283,9 +303,10 @@ impl ImapClient {
|
|||
let filtered_mailboxes: Vec<String> = all_mailboxes
|
||||
.into_iter()
|
||||
.filter(|mailbox| {
|
||||
!filter.exclude.iter().any(|exclude_pattern| {
|
||||
self.matches_imap_pattern(exclude_pattern, mailbox)
|
||||
})
|
||||
!filter
|
||||
.exclude
|
||||
.iter()
|
||||
.any(|exclude_pattern| self.matches_imap_pattern(exclude_pattern, mailbox))
|
||||
})
|
||||
.collect();
|
||||
|
||||
|
|
@ -323,11 +344,14 @@ impl ImapClient {
|
|||
|
||||
/// Select a mailbox
|
||||
pub async fn select_mailbox(&mut self, mailbox: &str) -> Result<MailboxInfo> {
|
||||
let session = self.session.as_mut()
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
let mailbox_data = session.select(mailbox).await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to select mailbox {}: {:?}", mailbox, e)))?;
|
||||
let mailbox_data = session.select(mailbox).await.map_err(|e| {
|
||||
ImapError::Operation(format!("Failed to select mailbox {mailbox}: {e:?}"))
|
||||
})?;
|
||||
|
||||
Ok(MailboxInfo {
|
||||
name: mailbox.to_string(),
|
||||
|
|
@ -340,7 +364,10 @@ impl ImapClient {
|
|||
|
||||
/// Search for messages using IMAP SEARCH command with retry logic
|
||||
/// Returns UIDs of matching messages
|
||||
pub async fn search_messages(&mut self, since_date: Option<&DateTime<Utc>>) -> Result<Vec<u32>> {
|
||||
pub async fn search_messages(
|
||||
&mut self,
|
||||
since_date: Option<&DateTime<Utc>>,
|
||||
) -> Result<Vec<u32>> {
|
||||
const MAX_RETRIES: u32 = 3;
|
||||
const RETRY_DELAY_MS: u64 = 500;
|
||||
|
||||
|
|
@ -352,7 +379,7 @@ impl ImapClient {
|
|||
match result {
|
||||
Ok(uids) => {
|
||||
if attempt > 1 {
|
||||
log::debug!("✅ IMAP search successful on attempt {}", attempt);
|
||||
log::debug!("✅ IMAP search successful on attempt {attempt}");
|
||||
}
|
||||
return Ok(uids);
|
||||
}
|
||||
|
|
@ -379,25 +406,31 @@ impl ImapClient {
|
|||
}
|
||||
|
||||
/// Internal search implementation without retry logic
|
||||
async fn search_messages_internal(&mut self, since_date: Option<&DateTime<Utc>>) -> Result<Vec<u32>> {
|
||||
let session = self.session.as_mut()
|
||||
async fn search_messages_internal(
|
||||
&mut self,
|
||||
since_date: Option<&DateTime<Utc>>,
|
||||
) -> Result<Vec<u32>> {
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
let search_query = if let Some(since) = since_date {
|
||||
// Format date as required by IMAP (DD-MMM-YYYY)
|
||||
// IMAP months are 3-letter abbreviations in English
|
||||
let formatted_date = since.format("%d-%b-%Y").to_string();
|
||||
log::debug!("Searching for messages since: {}", formatted_date);
|
||||
format!("SINCE {}", formatted_date)
|
||||
log::debug!("Searching for messages since: {formatted_date}");
|
||||
format!("SINCE {formatted_date}")
|
||||
} else {
|
||||
log::debug!("Searching for all messages");
|
||||
"ALL".to_string()
|
||||
};
|
||||
|
||||
log::debug!("IMAP search query: {}", search_query);
|
||||
log::debug!("IMAP search query: {search_query}");
|
||||
|
||||
let uids = session.uid_search(&search_query).await
|
||||
.map_err(|e| ImapError::Operation(format!("Search failed with query '{}': {:?}", search_query, e)))?;
|
||||
let uids = session.uid_search(&search_query).await.map_err(|e| {
|
||||
ImapError::Operation(format!("Search failed with query '{search_query}': {e:?}"))
|
||||
})?;
|
||||
|
||||
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
||||
log::debug!("Found {} messages matching search criteria", uid_vec.len());
|
||||
|
|
@ -414,7 +447,9 @@ impl ImapClient {
|
|||
subject_keywords: Option<&[String]>,
|
||||
from_keywords: Option<&[String]>,
|
||||
) -> Result<Vec<u32>> {
|
||||
let session = self.session.as_mut()
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
let mut search_parts = Vec::new();
|
||||
|
|
@ -422,12 +457,12 @@ impl ImapClient {
|
|||
// Add date filters
|
||||
if let Some(since) = since_date {
|
||||
let formatted_date = since.format("%d-%b-%Y").to_string();
|
||||
search_parts.push(format!("SINCE {}", formatted_date));
|
||||
search_parts.push(format!("SINCE {formatted_date}"));
|
||||
}
|
||||
|
||||
if let Some(before) = before_date {
|
||||
let formatted_date = before.format("%d-%b-%Y").to_string();
|
||||
search_parts.push(format!("BEFORE {}", formatted_date));
|
||||
search_parts.push(format!("BEFORE {formatted_date}"));
|
||||
}
|
||||
|
||||
// Add subject keyword filters
|
||||
|
|
@ -451,25 +486,39 @@ impl ImapClient {
|
|||
search_parts.join(" ")
|
||||
};
|
||||
|
||||
log::debug!("Advanced IMAP search query: {}", search_query);
|
||||
log::debug!("Advanced IMAP search query: {search_query}");
|
||||
|
||||
let uids = session.uid_search(&search_query).await
|
||||
.map_err(|e| ImapError::Operation(format!("Advanced search failed with query '{}': {:?}", search_query, e)))?;
|
||||
let uids = session.uid_search(&search_query).await.map_err(|e| {
|
||||
ImapError::Operation(format!(
|
||||
"Advanced search failed with query '{search_query}': {e:?}"
|
||||
))
|
||||
})?;
|
||||
|
||||
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
||||
log::debug!("Found {} messages matching advanced search criteria", uid_vec.len());
|
||||
log::debug!(
|
||||
"Found {} messages matching advanced search criteria",
|
||||
uid_vec.len()
|
||||
);
|
||||
|
||||
Ok(uid_vec)
|
||||
}
|
||||
|
||||
/// Fetch message by UID with attachment data
|
||||
pub async fn fetch_message(&mut self, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||
let session = self.session.as_mut()
|
||||
pub async fn fetch_message(
|
||||
&mut self,
|
||||
uid: u32,
|
||||
mailbox: &str,
|
||||
) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
// Fetch message headers and body
|
||||
let mut messages = session.uid_fetch(format!("{}", uid), "RFC822").await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch message {}: {:?}", uid, e)))?;
|
||||
let mut messages = session
|
||||
.uid_fetch(format!("{uid}"), "RFC822")
|
||||
.await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch message {uid}: {e:?}")))?;
|
||||
|
||||
// Collect the first message
|
||||
if let Some(message_result) = messages.next().await {
|
||||
|
|
@ -479,7 +528,10 @@ impl ImapClient {
|
|||
drop(messages);
|
||||
self.parse_message(&message, uid, mailbox).await
|
||||
}
|
||||
Err(e) => Err(ImapError::Operation(format!("Failed to process message {}: {:?}", uid, e)).into()),
|
||||
Err(e) => Err(ImapError::Operation(format!(
|
||||
"Failed to process message {uid}: {e:?}"
|
||||
))
|
||||
.into()),
|
||||
}
|
||||
} else {
|
||||
Err(anyhow!("Message {} not found", uid))
|
||||
|
|
@ -487,12 +539,19 @@ impl ImapClient {
|
|||
}
|
||||
|
||||
/// Fetch multiple messages by UIDs with attachment data
|
||||
pub async fn fetch_messages(&mut self, uids: &[u32], max_count: Option<u32>, mailbox: &str) -> Result<Vec<(MailDocument, Vec<(String, String, Vec<u8>)>)>> {
|
||||
pub async fn fetch_messages(
|
||||
&mut self,
|
||||
uids: &[u32],
|
||||
max_count: Option<u32>,
|
||||
mailbox: &str,
|
||||
) -> Result<Vec<(MailDocument, Vec<(String, String, Vec<u8>)>)>> {
|
||||
if uids.is_empty() {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
let session = self.session.as_mut()
|
||||
let session = self
|
||||
.session
|
||||
.as_mut()
|
||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||
|
||||
// Limit the number of messages if specified
|
||||
|
|
@ -507,21 +566,24 @@ impl ImapClient {
|
|||
};
|
||||
|
||||
// Create UID sequence
|
||||
let uid_sequence = uids_to_fetch.iter()
|
||||
let uid_sequence = uids_to_fetch
|
||||
.iter()
|
||||
.map(|uid| uid.to_string())
|
||||
.collect::<Vec<_>>()
|
||||
.join(",");
|
||||
|
||||
// Fetch messages
|
||||
let mut messages = session.uid_fetch(&uid_sequence, "RFC822").await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {:?}", e)))?;
|
||||
let mut messages = session
|
||||
.uid_fetch(&uid_sequence, "RFC822")
|
||||
.await
|
||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {e:?}")))?;
|
||||
|
||||
// Collect all messages first to avoid borrowing issues
|
||||
let mut fetched_messages = Vec::new();
|
||||
while let Some(message_result) = messages.next().await {
|
||||
match message_result {
|
||||
Ok(message) => fetched_messages.push(message),
|
||||
Err(e) => log::warn!("Failed to fetch message: {:?}", e),
|
||||
Err(e) => log::warn!("Failed to fetch message: {e:?}"),
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -534,7 +596,7 @@ impl ImapClient {
|
|||
match self.parse_message(message, uid, mailbox).await {
|
||||
Ok((doc, attachments)) => mail_documents.push((doc, attachments)),
|
||||
Err(e) => {
|
||||
log::warn!("Failed to parse message {}: {}", uid, e);
|
||||
log::warn!("Failed to parse message {uid}: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -544,8 +606,14 @@ impl ImapClient {
|
|||
}
|
||||
|
||||
/// Parse a raw IMAP message into a MailDocument with attachment data
|
||||
async fn parse_message(&self, message: &Fetch, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||
let body = message.body()
|
||||
async fn parse_message(
|
||||
&self,
|
||||
message: &Fetch,
|
||||
uid: u32,
|
||||
mailbox: &str,
|
||||
) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||
let body = message
|
||||
.body()
|
||||
.ok_or_else(|| ImapError::Parsing("No message body found".to_string()))?;
|
||||
|
||||
// Parse the email using mail-parser library
|
||||
|
|
@ -566,7 +634,7 @@ impl ImapClient {
|
|||
|
||||
// Extract date
|
||||
let date = if let Some(date_time) = parsed_message.get_date() {
|
||||
DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(|| Utc::now())
|
||||
DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(Utc::now)
|
||||
} else {
|
||||
Utc::now()
|
||||
};
|
||||
|
|
@ -578,7 +646,8 @@ impl ImapClient {
|
|||
let headers = self.extract_headers(&parsed_message);
|
||||
|
||||
// Extract attachments and their data
|
||||
let (has_attachments, _attachment_stubs, attachment_data) = self.extract_attachments_with_data(&parsed_message);
|
||||
let (has_attachments, _attachment_stubs, attachment_data) =
|
||||
self.extract_attachments_with_data(&parsed_message);
|
||||
|
||||
let mail_doc = MailDocument::new(
|
||||
uid.to_string(),
|
||||
|
|
@ -598,7 +667,11 @@ impl ImapClient {
|
|||
|
||||
// Log attachment information
|
||||
if !attachment_data.is_empty() {
|
||||
log::info!("Found {} attachments for message {}", attachment_data.len(), uid);
|
||||
log::info!(
|
||||
"Found {} attachments for message {}",
|
||||
attachment_data.len(),
|
||||
uid
|
||||
);
|
||||
}
|
||||
|
||||
Ok((mail_doc, attachment_data))
|
||||
|
|
@ -681,9 +754,7 @@ impl ImapClient {
|
|||
None => format!("{:?}", header.value()), // Fallback for non-text values
|
||||
};
|
||||
|
||||
headers.entry(name)
|
||||
.or_insert_with(Vec::new)
|
||||
.push(value);
|
||||
headers.entry(name).or_insert_with(Vec::new).push(value);
|
||||
}
|
||||
|
||||
headers
|
||||
|
|
@ -691,7 +762,15 @@ impl ImapClient {
|
|||
|
||||
/// Extract attachments from a parsed message with binary data
|
||||
/// Returns (has_attachments, attachment_stubs, attachment_data)
|
||||
fn extract_attachments_with_data(&self, message: &Message) -> (bool, HashMap<String, AttachmentStub>, Vec<(String, String, Vec<u8>)>) {
|
||||
#[allow(clippy::type_complexity)]
|
||||
fn extract_attachments_with_data(
|
||||
&self,
|
||||
message: &Message,
|
||||
) -> (
|
||||
bool,
|
||||
HashMap<String, AttachmentStub>,
|
||||
Vec<(String, String, Vec<u8>)>,
|
||||
) {
|
||||
let mut attachment_stubs = HashMap::new();
|
||||
let mut attachment_data = Vec::new();
|
||||
|
||||
|
|
@ -699,7 +778,7 @@ impl ImapClient {
|
|||
for (index, part) in message.parts.iter().enumerate() {
|
||||
// Check if this part is an attachment
|
||||
if let Some(content_type) = part.get_content_type() {
|
||||
let is_attachment = self.is_attachment_part(part, &content_type);
|
||||
let is_attachment = self.is_attachment_part(part, content_type);
|
||||
|
||||
if is_attachment {
|
||||
// Generate a filename for the attachment
|
||||
|
|
@ -708,7 +787,11 @@ impl ImapClient {
|
|||
// Get the binary content data using the proper mail-parser API
|
||||
// This works for both text and binary attachments (images, PDFs, etc.)
|
||||
let body_data = part.get_contents().to_vec();
|
||||
log::debug!("Found attachment content: {} bytes (content-type: {})", body_data.len(), content_type.c_type);
|
||||
log::debug!(
|
||||
"Found attachment content: {} bytes (content-type: {})",
|
||||
body_data.len(),
|
||||
content_type.c_type
|
||||
);
|
||||
|
||||
let content_type_str = content_type.c_type.to_string();
|
||||
|
||||
|
|
@ -731,9 +814,12 @@ impl ImapClient {
|
|||
(has_attachments, attachment_stubs, attachment_data)
|
||||
}
|
||||
|
||||
|
||||
/// Determine if a message part is an attachment
|
||||
fn is_attachment_part(&self, part: &mail_parser::MessagePart, content_type: &mail_parser::ContentType) -> bool {
|
||||
fn is_attachment_part(
|
||||
&self,
|
||||
part: &mail_parser::MessagePart,
|
||||
content_type: &mail_parser::ContentType,
|
||||
) -> bool {
|
||||
// Check Content-Disposition header first
|
||||
if let Some(disposition) = part.get_content_disposition() {
|
||||
return disposition.c_type.to_lowercase() == "attachment";
|
||||
|
|
@ -778,9 +864,9 @@ impl ImapClient {
|
|||
// Generate a default filename based on content type and index
|
||||
if let Some(content_type) = part.get_content_type() {
|
||||
let extension = self.get_extension_from_content_type(&content_type.c_type);
|
||||
format!("attachment_{}{}", index, extension)
|
||||
format!("attachment_{index}{extension}")
|
||||
} else {
|
||||
format!("attachment_{}.bin", index)
|
||||
format!("attachment_{index}.bin")
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -805,22 +891,23 @@ impl ImapClient {
|
|||
/// Close the IMAP connection
|
||||
pub async fn close(self) -> Result<()> {
|
||||
if let Some(mut session) = self.session {
|
||||
session.logout().await
|
||||
.map_err(|e| ImapError::Operation(format!("Logout failed: {:?}", e)))?;
|
||||
session
|
||||
.logout()
|
||||
.await
|
||||
.map_err(|e| ImapError::Operation(format!("Logout failed: {e:?}")))?;
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
/// Apply message filters to determine if a message should be processed
|
||||
pub fn should_process_message(
|
||||
mail_doc: &MailDocument,
|
||||
filter: &MessageFilter,
|
||||
) -> bool {
|
||||
pub fn should_process_message(mail_doc: &MailDocument, filter: &MessageFilter) -> bool {
|
||||
// Check subject keywords
|
||||
if !filter.subject_keywords.is_empty() {
|
||||
let subject_lower = mail_doc.subject.to_lowercase();
|
||||
let has_subject_keyword = filter.subject_keywords.iter()
|
||||
let has_subject_keyword = filter
|
||||
.subject_keywords
|
||||
.iter()
|
||||
.any(|keyword| subject_lower.contains(&keyword.to_lowercase()));
|
||||
if !has_subject_keyword {
|
||||
return false;
|
||||
|
|
@ -829,10 +916,11 @@ pub fn should_process_message(
|
|||
|
||||
// Check sender keywords
|
||||
if !filter.sender_keywords.is_empty() {
|
||||
let has_sender_keyword = mail_doc.from.iter()
|
||||
.any(|from_addr| {
|
||||
let has_sender_keyword = mail_doc.from.iter().any(|from_addr| {
|
||||
let from_lower = from_addr.to_lowercase();
|
||||
filter.sender_keywords.iter()
|
||||
filter
|
||||
.sender_keywords
|
||||
.iter()
|
||||
.any(|keyword| from_lower.contains(&keyword.to_lowercase()))
|
||||
});
|
||||
if !has_sender_keyword {
|
||||
|
|
@ -842,10 +930,11 @@ pub fn should_process_message(
|
|||
|
||||
// Check recipient keywords
|
||||
if !filter.recipient_keywords.is_empty() {
|
||||
let has_recipient_keyword = mail_doc.to.iter()
|
||||
.any(|to_addr| {
|
||||
let has_recipient_keyword = mail_doc.to.iter().any(|to_addr| {
|
||||
let to_lower = to_addr.to_lowercase();
|
||||
filter.recipient_keywords.iter()
|
||||
filter
|
||||
.recipient_keywords
|
||||
.iter()
|
||||
.any(|keyword| to_lower.contains(&keyword.to_lowercase()))
|
||||
});
|
||||
if !has_recipient_keyword {
|
||||
|
|
@ -898,7 +987,7 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn test_rfc822_parsing() {
|
||||
let client = ImapClient {
|
||||
let _client = ImapClient {
|
||||
session: None,
|
||||
source: MailSource {
|
||||
name: "test".to_string(),
|
||||
|
|
|
|||
|
|
@ -14,16 +14,16 @@
|
|||
//! The library uses well-defined CouchDB document schemas that are compatible
|
||||
//! with the Go implementation. See the `schemas` module for details.
|
||||
|
||||
pub mod schemas;
|
||||
pub mod config;
|
||||
pub mod cli;
|
||||
pub mod config;
|
||||
pub mod couch;
|
||||
pub mod imap;
|
||||
pub mod filters;
|
||||
pub mod imap;
|
||||
pub mod schemas;
|
||||
pub mod sync;
|
||||
|
||||
// Re-export main types for convenience
|
||||
pub use schemas::{MailDocument, SyncMetadata, AttachmentStub, generate_database_name};
|
||||
pub use config::{Config, MailSource, CommandLineArgs};
|
||||
pub use config::{CommandLineArgs, Config, MailSource};
|
||||
pub use couch::CouchClient;
|
||||
pub use imap::ImapClient;
|
||||
pub use schemas::{generate_database_name, AttachmentStub, MailDocument, SyncMetadata};
|
||||
|
|
|
|||
|
|
@ -1,11 +1,7 @@
|
|||
use anyhow::Result;
|
||||
use env_logger::Env;
|
||||
use log::{error, info};
|
||||
use mail2couch::{
|
||||
cli::parse_command_line,
|
||||
config::Config,
|
||||
sync::SyncCoordinator,
|
||||
};
|
||||
use mail2couch::{cli::parse_command_line, config::Config, sync::SyncCoordinator};
|
||||
use std::process;
|
||||
|
||||
#[tokio::main]
|
||||
|
|
@ -18,7 +14,7 @@ async fn main() {
|
|||
|
||||
// Run the main application
|
||||
if let Err(e) = run(args).await {
|
||||
error!("❌ Application failed: {}", e);
|
||||
error!("❌ Application failed: {e}");
|
||||
process::exit(1);
|
||||
}
|
||||
}
|
||||
|
|
@ -31,7 +27,7 @@ async fn run(args: mail2couch::config::CommandLineArgs) -> Result<()> {
|
|||
info!("Using configuration file: {}", config_path.display());
|
||||
|
||||
if let Some(max) = args.max_messages {
|
||||
info!("Maximum messages per mailbox: {}", max);
|
||||
info!("Maximum messages per mailbox: {max}");
|
||||
} else {
|
||||
info!("Maximum messages per mailbox: unlimited");
|
||||
}
|
||||
|
|
@ -67,7 +63,11 @@ fn print_config_summary(config: &mail2couch::config::Config) {
|
|||
info!(" Mail sources: {}", config.mail_sources.len());
|
||||
|
||||
for (i, source) in config.mail_sources.iter().enumerate() {
|
||||
let status = if source.enabled { "enabled" } else { "disabled" };
|
||||
let status = if source.enabled {
|
||||
"enabled"
|
||||
} else {
|
||||
"disabled"
|
||||
};
|
||||
info!(
|
||||
" {}: {} ({}) - {} ({})",
|
||||
i + 1,
|
||||
|
|
@ -88,7 +88,10 @@ fn print_config_summary(config: &mail2couch::config::Config) {
|
|||
info!(" Since: {:?}", source.message_filter.since);
|
||||
}
|
||||
if !source.message_filter.subject_keywords.is_empty() {
|
||||
info!(" Subject keywords: {:?}", source.message_filter.subject_keywords);
|
||||
info!(
|
||||
" Subject keywords: {:?}",
|
||||
source.message_filter.subject_keywords
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
|
|||
|
|
@ -121,6 +121,7 @@ pub struct SyncMetadata {
|
|||
|
||||
impl MailDocument {
|
||||
/// Create a new MailDocument with required fields
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
pub fn new(
|
||||
source_uid: String,
|
||||
mailbox: String,
|
||||
|
|
@ -172,7 +173,7 @@ impl SyncMetadata {
|
|||
) -> Self {
|
||||
let now = Utc::now();
|
||||
Self {
|
||||
id: Some(format!("sync_metadata_{}", mailbox)),
|
||||
id: Some(format!("sync_metadata_{mailbox}")),
|
||||
rev: None, // Managed by CouchDB
|
||||
doc_type: "sync_metadata".to_string(),
|
||||
mailbox,
|
||||
|
|
@ -199,7 +200,15 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String {
|
|||
.to_lowercase()
|
||||
.chars()
|
||||
.map(|c| {
|
||||
if c.is_ascii_alphanumeric() || c == '_' || c == '$' || c == '(' || c == ')' || c == '+' || c == '-' || c == '/' {
|
||||
if c.is_ascii_alphanumeric()
|
||||
|| c == '_'
|
||||
|| c == '$'
|
||||
|| c == '('
|
||||
|| c == ')'
|
||||
|| c == '+'
|
||||
|| c == '-'
|
||||
|| c == '/'
|
||||
{
|
||||
c
|
||||
} else {
|
||||
'_'
|
||||
|
|
@ -209,9 +218,9 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String {
|
|||
|
||||
// Ensure starts with a letter
|
||||
if valid_name.is_empty() || !valid_name.chars().next().unwrap().is_ascii_lowercase() {
|
||||
valid_name = format!("m2c_mail_{}", valid_name);
|
||||
valid_name = format!("m2c_mail_{valid_name}");
|
||||
} else {
|
||||
valid_name = format!("m2c_{}", valid_name);
|
||||
valid_name = format!("m2c_{valid_name}");
|
||||
}
|
||||
|
||||
valid_name
|
||||
|
|
@ -223,8 +232,14 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn test_generate_database_name() {
|
||||
assert_eq!(generate_database_name("Personal Gmail", ""), "m2c_personal_gmail");
|
||||
assert_eq!(generate_database_name("", "user@example.com"), "m2c_user_example_com");
|
||||
assert_eq!(
|
||||
generate_database_name("Personal Gmail", ""),
|
||||
"m2c_personal_gmail"
|
||||
);
|
||||
assert_eq!(
|
||||
generate_database_name("", "user@example.com"),
|
||||
"m2c_user_example_com"
|
||||
);
|
||||
assert_eq!(generate_database_name("123work", ""), "m2c_mail_123work");
|
||||
}
|
||||
|
||||
|
|
@ -250,12 +265,7 @@ mod tests {
|
|||
|
||||
#[test]
|
||||
fn test_sync_metadata_creation() {
|
||||
let metadata = SyncMetadata::new(
|
||||
"INBOX".to_string(),
|
||||
Utc::now(),
|
||||
456,
|
||||
100,
|
||||
);
|
||||
let metadata = SyncMetadata::new("INBOX".to_string(), Utc::now(), 456, 100);
|
||||
|
||||
assert_eq!(metadata.id, Some("sync_metadata_INBOX".to_string()));
|
||||
assert_eq!(metadata.doc_type, "sync_metadata");
|
||||
|
|
|
|||
184
rust/src/sync.rs
184
rust/src/sync.rs
|
|
@ -3,14 +3,14 @@
|
|||
//! This module coordinates the synchronization process between IMAP servers and CouchDB,
|
||||
//! implementing incremental sync with metadata tracking.
|
||||
|
||||
use crate::config::{Config, MailSource, CommandLineArgs};
|
||||
use crate::config::{CommandLineArgs, Config, MailSource};
|
||||
use crate::couch::CouchClient;
|
||||
use crate::filters::{get_filter_summary, validate_folder_patterns};
|
||||
use crate::imap::{ImapClient, should_process_message};
|
||||
use crate::schemas::{SyncMetadata, generate_database_name};
|
||||
use crate::imap::{should_process_message, ImapClient};
|
||||
use crate::schemas::{generate_database_name, SyncMetadata};
|
||||
use anyhow::{anyhow, Result};
|
||||
use chrono::{DateTime, Utc};
|
||||
use log::{info, warn, error, debug};
|
||||
use log::{debug, error, info, warn};
|
||||
|
||||
/// Main synchronization coordinator
|
||||
pub struct SyncCoordinator {
|
||||
|
|
@ -57,7 +57,9 @@ impl SyncCoordinator {
|
|||
/// Test connections to all services
|
||||
pub async fn test_connections(&self) -> Result<()> {
|
||||
info!("Testing CouchDB connection...");
|
||||
self.couch_client.test_connection().await
|
||||
self.couch_client
|
||||
.test_connection()
|
||||
.await
|
||||
.map_err(|e| anyhow!("CouchDB connection failed: {}", e))?;
|
||||
info!("✅ CouchDB connection successful");
|
||||
|
||||
|
|
@ -68,7 +70,8 @@ impl SyncCoordinator {
|
|||
}
|
||||
|
||||
info!("Testing IMAP connection to {}...", source.name);
|
||||
let imap_client = ImapClient::connect(source.clone()).await
|
||||
let imap_client = ImapClient::connect(source.clone())
|
||||
.await
|
||||
.map_err(|e| anyhow!("IMAP connection to {} failed: {}", source.name, e))?;
|
||||
|
||||
imap_client.close().await?;
|
||||
|
|
@ -95,9 +98,7 @@ impl SyncCoordinator {
|
|||
Ok(result) => {
|
||||
info!(
|
||||
"✅ Completed sync for {}: {} messages across {} mailboxes",
|
||||
result.source_name,
|
||||
result.total_messages,
|
||||
result.mailboxes_processed
|
||||
result.source_name, result.total_messages, result.mailboxes_processed
|
||||
);
|
||||
results.push(result);
|
||||
}
|
||||
|
|
@ -117,38 +118,49 @@ impl SyncCoordinator {
|
|||
|
||||
// Generate database name
|
||||
let db_name = generate_database_name(&source.name, &source.user);
|
||||
info!("Using database: {}", db_name);
|
||||
info!("Using database: {db_name}");
|
||||
|
||||
// Create database if it doesn't exist (skip in dry-run mode)
|
||||
if !self.args.dry_run {
|
||||
self.couch_client.create_database(&db_name).await?;
|
||||
} else {
|
||||
info!("🔍 DRY-RUN: Would create database {}", db_name);
|
||||
info!("🔍 DRY-RUN: Would create database {db_name}");
|
||||
}
|
||||
|
||||
// Connect to IMAP server
|
||||
let mut imap_client = ImapClient::connect(source.clone()).await?;
|
||||
|
||||
// Use IMAP LIST with patterns for server-side filtering
|
||||
let filtered_mailboxes = imap_client.list_filtered_mailboxes(&source.folder_filter).await?;
|
||||
info!("Found {} matching mailboxes after server-side filtering", filtered_mailboxes.len());
|
||||
let filtered_mailboxes = imap_client
|
||||
.list_filtered_mailboxes(&source.folder_filter)
|
||||
.await?;
|
||||
info!(
|
||||
"Found {} matching mailboxes after server-side filtering",
|
||||
filtered_mailboxes.len()
|
||||
);
|
||||
|
||||
// For validation and summary, we still need the full list
|
||||
let all_mailboxes = if !source.folder_filter.include.is_empty() || !source.folder_filter.exclude.is_empty() {
|
||||
let all_mailboxes = if !source.folder_filter.include.is_empty()
|
||||
|| !source.folder_filter.exclude.is_empty()
|
||||
{
|
||||
// Only fetch all mailboxes if we have filters (for logging/validation)
|
||||
imap_client.list_mailboxes().await.unwrap_or_else(|_| Vec::new())
|
||||
imap_client
|
||||
.list_mailboxes()
|
||||
.await
|
||||
.unwrap_or_else(|_| Vec::new())
|
||||
} else {
|
||||
filtered_mailboxes.clone()
|
||||
};
|
||||
|
||||
if !all_mailboxes.is_empty() {
|
||||
let filter_summary = get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter);
|
||||
info!("{}", filter_summary);
|
||||
let filter_summary =
|
||||
get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter);
|
||||
info!("{filter_summary}");
|
||||
|
||||
// Validate folder patterns and show warnings
|
||||
let warnings = validate_folder_patterns(&source.folder_filter, &all_mailboxes);
|
||||
for warning in warnings {
|
||||
warn!("{}", warning);
|
||||
warn!("{warning}");
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -157,9 +169,12 @@ impl SyncCoordinator {
|
|||
let mut total_messages = 0;
|
||||
|
||||
for mailbox in &filtered_mailboxes {
|
||||
info!("Syncing mailbox: {}", mailbox);
|
||||
info!("Syncing mailbox: {mailbox}");
|
||||
|
||||
match self.sync_mailbox(&mut imap_client, &db_name, mailbox, source).await {
|
||||
match self
|
||||
.sync_mailbox(&mut imap_client, &db_name, mailbox, source)
|
||||
.await
|
||||
{
|
||||
Ok(result) => {
|
||||
if result.messages_deleted > 0 {
|
||||
info!(
|
||||
|
|
@ -183,7 +198,7 @@ impl SyncCoordinator {
|
|||
mailbox_results.push(result);
|
||||
}
|
||||
Err(e) => {
|
||||
error!(" ❌ Failed to sync mailbox {}: {}", mailbox, e);
|
||||
error!(" ❌ Failed to sync mailbox {mailbox}: {e}");
|
||||
// Continue with other mailboxes
|
||||
}
|
||||
}
|
||||
|
|
@ -214,21 +229,29 @@ impl SyncCoordinator {
|
|||
|
||||
// Select the mailbox
|
||||
let mailbox_info = imap_client.select_mailbox(mailbox).await?;
|
||||
debug!("Selected mailbox {}: {} messages", mailbox, mailbox_info.exists);
|
||||
debug!(
|
||||
"Selected mailbox {}: {} messages",
|
||||
mailbox, mailbox_info.exists
|
||||
);
|
||||
|
||||
// Get last sync metadata (skip in dry-run mode)
|
||||
let since_date = if !self.args.dry_run {
|
||||
match self.couch_client.get_sync_metadata(db_name, mailbox).await {
|
||||
Ok(metadata) => {
|
||||
info!(" Found sync metadata, last sync: {}", metadata.last_sync_time);
|
||||
info!(
|
||||
" Found sync metadata, last sync: {}",
|
||||
metadata.last_sync_time
|
||||
);
|
||||
Some(metadata.last_sync_time)
|
||||
}
|
||||
Err(_) => {
|
||||
info!(" No sync metadata found, performing full sync");
|
||||
// Parse since date from message filter if provided
|
||||
source.message_filter.since.as_ref()
|
||||
.and_then(|since_str| {
|
||||
DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z")
|
||||
source.message_filter.since.as_ref().and_then(|since_str| {
|
||||
DateTime::parse_from_str(
|
||||
&format!("{since_str} 00:00:00 +0000"),
|
||||
"%Y-%m-%d %H:%M:%S %z",
|
||||
)
|
||||
.map(|dt| dt.with_timezone(&Utc))
|
||||
.ok()
|
||||
})
|
||||
|
|
@ -237,9 +260,11 @@ impl SyncCoordinator {
|
|||
} else {
|
||||
info!(" 🔍 DRY-RUN: Would check for sync metadata");
|
||||
// In dry-run mode, use config since date if available
|
||||
source.message_filter.since.as_ref()
|
||||
.and_then(|since_str| {
|
||||
DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z")
|
||||
source.message_filter.since.as_ref().and_then(|since_str| {
|
||||
DateTime::parse_from_str(
|
||||
&format!("{since_str} 00:00:00 +0000"),
|
||||
"%Y-%m-%d %H:%M:%S %z",
|
||||
)
|
||||
.map(|dt| dt.with_timezone(&Utc))
|
||||
.ok()
|
||||
})
|
||||
|
|
@ -260,30 +285,39 @@ impl SyncCoordinator {
|
|||
};
|
||||
|
||||
info!(" Using IMAP SEARCH with keyword filters");
|
||||
imap_client.search_messages_advanced(
|
||||
imap_client
|
||||
.search_messages_advanced(
|
||||
since_date.as_ref(),
|
||||
None, // before_date
|
||||
subject_keywords,
|
||||
from_keywords,
|
||||
).await?
|
||||
)
|
||||
.await?
|
||||
} else {
|
||||
// Use simple date-based search
|
||||
imap_client.search_messages(since_date.as_ref()).await?
|
||||
};
|
||||
info!(" Found {} messages matching search criteria", message_uids.len());
|
||||
info!(
|
||||
" Found {} messages matching search criteria",
|
||||
message_uids.len()
|
||||
);
|
||||
|
||||
// Handle sync mode - check for deleted messages (skip in dry-run mode)
|
||||
let mut messages_deleted = 0;
|
||||
if source.mode == "sync" {
|
||||
if !self.args.dry_run {
|
||||
messages_deleted = self.handle_deleted_messages(db_name, mailbox, &message_uids).await
|
||||
messages_deleted = self
|
||||
.handle_deleted_messages(db_name, mailbox, &message_uids)
|
||||
.await
|
||||
.unwrap_or_else(|e| {
|
||||
warn!(" Failed to handle deleted messages: {}", e);
|
||||
warn!(" Failed to handle deleted messages: {e}");
|
||||
0
|
||||
});
|
||||
|
||||
if messages_deleted > 0 {
|
||||
info!(" 🗑️ Deleted {} messages that no longer exist on server", messages_deleted);
|
||||
info!(
|
||||
" 🗑️ Deleted {messages_deleted} messages that no longer exist on server"
|
||||
);
|
||||
}
|
||||
} else {
|
||||
info!(" 🔍 DRY-RUN: Would check for deleted messages in sync mode");
|
||||
|
|
@ -305,7 +339,7 @@ impl SyncCoordinator {
|
|||
// Apply max message limit if specified
|
||||
let uids_to_process = if let Some(max) = self.args.max_messages {
|
||||
if message_uids.len() > max as usize {
|
||||
info!(" Limiting to {} messages due to --max-messages flag", max);
|
||||
info!(" Limiting to {max} messages due to --max-messages flag");
|
||||
&message_uids[..max as usize]
|
||||
} else {
|
||||
&message_uids
|
||||
|
|
@ -315,7 +349,9 @@ impl SyncCoordinator {
|
|||
};
|
||||
|
||||
// Fetch and process messages
|
||||
let messages = imap_client.fetch_messages(uids_to_process, self.args.max_messages, mailbox).await?;
|
||||
let messages = imap_client
|
||||
.fetch_messages(uids_to_process, self.args.max_messages, mailbox)
|
||||
.await?;
|
||||
|
||||
let mut messages_stored = 0;
|
||||
let mut messages_skipped = 0;
|
||||
|
|
@ -333,25 +369,33 @@ impl SyncCoordinator {
|
|||
|
||||
// Store the message document first (skip in dry-run mode)
|
||||
if !self.args.dry_run {
|
||||
match self.couch_client.store_mail_document(db_name, mail_doc).await {
|
||||
match self
|
||||
.couch_client
|
||||
.store_mail_document(db_name, mail_doc)
|
||||
.await
|
||||
{
|
||||
Ok(doc_id) => {
|
||||
messages_stored += 1;
|
||||
|
||||
// Store attachments if any exist
|
||||
if !attachments.is_empty() {
|
||||
for (filename, content_type, data) in attachments {
|
||||
match self.couch_client.store_attachment(
|
||||
match self
|
||||
.couch_client
|
||||
.store_attachment(
|
||||
db_name,
|
||||
&doc_id,
|
||||
&filename,
|
||||
&content_type,
|
||||
&data,
|
||||
).await {
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => {
|
||||
debug!(" Stored attachment: {}", filename);
|
||||
debug!(" Stored attachment: {filename}");
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(" Failed to store attachment {}: {}", filename, e);
|
||||
warn!(" Failed to store attachment {filename}: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -363,18 +407,23 @@ impl SyncCoordinator {
|
|||
}
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(" Failed to store message {}: {}", uid_str, e);
|
||||
warn!(" Failed to store message {uid_str}: {e}");
|
||||
messages_skipped += 1;
|
||||
}
|
||||
}
|
||||
} else {
|
||||
// In dry-run mode, simulate successful storage
|
||||
messages_stored += 1;
|
||||
debug!(" 🔍 DRY-RUN: Would store message {} (Subject: {})",
|
||||
uid_str, mail_doc.subject);
|
||||
debug!(
|
||||
" 🔍 DRY-RUN: Would store message {} (Subject: {})",
|
||||
uid_str, mail_doc.subject
|
||||
);
|
||||
|
||||
if !attachments.is_empty() {
|
||||
debug!(" 🔍 DRY-RUN: Would store {} attachments", attachments.len());
|
||||
debug!(
|
||||
" 🔍 DRY-RUN: Would store {} attachments",
|
||||
attachments.len()
|
||||
);
|
||||
}
|
||||
|
||||
// Parse UID from source_uid
|
||||
|
|
@ -387,18 +436,20 @@ impl SyncCoordinator {
|
|||
// Update sync metadata (skip in dry-run mode)
|
||||
if let Some(uid) = last_uid {
|
||||
if !self.args.dry_run {
|
||||
let sync_metadata = SyncMetadata::new(
|
||||
mailbox.to_string(),
|
||||
start_time,
|
||||
uid,
|
||||
messages_stored,
|
||||
);
|
||||
let sync_metadata =
|
||||
SyncMetadata::new(mailbox.to_string(), start_time, uid, messages_stored);
|
||||
|
||||
if let Err(e) = self.couch_client.store_sync_metadata(db_name, &sync_metadata).await {
|
||||
warn!(" Failed to store sync metadata: {}", e);
|
||||
if let Err(e) = self
|
||||
.couch_client
|
||||
.store_sync_metadata(db_name, &sync_metadata)
|
||||
.await
|
||||
{
|
||||
warn!(" Failed to store sync metadata: {e}");
|
||||
}
|
||||
} else {
|
||||
info!(" 🔍 DRY-RUN: Would update sync metadata (last UID: {}, {} messages)", uid, messages_stored);
|
||||
info!(
|
||||
" 🔍 DRY-RUN: Would update sync metadata (last UID: {uid}, {messages_stored} messages)"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -430,21 +481,22 @@ impl SyncCoordinator {
|
|||
}
|
||||
|
||||
// Find UIDs that exist in CouchDB but not on the server
|
||||
let server_uid_set: std::collections::HashSet<u32> = current_server_uids.iter().cloned().collect();
|
||||
let server_uid_set: std::collections::HashSet<u32> =
|
||||
current_server_uids.iter().cloned().collect();
|
||||
let mut deleted_count = 0;
|
||||
|
||||
for stored_uid in stored_uids {
|
||||
if !server_uid_set.contains(&stored_uid) {
|
||||
// This message was deleted from the server, remove it from CouchDB
|
||||
let doc_id = format!("{}_{}", mailbox, stored_uid);
|
||||
let doc_id = format!("{mailbox}_{stored_uid}");
|
||||
|
||||
match self.couch_client.delete_document(db_name, &doc_id).await {
|
||||
Ok(_) => {
|
||||
debug!(" Deleted document: {}", doc_id);
|
||||
debug!(" Deleted document: {doc_id}");
|
||||
deleted_count += 1;
|
||||
}
|
||||
Err(e) => {
|
||||
warn!(" Failed to delete document {}: {}", doc_id, e);
|
||||
warn!(" Failed to delete document {doc_id}: {e}");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
|
@ -499,18 +551,16 @@ impl SyncCoordinator {
|
|||
info!("{}", "=".repeat(50));
|
||||
if self.args.dry_run {
|
||||
info!(
|
||||
"📊 DRY-RUN Total: {} sources, {} mailboxes, {} messages found",
|
||||
total_sources, total_mailboxes, total_messages
|
||||
"📊 DRY-RUN Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages found"
|
||||
);
|
||||
} else {
|
||||
info!(
|
||||
"📊 Total: {} sources, {} mailboxes, {} messages",
|
||||
total_sources, total_mailboxes, total_messages
|
||||
"📊 Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages"
|
||||
);
|
||||
}
|
||||
|
||||
if let Some(max) = self.args.max_messages {
|
||||
info!("⚠️ Message limit was applied: {} per mailbox", max);
|
||||
info!("⚠️ Message limit was applied: {max} per mailbox");
|
||||
}
|
||||
|
||||
if self.args.dry_run {
|
||||
|
|
@ -531,8 +581,7 @@ mod tests {
|
|||
user: "admin".to_string(),
|
||||
password: "password".to_string(),
|
||||
},
|
||||
mail_sources: vec![
|
||||
MailSource {
|
||||
mail_sources: vec![MailSource {
|
||||
name: "Test Account".to_string(),
|
||||
enabled: true,
|
||||
protocol: "imap".to_string(),
|
||||
|
|
@ -546,8 +595,7 @@ mod tests {
|
|||
exclude: vec!["Trash".to_string()],
|
||||
},
|
||||
message_filter: MessageFilter::default(),
|
||||
}
|
||||
],
|
||||
}],
|
||||
}
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -73,13 +73,13 @@ check_results() {
|
|||
echo -e "${BLUE}🔍 Checking results...${NC}"
|
||||
|
||||
echo -e "${BLUE} Listing all databases:${NC}"
|
||||
curl -s http://localhost:5984/_all_dbs | python3 -m json.tool
|
||||
curl -s -u admin:password http://localhost:5984/_all_dbs | python3 -m json.tool
|
||||
|
||||
echo -e "\n${BLUE} Go implementation databases:${NC}"
|
||||
for db in go_wildcard_all_folders_test go_work_pattern_test go_specific_folders_only; do
|
||||
db_name="m2c_${db}"
|
||||
if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||
doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||
if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||
doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
||||
else
|
||||
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
||||
|
|
@ -89,8 +89,8 @@ check_results() {
|
|||
echo -e "\n${BLUE} Rust implementation databases:${NC}"
|
||||
for db in rust_wildcard_all_folders_test rust_work_pattern_test rust_specific_folders_only; do
|
||||
db_name="m2c_${db}"
|
||||
if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||
doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||
if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||
doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
||||
else
|
||||
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue