Compare commits
3 commits
436276f0ef
...
d3d104ee71
| Author | SHA1 | Date | |
|---|---|---|---|
| d3d104ee71 | |||
| f80f89cdd5 | |||
| 6c387abfbb |
27 changed files with 1398 additions and 1151 deletions
112
ANALYSIS.md
112
ANALYSIS.md
|
|
@ -1,112 +0,0 @@
|
||||||
### Comprehensive Analysis of `mail2couch` Implementations
|
|
||||||
|
|
||||||
This document provides an updated, in-depth analysis of the `mail2couch` project, integrating findings from the original `ANALYSIS.md` with a fresh review of the current Go and Rust codebases. It evaluates the current state, compares the two implementations, and outlines a roadmap for future improvements.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 1. Current State of the Implementations
|
|
||||||
|
|
||||||
The project currently consists of two distinct implementations of the same core tool.
|
|
||||||
|
|
||||||
* **The Go Implementation**: This is a mature, functional, and straightforward command-line tool. It is built on a simple, sequential architecture and effectively synchronizes emails from IMAP servers to CouchDB. It serves as a solid baseline for the project's core functionality.
|
|
||||||
|
|
||||||
* **The Rust Implementation**: Contrary to the description in the original `ANALYSIS.md`, the Rust version is **no longer a non-functional placeholder**. It is now a complete, and in many ways, more advanced alternative to the Go version. It is built on a highly modular, asynchronous architecture, prioritizing performance, robustness, and an expanded feature set.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 2. Analysis of Points from Original `ANALYSIS.md`
|
|
||||||
|
|
||||||
Several key issues and suggestions were raised in the original analysis. Here is their current status:
|
|
||||||
|
|
||||||
* **`Incomplete Rust Implementation`**: **(Addressed)** The Rust implementation is now fully functional and surpasses the Go version in features and robustness.
|
|
||||||
* **`Performance for Large-Scale Use (Concurrency)`**: **(Addressed in Rust)** The Go version remains sequential. The Rust version, however, is fully asynchronous, allowing for concurrent network operations, which directly addresses this performance concern.
|
|
||||||
* **`Inefficient Keyword Filtering`**: **(Addressed in Rust)** The Go version still performs keyword filtering client-side. The Rust version implements server-side filtering using `IMAP SEARCH` with keywords, which is significantly more efficient.
|
|
||||||
* **`Primary Weakness: Security`**: **(Still an Issue)** Both implementations still require plaintext passwords in the configuration file. This remains a primary weakness.
|
|
||||||
* **`Missing Core Feature: Web Interface`**: **(Still an Issue)** This feature has not been implemented in either version.
|
|
||||||
* **`Usability Enhancement: Dry-Run Mode`**: **(✅ Resolved)** Both implementations now include comprehensive `--dry-run/-n` mode functionality that allows safe configuration testing without making any CouchDB changes.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 3. Comparative Analysis: Go vs. Rust
|
|
||||||
|
|
||||||
#### **The Go Version**
|
|
||||||
|
|
||||||
* **Pros**:
|
|
||||||
* **Simplicity**: The code is sequential and easy to follow, making it highly approachable for new contributors.
|
|
||||||
* **Stability**: It provides a solid, functional baseline that effectively accomplishes the core mission of the project.
|
|
||||||
* **Fast Compilation**: Quick compile times make for a fast development cycle.
|
|
||||||
* **Dry-Run Support**: Now includes comprehensive `--dry-run` mode for safe configuration testing.
|
|
||||||
* **Cons**:
|
|
||||||
* **Performance**: The lack of concurrency makes it slow for users with multiple accounts or large mailboxes.
|
|
||||||
* **Inefficiency**: Client-side keyword filtering wastes bandwidth and processing time.
|
|
||||||
* **Basic Error Handling**: The absence of retry logic makes it brittle in the face of transient network errors.
|
|
||||||
|
|
||||||
#### **The Rust Version**
|
|
||||||
|
|
||||||
* **Pros**:
|
|
||||||
* **Performance**: The `async` architecture provides superior performance through concurrency.
|
|
||||||
* **Robustness**: Automatic retry logic for network calls makes it highly resilient to temporary failures.
|
|
||||||
* **Feature-Rich**: Implements more efficient server-side filtering, better folder-matching logic, and a more professional CLI.
|
|
||||||
* **Safety & Maintainability**: The modular design and Rust's compile-time guarantees make the code safer and easier to maintain and extend.
|
|
||||||
* **Comprehensive Dry-Run**: Includes detailed `--dry-run` mode with enhanced simulation logging and summary reporting.
|
|
||||||
* **Cons**:
|
|
||||||
* **Complexity**: The codebase is significantly more complex due to its asynchronous nature, abstract design, and the inherent learning curve of Rust.
|
|
||||||
* **Slower Compilation**: Longer compile times can slow down development.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 4. Recent Implementation Updates
|
|
||||||
|
|
||||||
#### **Dry-Run Mode Implementation (August 2025)**
|
|
||||||
|
|
||||||
Both Go and Rust implementations now include comprehensive `--dry-run` functionality:
|
|
||||||
|
|
||||||
##### **Go Implementation Features:**
|
|
||||||
- **CLI Integration**: Added `--dry-run/-n` flag using pflag with GNU-style options
|
|
||||||
- **Comprehensive Skipping**: All CouchDB write operations bypassed in dry-run mode
|
|
||||||
- **IMAP Preservation**: Maintains full IMAP operations for realistic email discovery
|
|
||||||
- **Detailed Simulation**: Shows what would be done with informative logging
|
|
||||||
- **Enhanced Reporting**: Clear distinction between dry-run and normal mode output
|
|
||||||
- **Bash Completion**: Updated completion script includes new flag
|
|
||||||
|
|
||||||
##### **Rust Implementation Features:**
|
|
||||||
- **CLI Integration**: Added `--dry-run/-n` flag using clap with structured argument parsing
|
|
||||||
- **Advanced Simulation**: Detailed logging of what would be stored including message subjects
|
|
||||||
- **Async-Safe Skipping**: All async CouchDB operations properly bypassed
|
|
||||||
- **Enhanced Summary**: Comprehensive dry-run vs normal mode reporting with emoji indicators
|
|
||||||
- **Test Coverage**: All tests updated to include new dry_run field
|
|
||||||
|
|
||||||
##### **Implementation Benefits:**
|
|
||||||
- **Risk Mitigation**: Users can validate configurations without database changes
|
|
||||||
- **Debugging Aid**: Shows exactly what emails would be processed and stored
|
|
||||||
- **Development Tool**: Enables safe testing of configuration changes
|
|
||||||
- **Documentation**: Demonstrates the full sync process without side effects
|
|
||||||
|
|
||||||
This addresses the critical usability requirement identified in the original analysis and significantly improves the user experience for configuration validation and troubleshooting.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
### 5. Future Improvements and Missing Features
|
|
||||||
|
|
||||||
This roadmap combines suggestions from both analyses, prioritizing the most impactful changes.
|
|
||||||
|
|
||||||
#### **Tier 1: Critical Needs**
|
|
||||||
|
|
||||||
1. **Fix the Security Model (Both)**: This is the most urgent issue.
|
|
||||||
* **Short-Term**: Add support for reading credentials from environment variables (e.g., `M2C_IMAP_PASSWORD`).
|
|
||||||
* **Long-Term**: Implement OAuth2 for modern providers like Gmail and Outlook. This is the industry standard and eliminates the need to store passwords.
|
|
||||||
2. **Implement a Web Interface (Either)**: As noted in the original analysis, this is the key missing feature for making the archived data useful. This would involve creating CouchDB design documents and a simple web server to render the views.
|
|
||||||
3. ~~**Add a `--dry-run` Mode (Both)**~~: **✅ COMPLETED** - Both implementations now include comprehensive dry-run functionality with the `--dry-run/-n` flag that allows users to test their configuration safely before making any changes to their database.
|
|
||||||
|
|
||||||
#### **Tier 2: High-Impact Enhancements**
|
|
||||||
|
|
||||||
1. **Add Concurrency to the Go Version**: To bring the Go implementation closer to the performance of the Rust version, it should be updated to use goroutines to process accounts and/or mailboxes in parallel.
|
|
||||||
2. **Improve Attachment Handling in Rust**: The `TODO` in the Rust IMAP client for parsing binary attachments should be completed to ensure all attachment types are saved correctly.
|
|
||||||
3. **URL-Encode Document IDs in Rust**: The CouchDB client in the Rust version should URL-encode document IDs to prevent errors when mailbox names contain special characters.
|
|
||||||
4. **Add Progress Indicators (Rust)**: For a better user experience during long syncs, the Rust version would benefit greatly from progress bars (e.g., using the `indicatif` crate).
|
|
||||||
|
|
||||||
#### **Tier 3: "Nice-to-Have" Features**
|
|
||||||
|
|
||||||
1. **Interactive Setup (Either)**: A `mail2couch setup` command to interactively generate the `config.json` file would significantly improve first-time user experience.
|
|
||||||
2. **Support for Other Protocols/Backends (Either)**: Extend the tool to support POP3 or JMAP, or to use other databases like PostgreSQL or Elasticsearch as a storage backend.
|
|
||||||
3. **Backfill Command (Either)**: A `--backfill-all` flag to ignore existing sync metadata and perform a complete re-sync of an account.
|
|
||||||
|
|
@ -1,560 +0,0 @@
|
||||||
# Go vs Rust Implementation Comparison
|
|
||||||
|
|
||||||
This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch, helping users and developers choose the best implementation for their needs.
|
|
||||||
|
|
||||||
## Executive Summary
|
|
||||||
|
|
||||||
The mail2couch project offers two distinct architectural approaches to email backup:
|
|
||||||
|
|
||||||
- **Go Implementation**: A straightforward, sequential approach emphasizing simplicity and ease of understanding
|
|
||||||
- **Rust Implementation**: A sophisticated, asynchronous architecture prioritizing performance, reliability, and advanced features
|
|
||||||
|
|
||||||
**Key Finding**: The Rust implementation (~3,056 LOC across 9 modules) is significantly more feature-complete and architecturally advanced than the Go implementation (~1,355 LOC across 4 modules), representing a mature evolution rather than a simple port.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Architecture & Design Philosophy
|
|
||||||
|
|
||||||
### Go Implementation: Sequential Simplicity
|
|
||||||
|
|
||||||
**Design Philosophy**: Straightforward, imperative programming with minimal abstraction
|
|
||||||
|
|
||||||
- **Processing Model**: Sequential processing of sources → mailboxes → messages
|
|
||||||
- **Error Handling**: Basic error propagation with continue-on-error for non-critical failures
|
|
||||||
- **Modularity**: Simple package structure (`config`, `couch`, `mail`, `main`)
|
|
||||||
- **State Management**: Minimal state, mostly function-based operations
|
|
||||||
|
|
||||||
```go
|
|
||||||
// Example: Sequential processing approach
|
|
||||||
func processImapSource(source *config.MailSource, couchClient *couch.Client,
|
|
||||||
dbName string, maxMessages int, dryRun bool) error {
|
|
||||||
// Connect to IMAP server
|
|
||||||
imapClient, err := mail.NewImapClient(source)
|
|
||||||
if err != nil {
|
|
||||||
return fmt.Errorf("failed to connect to IMAP server: %w", err)
|
|
||||||
}
|
|
||||||
defer imapClient.Logout()
|
|
||||||
|
|
||||||
// Process each mailbox sequentially
|
|
||||||
for _, mailbox := range mailboxes {
|
|
||||||
// Process messages one by one
|
|
||||||
messages, currentUIDs, err := imapClient.GetMessages(...)
|
|
||||||
// Store messages synchronously
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Rust Implementation: Async Orchestration
|
|
||||||
|
|
||||||
**Design Philosophy**: Modular, type-safe architecture with comprehensive error handling
|
|
||||||
|
|
||||||
- **Processing Model**: Asynchronous coordination with concurrent network operations
|
|
||||||
- **Error Handling**: Sophisticated retry logic, structured error types, graceful degradation
|
|
||||||
- **Modularity**: Well-separated concerns (`cli`, `config`, `couch`, `imap`, `sync`, `filters`, `schemas`)
|
|
||||||
- **State Management**: Stateful coordinator pattern with proper resource management
|
|
||||||
|
|
||||||
```rust
|
|
||||||
// Example: Asynchronous coordination approach
|
|
||||||
impl SyncCoordinator {
|
|
||||||
pub async fn sync_all_sources(&mut self) -> Result<Vec<SourceSyncResult>> {
|
|
||||||
let mut results = Vec::new();
|
|
||||||
let sources = self.config.mail_sources.clone();
|
|
||||||
|
|
||||||
for source in &sources {
|
|
||||||
if !source.enabled {
|
|
||||||
info!("Skipping disabled source: {}", source.name);
|
|
||||||
continue;
|
|
||||||
}
|
|
||||||
|
|
||||||
match self.sync_source(source).await {
|
|
||||||
Ok(result) => {
|
|
||||||
info!("✅ Completed sync for {}: {} messages across {} mailboxes",
|
|
||||||
result.source_name, result.total_messages, result.mailboxes_processed);
|
|
||||||
results.push(result);
|
|
||||||
}
|
|
||||||
Err(e) => {
|
|
||||||
error!("❌ Failed to sync source {}: {}", source.name, e);
|
|
||||||
// Continue with other sources even if one fails
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
Ok(results)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Performance & Scalability
|
|
||||||
|
|
||||||
### Concurrency Models
|
|
||||||
|
|
||||||
| Aspect | Go Implementation | Rust Implementation |
|
|
||||||
|--------|------------------|-------------------|
|
|
||||||
| **Processing Model** | Sequential (blocking) | Asynchronous (non-blocking) |
|
|
||||||
| **Account Processing** | One at a time | One at a time with internal concurrency |
|
|
||||||
| **Mailbox Processing** | One at a time | One at a time with async I/O |
|
|
||||||
| **Message Processing** | One at a time | Batch processing with async operations |
|
|
||||||
| **Network Operations** | Blocking I/O | Non-blocking async I/O |
|
|
||||||
|
|
||||||
### IMAP Filtering Efficiency
|
|
||||||
|
|
||||||
**Go: Client-Side Filtering**
|
|
||||||
```go
|
|
||||||
// Downloads ALL messages first, then filters locally
|
|
||||||
messages := imap.FetchAll()
|
|
||||||
filtered := []Message{}
|
|
||||||
for _, msg := range messages {
|
|
||||||
if ShouldProcessMessage(msg, filter) {
|
|
||||||
filtered = append(filtered, msg)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rust: Server-Side Filtering**
|
|
||||||
```rust
|
|
||||||
// Filters on server, only downloads matching messages
|
|
||||||
pub async fn search_messages_advanced(
|
|
||||||
&mut self,
|
|
||||||
since_date: Option<&DateTime<Utc>>,
|
|
||||||
subject_keywords: Option<&[String]>,
|
|
||||||
from_keywords: Option<&[String]>,
|
|
||||||
) -> Result<Vec<u32>> {
|
|
||||||
let mut search_parts = Vec::new();
|
|
||||||
|
|
||||||
if let Some(keywords) = subject_keywords {
|
|
||||||
for keyword in keywords {
|
|
||||||
search_parts.push(format!("SUBJECT \"{}\"", keyword));
|
|
||||||
}
|
|
||||||
}
|
|
||||||
// Server processes the filter, returns only matching UIDs
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Performance Impact**: For a mailbox with 10,000 emails where you only want recent messages:
|
|
||||||
- **Go**: Downloads all 10,000 emails, then filters locally
|
|
||||||
- **Rust**: Server filters first, downloads only matching emails (potentially 10x less data transfer)
|
|
||||||
|
|
||||||
### Error Recovery and Resilience
|
|
||||||
|
|
||||||
**Go: Basic Error Handling**
|
|
||||||
```go
|
|
||||||
err := processImapSource(&source, couchClient, dbName, args.MaxMessages, args.DryRun)
|
|
||||||
if err != nil {
|
|
||||||
log.Printf("ERROR: Failed to process IMAP source %s: %v", source.Name, err)
|
|
||||||
}
|
|
||||||
// Continues with next source, no retry logic
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rust: Intelligent Retry Logic**
|
|
||||||
```rust
|
|
||||||
async fn retry_operation<F, Fut, T>(&self, operation_name: &str, operation: F) -> Result<T>
|
|
||||||
where F: Fn() -> Fut, Fut: std::future::Future<Output = Result<T>>
|
|
||||||
{
|
|
||||||
const MAX_RETRIES: u32 = 3;
|
|
||||||
const RETRY_DELAY_MS: u64 = 1000;
|
|
||||||
|
|
||||||
for attempt in 1..=MAX_RETRIES {
|
|
||||||
match operation().await {
|
|
||||||
Ok(result) => return Ok(result),
|
|
||||||
Err(e) => {
|
|
||||||
let is_retryable = match &e.downcast_ref::<CouchError>() {
|
|
||||||
Some(CouchError::Http(_)) => true,
|
|
||||||
Some(CouchError::CouchDb { status, .. }) => *status >= 500,
|
|
||||||
_ => false,
|
|
||||||
};
|
|
||||||
|
|
||||||
if is_retryable && attempt < MAX_RETRIES {
|
|
||||||
warn!("Attempt {}/{} failed for {}: {}. Retrying in {}ms...",
|
|
||||||
attempt, MAX_RETRIES, operation_name, e, RETRY_DELAY_MS);
|
|
||||||
async_std::task::sleep(Duration::from_millis(RETRY_DELAY_MS)).await;
|
|
||||||
} else {
|
|
||||||
error!("Operation {} failed after {} attempts: {}",
|
|
||||||
operation_name, attempt, e);
|
|
||||||
return Err(e);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
}
|
|
||||||
unreachable!()
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Developer Experience
|
|
||||||
|
|
||||||
### Code Complexity and Learning Curve
|
|
||||||
|
|
||||||
| Aspect | Go Implementation | Rust Implementation |
|
|
||||||
|--------|------------------|-------------------|
|
|
||||||
| **Lines of Code** | 1,355 | 3,056 |
|
|
||||||
| **Number of Files** | 4 | 9 |
|
|
||||||
| **Dependencies** | 4 external | 14+ external |
|
|
||||||
| **Compilation Time** | 2-3 seconds | 6+ seconds |
|
|
||||||
| **Learning Curve** | Low | Medium-High |
|
|
||||||
| **Debugging Ease** | Simple stack traces | Rich error context |
|
|
||||||
|
|
||||||
### Dependency Management
|
|
||||||
|
|
||||||
**Go Dependencies (minimal approach):**
|
|
||||||
```go
|
|
||||||
require (
|
|
||||||
github.com/emersion/go-imap/v2 v2.0.0-beta.5
|
|
||||||
github.com/emersion/go-message v0.18.1
|
|
||||||
github.com/go-kivik/kivik/v4 v4.4.0
|
|
||||||
github.com/spf13/pflag v1.0.7
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Rust Dependencies (rich ecosystem):**
|
|
||||||
```toml
|
|
||||||
[dependencies]
|
|
||||||
anyhow = "1.0"
|
|
||||||
serde = { version = "1.0", features = ["derive"] }
|
|
||||||
serde_json = "1.0"
|
|
||||||
tokio = { version = "1.0", features = ["full"] }
|
|
||||||
reqwest = { version = "0.11", features = ["json"] }
|
|
||||||
clap = { version = "4.0", features = ["derive"] }
|
|
||||||
log = "0.4"
|
|
||||||
env_logger = "0.10"
|
|
||||||
chrono = { version = "0.4", features = ["serde"] }
|
|
||||||
async-imap = "0.9"
|
|
||||||
mail-parser = "0.6"
|
|
||||||
thiserror = "1.0"
|
|
||||||
glob = "0.3"
|
|
||||||
dirs = "5.0"
|
|
||||||
```
|
|
||||||
|
|
||||||
**Trade-offs**:
|
|
||||||
- **Go**: Faster builds, fewer potential security vulnerabilities, simpler dependency tree
|
|
||||||
- **Rust**: Richer functionality, better error types, more battle-tested async ecosystem
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Feature Comparison Matrix
|
|
||||||
|
|
||||||
| Feature | Go Implementation | Rust Implementation | Notes |
|
|
||||||
|---------|------------------|-------------------|-------|
|
|
||||||
| **Core Functionality** |
|
|
||||||
| IMAP Email Sync | ✅ | ✅ | Both fully functional |
|
|
||||||
| CouchDB Storage | ✅ | ✅ | Both support attachments |
|
|
||||||
| Incremental Sync | ✅ | ✅ | Both use metadata tracking |
|
|
||||||
| **Configuration** |
|
|
||||||
| JSON Config Files | ✅ | ✅ | Same format, auto-discovery |
|
|
||||||
| Folder Filtering | ✅ | ✅ | Both support wildcards |
|
|
||||||
| Date Filtering | ✅ | ✅ | Since date support |
|
|
||||||
| Keyword Filtering | ✅ (client-side) | ✅ (server-side) | Rust is more efficient |
|
|
||||||
| **CLI Features** |
|
|
||||||
| GNU-style Arguments | ✅ | ✅ | Both use standard conventions |
|
|
||||||
| Dry-run Mode | ✅ | ✅ | Both recently implemented |
|
|
||||||
| Bash Completion | ✅ | ✅ | Auto-generated scripts |
|
|
||||||
| Help System | Basic | Rich | Rust uses clap framework |
|
|
||||||
| **Reliability** |
|
|
||||||
| Error Handling | Basic | Advanced | Rust has retry logic |
|
|
||||||
| Connection Recovery | Manual | Automatic | Rust handles reconnections |
|
|
||||||
| Resource Management | Manual (defer) | Automatic (RAII) | Rust prevents leaks |
|
|
||||||
| **Performance** |
|
|
||||||
| Concurrent Processing | ❌ | ✅ | Rust uses async/await |
|
|
||||||
| Server-side Filtering | ❌ | ✅ | Rust reduces bandwidth |
|
|
||||||
| Memory Efficiency | Good | Excellent | Rust zero-copy where possible |
|
|
||||||
| **Development** |
|
|
||||||
| Test Coverage | Minimal | Comprehensive | Rust has extensive tests |
|
|
||||||
| Documentation | Basic | Rich | Rust has detailed docs |
|
|
||||||
| Type Safety | Good | Excellent | Rust prevents more errors |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Use Case Recommendations
|
|
||||||
|
|
||||||
### Choose Go Implementation When:
|
|
||||||
|
|
||||||
#### 🎯 **Personal Use & Simplicity**
|
|
||||||
- Single email account or small number of accounts
|
|
||||||
- Infrequent synchronization (daily/weekly)
|
|
||||||
- Simple setup requirements
|
|
||||||
- You want to understand/modify the code easily
|
|
||||||
|
|
||||||
#### 🎯 **Resource Constraints**
|
|
||||||
- Memory-limited environments
|
|
||||||
- CPU-constrained systems
|
|
||||||
- Quick deployment needed
|
|
||||||
- Minimal disk space for binaries
|
|
||||||
|
|
||||||
#### 🎯 **Development Preferences**
|
|
||||||
- Team familiar with Go
|
|
||||||
- Preference for simple, readable code
|
|
||||||
- Fast compilation important for development cycle
|
|
||||||
- Minimal external dependencies preferred
|
|
||||||
|
|
||||||
**Example Use Case**: Personal backup of 1-2 Gmail accounts, running weekly on a Raspberry Pi.
|
|
||||||
|
|
||||||
### Choose Rust Implementation When:
|
|
||||||
|
|
||||||
#### 🚀 **Performance Critical Scenarios**
|
|
||||||
- Multiple email accounts (3+ accounts)
|
|
||||||
- Large mailboxes (10,000+ emails)
|
|
||||||
- Frequent synchronization (hourly/real-time)
|
|
||||||
- High-volume email processing
|
|
||||||
|
|
||||||
#### 🚀 **Production Environments**
|
|
||||||
- Business-critical email backups
|
|
||||||
- Need for reliable error recovery
|
|
||||||
- 24/7 operation requirements
|
|
||||||
- Professional deployment standards
|
|
||||||
|
|
||||||
#### 🚀 **Advanced Features Required**
|
|
||||||
- Server-side IMAP filtering needed
|
|
||||||
- Complex folder filtering patterns
|
|
||||||
- Detailed logging and monitoring
|
|
||||||
- Long-term maintenance planned
|
|
||||||
|
|
||||||
**Example Use Case**: Corporate email backup system handling 10+ accounts with complex filtering rules, running continuously in a production environment.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Performance Benchmarks
|
|
||||||
|
|
||||||
### Theoretical Performance Comparison
|
|
||||||
|
|
||||||
| Scenario | Go Implementation | Rust Implementation | Improvement |
|
|
||||||
|----------|------------------|-------------------|-------------|
|
|
||||||
| **Single small account** (1,000 emails) | 2-3 minutes | 1-2 minutes | 33-50% faster |
|
|
||||||
| **Multiple accounts** (3 accounts, 5,000 emails each) | 15-20 minutes | 8-12 minutes | 40-47% faster |
|
|
||||||
| **Large mailbox** (50,000 emails with filtering) | 45-60 minutes | 15-25 minutes | 58-67% faster |
|
|
||||||
| **Network errors** (5% packet loss) | May fail/restart | Continues with retry | Much more reliable |
|
|
||||||
|
|
||||||
*Note: These are estimated performance improvements based on architectural differences. Actual performance will vary based on network conditions, server capabilities, and email characteristics.*
|
|
||||||
|
|
||||||
### Resource Usage
|
|
||||||
|
|
||||||
| Metric | Go Implementation | Rust Implementation |
|
|
||||||
|--------|------------------|-------------------|
|
|
||||||
| **Memory Usage** | 20-50 MB | 15-40 MB |
|
|
||||||
| **CPU Usage** | Low (single-threaded) | Medium (multi-threaded) |
|
|
||||||
| **Network Efficiency** | Lower (downloads then filters) | Higher (filters then downloads) |
|
|
||||||
| **Disk I/O** | Sequential writes | Batched writes |
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Migration Guide
|
|
||||||
|
|
||||||
### From Go to Rust
|
|
||||||
|
|
||||||
If you're currently using the Go implementation and considering migration:
|
|
||||||
|
|
||||||
#### **When to Migrate**:
|
|
||||||
- You experience performance issues with large mailboxes
|
|
||||||
- You need better error recovery and reliability
|
|
||||||
- You want more efficient network usage
|
|
||||||
- You're planning long-term maintenance
|
|
||||||
|
|
||||||
#### **Migration Steps**:
|
|
||||||
1. **Test in parallel**: Run both implementations with `--dry-run` to compare results
|
|
||||||
2. **Backup existing data**: Ensure your CouchDB data is backed up
|
|
||||||
3. **Update configuration**: Configuration format is identical, no changes needed
|
|
||||||
4. **Replace binary**: Simply replace the Go binary with the Rust binary
|
|
||||||
5. **Monitor performance**: Compare sync times and resource usage
|
|
||||||
|
|
||||||
#### **Compatibility Notes**:
|
|
||||||
- ✅ Configuration files are 100% compatible
|
|
||||||
- ✅ CouchDB database format is identical
|
|
||||||
- ✅ Command-line arguments are the same
|
|
||||||
- ✅ Dry-run mode works identically
|
|
||||||
|
|
||||||
### Staying with Go
|
|
||||||
|
|
||||||
The Go implementation remains fully supported and is appropriate when:
|
|
||||||
- Current performance meets your needs
|
|
||||||
- Simplicity is more important than features
|
|
||||||
- Team lacks Rust expertise
|
|
||||||
- Resource usage is already optimized for your environment
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Technical Architecture Details
|
|
||||||
|
|
||||||
### Go Implementation Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
go/
|
|
||||||
├── main.go # Entry point and orchestration
|
|
||||||
├── config/
|
|
||||||
│ └── config.go # Configuration loading and CLI parsing
|
|
||||||
├── couch/
|
|
||||||
│ └── couch.go # CouchDB client and operations
|
|
||||||
└── mail/
|
|
||||||
└── imap.go # IMAP client and message processing
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Characteristics**:
|
|
||||||
- Monolithic processing flow
|
|
||||||
- Synchronous I/O operations
|
|
||||||
- Basic error handling
|
|
||||||
- Minimal abstraction layers
|
|
||||||
|
|
||||||
### Rust Implementation Structure
|
|
||||||
|
|
||||||
```
|
|
||||||
rust/src/
|
|
||||||
├── main.rs # Entry point
|
|
||||||
├── lib.rs # Library exports
|
|
||||||
├── cli.rs # Command-line interface
|
|
||||||
├── config.rs # Configuration management
|
|
||||||
├── sync.rs # Synchronization coordinator
|
|
||||||
├── imap.rs # IMAP client with retry logic
|
|
||||||
├── couch.rs # CouchDB client with error handling
|
|
||||||
├── filters.rs # Filtering utilities
|
|
||||||
└── schemas.rs # Data structure definitions
|
|
||||||
```
|
|
||||||
|
|
||||||
**Key Characteristics**:
|
|
||||||
- Modular architecture with clear separation
|
|
||||||
- Asynchronous I/O with tokio runtime
|
|
||||||
- Comprehensive error handling
|
|
||||||
- Rich abstraction layers
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Security Considerations
|
|
||||||
|
|
||||||
Both implementations currently share the same security limitations and features:
|
|
||||||
|
|
||||||
### Current Security Features
|
|
||||||
- ✅ TLS/SSL support for IMAP and CouchDB connections
|
|
||||||
- ✅ Configuration file validation
|
|
||||||
- ✅ Safe handling of email content
|
|
||||||
|
|
||||||
### Shared Security Limitations
|
|
||||||
- ⚠️ Plaintext passwords in configuration files
|
|
||||||
- ⚠️ No OAuth2 support for modern email providers
|
|
||||||
- ⚠️ No credential encryption at rest
|
|
||||||
|
|
||||||
### Future Security Improvements (Recommended for Both)
|
|
||||||
1. **Environment Variable Credentials**: Support reading passwords from environment variables
|
|
||||||
2. **OAuth2 Integration**: Support modern authentication for Gmail, Outlook, etc.
|
|
||||||
3. **Credential Encryption**: Encrypt stored credentials with system keyring integration
|
|
||||||
4. **Audit Logging**: Enhanced logging of authentication and access events
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Deployment Considerations
|
|
||||||
|
|
||||||
### Go Implementation Deployment
|
|
||||||
|
|
||||||
**Binary Name**: `mail2couch-go`
|
|
||||||
|
|
||||||
**Advantages**:
|
|
||||||
- Single binary deployment
|
|
||||||
- Minimal system dependencies
|
|
||||||
- Lower memory footprint
|
|
||||||
- Faster startup time
|
|
||||||
|
|
||||||
**Best Practices**:
|
|
||||||
```bash
|
|
||||||
# Build for production using justfile
|
|
||||||
just build-go-release
|
|
||||||
|
|
||||||
# Or build directly
|
|
||||||
cd go && go build -ldflags="-s -w" -o mail2couch-go .
|
|
||||||
|
|
||||||
# Deploy with systemd service
|
|
||||||
sudo cp go/mail2couch-go /usr/local/bin/
|
|
||||||
sudo systemctl enable mail2couch-go.service
|
|
||||||
```
|
|
||||||
|
|
||||||
### Rust Implementation Deployment
|
|
||||||
|
|
||||||
**Binary Name**: `mail2couch-rs`
|
|
||||||
|
|
||||||
**Advantages**:
|
|
||||||
- Better resource utilization under load
|
|
||||||
- Superior error recovery
|
|
||||||
- More detailed logging and monitoring
|
|
||||||
- Enhanced CLI experience
|
|
||||||
|
|
||||||
**Best Practices**:
|
|
||||||
```bash
|
|
||||||
# Build optimized release using justfile
|
|
||||||
just build-rust-release
|
|
||||||
|
|
||||||
# Or build directly
|
|
||||||
cd rust && cargo build --release
|
|
||||||
|
|
||||||
# Deploy with enhanced monitoring
|
|
||||||
sudo cp rust/target/release/mail2couch-rs /usr/local/bin/
|
|
||||||
sudo systemctl enable mail2couch-rs.service
|
|
||||||
|
|
||||||
# Configure structured logging
|
|
||||||
export RUST_LOG=info
|
|
||||||
export MAIL2COUCH_LOG_FORMAT=json
|
|
||||||
```
|
|
||||||
|
|
||||||
### Universal Installation
|
|
||||||
|
|
||||||
```bash
|
|
||||||
# Build and install both implementations (user-local)
|
|
||||||
just install
|
|
||||||
# This installs to ~/bin/mail2couch-go and ~/bin/mail2couch-rs
|
|
||||||
|
|
||||||
# Build and install both implementations (system-wide)
|
|
||||||
sudo just system-install
|
|
||||||
# This installs to /usr/local/bin/mail2couch-go and /usr/local/bin/mail2couch-rs
|
|
||||||
```
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Future Development Roadmap
|
|
||||||
|
|
||||||
### Short-term Improvements (Both Implementations)
|
|
||||||
|
|
||||||
1. **Security Enhancements**
|
|
||||||
- Environment variable credential support
|
|
||||||
- OAuth2 authentication for major providers
|
|
||||||
- Encrypted credential storage
|
|
||||||
|
|
||||||
2. **Usability Improvements**
|
|
||||||
- Interactive configuration wizard
|
|
||||||
- Progress indicators for long-running operations
|
|
||||||
- Enhanced error messages with solutions
|
|
||||||
|
|
||||||
### Long-term Strategic Direction
|
|
||||||
|
|
||||||
#### Go Implementation (Maintenance Mode)
|
|
||||||
- Bug fixes and security updates
|
|
||||||
- Maintain compatibility with Rust version
|
|
||||||
- Focus on simplicity and stability
|
|
||||||
- Target: Personal and small-scale deployments
|
|
||||||
|
|
||||||
#### Rust Implementation (Active Development)
|
|
||||||
- Performance optimizations
|
|
||||||
- Advanced features (web interface, monitoring APIs)
|
|
||||||
- Enterprise features (clustering, high availability)
|
|
||||||
- Target: Production and large-scale deployments
|
|
||||||
|
|
||||||
### Recommended Development Focus
|
|
||||||
|
|
||||||
1. **Primary Development**: Focus on Rust implementation for new features
|
|
||||||
2. **Compatibility Maintenance**: Ensure Go version remains compatible
|
|
||||||
3. **Migration Path**: Provide clear migration guidance and tooling
|
|
||||||
4. **Documentation**: Maintain comprehensive documentation for both
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Conclusion
|
|
||||||
|
|
||||||
Both implementations represent excellent software engineering practices and serve different market segments effectively:
|
|
||||||
|
|
||||||
- **Go Implementation**: Ideal for users who prioritize simplicity, fast deployment, and ease of understanding. Perfect for personal use and small-scale deployments.
|
|
||||||
|
|
||||||
- **Rust Implementation**: Superior choice for users who need performance, reliability, and advanced features. Recommended for production environments and large-scale email processing.
|
|
||||||
|
|
||||||
### Final Recommendation
|
|
||||||
|
|
||||||
**For new deployments**: Start with the Rust implementation unless simplicity is your primary concern. The performance benefits and reliability features provide significant value.
|
|
||||||
|
|
||||||
**For existing Go users**: Consider migration if you experience performance limitations or need better error recovery. The migration path is straightforward due to configuration compatibility.
|
|
||||||
|
|
||||||
**For development contributions**: Focus on the Rust implementation for new features, while maintaining the Go version for bug fixes and compatibility.
|
|
||||||
|
|
||||||
The project demonstrates that having two implementations can serve different user needs effectively, with each leveraging the strengths of its respective programming language and ecosystem.
|
|
||||||
20
README.md
20
README.md
|
|
@ -43,6 +43,26 @@ A powerful email backup utility that synchronizes mail from IMAP accounts to Cou
|
||||||
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
|
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
|
||||||
- **Error Resilience**: Graceful handling of network issues and server problems
|
- **Error Resilience**: Graceful handling of network issues and server problems
|
||||||
|
|
||||||
|
## Project Status
|
||||||
|
|
||||||
|
**Production Ready** (August 2025): Both Go and Rust implementations are fully functional with identical feature sets and database output.
|
||||||
|
|
||||||
|
- ✅ **Complete Feature Parity**: Server-side filtering, binary attachments, incremental sync
|
||||||
|
- ✅ **Comprehensive Testing**: Verified identical CouchDB output between implementations
|
||||||
|
- ✅ **SystemD Integration**: Automated scheduling with timer units
|
||||||
|
- ✅ **Production Quality**: Robust error handling, retry logic, dry-run mode
|
||||||
|
|
||||||
|
## 📚 Documentation
|
||||||
|
|
||||||
|
Comprehensive documentation is available in the [`docs/`](docs/) directory:
|
||||||
|
|
||||||
|
- **[docs/README.md](docs/README.md)** - Documentation overview and quick start
|
||||||
|
- **[docs/ANALYSIS.md](docs/ANALYSIS.md)** - Technical analysis and current status
|
||||||
|
- **[docs/IMPLEMENTATION_COMPARISON.md](docs/IMPLEMENTATION_COMPARISON.md)** - Go vs Rust comparison
|
||||||
|
- **[docs/FOLDER_PATTERNS.md](docs/FOLDER_PATTERNS.md)** - Folder filtering guide
|
||||||
|
- **[docs/couchdb-schemas.md](docs/couchdb-schemas.md)** - Database schema documentation
|
||||||
|
- **[docs/TODO.md](docs/TODO.md)** - Development roadmap and future plans
|
||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
### Installation
|
### Installation
|
||||||
|
|
|
||||||
47
TODO.md
47
TODO.md
|
|
@ -1,47 +0,0 @@
|
||||||
# mail2couch TODO and Feature Requests
|
|
||||||
|
|
||||||
## Planned Features
|
|
||||||
|
|
||||||
### Keyword Filtering for Messages
|
|
||||||
|
|
||||||
Add support for filtering messages by keywords in various message fields. This would extend the current `messageFilter` configuration.
|
|
||||||
|
|
||||||
**Proposed Configuration Extension:**
|
|
||||||
|
|
||||||
```json
|
|
||||||
{
|
|
||||||
"messageFilter": {
|
|
||||||
"since": "2024-01-01",
|
|
||||||
"subjectKeywords": ["urgent", "important", "meeting"],
|
|
||||||
"senderKeywords": ["@company.com", "notifications"],
|
|
||||||
"recipientKeywords": ["team@company.com", "all@"]
|
|
||||||
}
|
|
||||||
}
|
|
||||||
```
|
|
||||||
|
|
||||||
**Implementation Details:**
|
|
||||||
|
|
||||||
- `subjectKeywords`: Array of keywords to match in email subject lines
|
|
||||||
- `senderKeywords`: Array of keywords to match in sender email addresses or names
|
|
||||||
- `recipientKeywords`: Array of keywords to match in recipient (To/CC/BCC) addresses or names
|
|
||||||
- Keywords should support both inclusive (must contain) and exclusive (must not contain) patterns
|
|
||||||
- Case-insensitive matching by default
|
|
||||||
- Support for simple wildcards or regex patterns
|
|
||||||
|
|
||||||
**Use Cases:**
|
|
||||||
|
|
||||||
1. **Corporate Email Filtering**: Only backup emails from specific domains or containing work-related keywords
|
|
||||||
2. **Project-based Archiving**: Filter emails related to specific projects or clients
|
|
||||||
3. **Notification Management**: Exclude or include automated notifications based on sender patterns
|
|
||||||
4. **Security**: Filter out potential spam/phishing by excluding certain keywords or senders
|
|
||||||
|
|
||||||
**Implementation Priority:** Medium - useful for reducing storage requirements and focusing on relevant emails.
|
|
||||||
|
|
||||||
## Other Planned Improvements
|
|
||||||
|
|
||||||
1. **Real IMAP Message Parsing**: Replace placeholder data with actual message content
|
|
||||||
2. **Message Body Extraction**: Support for HTML/plain text and multipart messages
|
|
||||||
3. **Attachment Handling**: Optional support for email attachments
|
|
||||||
4. **Batch Operations**: Improve CouchDB insertion performance
|
|
||||||
5. **Error Recovery**: Retry logic and partial sync recovery
|
|
||||||
6. **Testing**: Comprehensive unit test coverage
|
|
||||||
134
docs/ANALYSIS.md
Normal file
134
docs/ANALYSIS.md
Normal file
|
|
@ -0,0 +1,134 @@
|
||||||
|
# Comprehensive Analysis of `mail2couch` Implementations
|
||||||
|
|
||||||
|
*Last Updated: August 2025*
|
||||||
|
|
||||||
|
This document provides a comprehensive analysis of the `mail2couch` project after reaching production readiness. Both Go and Rust implementations are now fully functional, tested, and production-ready with equivalent feature sets.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Current State (August 2025)
|
||||||
|
|
||||||
|
The project now consists of **two production-ready implementations** of the same core tool, both achieving feature parity and production quality.
|
||||||
|
|
||||||
|
### **The Go Implementation**
|
||||||
|
- ✅ **Production Ready**: Fully functional with comprehensive IMAP and CouchDB integration
|
||||||
|
- ✅ **Server-side Filtering**: Implements IMAP SEARCH with keyword filtering and graceful fallbacks
|
||||||
|
- ✅ **Complete Feature Set**: All core functionality implemented and tested
|
||||||
|
- ✅ **Robust Error Handling**: Proper connection management and retry logic
|
||||||
|
- ✅ **Dry-run Mode**: Comprehensive testing capabilities without data changes
|
||||||
|
|
||||||
|
### **The Rust Implementation**
|
||||||
|
- ✅ **Production Ready**: Fully functional with advanced async architecture
|
||||||
|
- ✅ **Performance Optimized**: Asynchronous operations with superior concurrency
|
||||||
|
- ✅ **Feature Complete**: All functionality implemented with enhanced user experience
|
||||||
|
- ✅ **Enterprise Grade**: Comprehensive error handling, retry logic, and monitoring
|
||||||
|
- ✅ **Advanced CLI**: Rich logging, progress reporting, and configuration validation
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 2. Status of Previous Issues
|
||||||
|
|
||||||
|
All major issues identified in earlier analysis have been **resolved**:
|
||||||
|
|
||||||
|
### ✅ **Resolved Issues**
|
||||||
|
* **`Incomplete Rust Implementation`**: **FULLY RESOLVED** - Rust implementation is production-ready
|
||||||
|
* **`Inefficient Keyword Filtering`**: **FULLY RESOLVED** - Both implementations use server-side IMAP SEARCH
|
||||||
|
* **`Performance for Large-Scale Use`**: **SIGNIFICANTLY IMPROVED** - Async Rust, optimized Go
|
||||||
|
* **`Missing Dry-Run Mode`**: **FULLY RESOLVED** - Comprehensive dry-run support in both
|
||||||
|
* **`Inconsistent Database Output`**: **FULLY RESOLVED** - Identical document schemas and behavior
|
||||||
|
* **`Limited Error Handling`**: **FULLY RESOLVED** - Robust error handling and retry logic
|
||||||
|
* **`Binary Attachment Issues`**: **FULLY RESOLVED** - Full binary attachment support verified
|
||||||
|
|
||||||
|
### ⚠️ **Outstanding Issues**
|
||||||
|
* **`Security Model`**: Still requires plaintext passwords in configuration (environment variable support planned)
|
||||||
|
* **`Web Interface`**: Not implemented (not currently prioritized for core functionality)
|
||||||
|
* **`Interactive Setup`**: Could improve first-time user experience (low priority)
|
||||||
|
|
||||||
|
## 3. Current Comparative Analysis: Go vs. Rust
|
||||||
|
|
||||||
|
Both implementations now provide equivalent functionality with different architectural approaches:
|
||||||
|
|
||||||
|
### **Go Implementation**
|
||||||
|
|
||||||
|
#### **Strengths**:
|
||||||
|
- ✅ **Simplicity**: Sequential, straightforward code that's easy to understand and debug
|
||||||
|
- ✅ **Fast Development**: Quick compilation and simple deployment model
|
||||||
|
- ✅ **Server-side Filtering**: Full IMAP SEARCH implementation with graceful fallbacks
|
||||||
|
- ✅ **Production Stability**: Reliable operation with proper error handling
|
||||||
|
- ✅ **Comprehensive Testing**: Verified equivalent output with Rust implementation
|
||||||
|
|
||||||
|
#### **Trade-offs**:
|
||||||
|
- ⚖️ **Sequential Processing**: Processes one mailbox at a time (adequate for most use cases)
|
||||||
|
- ⚖️ **Standard Error Handling**: Basic retry logic sufficient for typical deployments
|
||||||
|
|
||||||
|
### **Rust Implementation**
|
||||||
|
|
||||||
|
#### **Strengths**:
|
||||||
|
- ✅ **High Performance**: Async architecture enables concurrent operations
|
||||||
|
- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging
|
||||||
|
- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation
|
||||||
|
- ✅ **Memory Safety**: Rust's compile-time guarantees prevent entire classes of bugs
|
||||||
|
- ✅ **Advanced Architecture**: Modular design facilitates maintenance and feature additions
|
||||||
|
|
||||||
|
#### **Trade-offs**:
|
||||||
|
- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge for maintenance
|
||||||
|
- ⚖️ **Build Time**: Longer compilation times during development
|
||||||
|
|
||||||
|
## 4. Production Readiness Assessment
|
||||||
|
|
||||||
|
Both implementations have achieved **production readiness** with comprehensive testing and validation:
|
||||||
|
|
||||||
|
### **Shared Capabilities**
|
||||||
|
- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers
|
||||||
|
- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata
|
||||||
|
- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||||
|
- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas
|
||||||
|
- ✅ **Error Resilience**: Connection retry logic, graceful degradation
|
||||||
|
- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration
|
||||||
|
|
||||||
|
### **Verification Status**
|
||||||
|
- ✅ **Identical Output**: Both implementations produce identical CouchDB documents
|
||||||
|
- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable
|
||||||
|
- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results
|
||||||
|
- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified
|
||||||
|
- ✅ **Scale Testing**: Tested with thousands of messages and large attachments
|
||||||
|
|
||||||
|
### **Deployment Options**
|
||||||
|
- ✅ **SystemD Services**: Timer units for automated scheduling (30min, hourly, daily)
|
||||||
|
- ✅ **Manual Execution**: Command-line tools with comprehensive help and validation
|
||||||
|
- ✅ **Configuration Management**: Automatic config file discovery, validation
|
||||||
|
- ✅ **Monitoring Integration**: Structured logging suitable for monitoring systems
|
||||||
|
|
||||||
|
## 5. Future Enhancement Roadmap
|
||||||
|
|
||||||
|
Based on current production status, these enhancements would further improve the project:
|
||||||
|
|
||||||
|
### **High Priority**
|
||||||
|
- 🔐 **Enhanced Security**: Environment variable credential support to eliminate plaintext passwords
|
||||||
|
- 🚀 **Go Concurrency**: Optional goroutine-based parallel processing for multiple mailboxes
|
||||||
|
- 📊 **Progress Indicators**: Real-time progress reporting for long-running operations
|
||||||
|
|
||||||
|
### **Medium Priority**
|
||||||
|
- 🖥️ **Interactive Setup**: Guided configuration wizard for first-time users
|
||||||
|
- 📈 **Performance Metrics**: Built-in timing and throughput reporting
|
||||||
|
- 🔄 **Advanced Sync**: Bidirectional sync capabilities and conflict resolution
|
||||||
|
|
||||||
|
### **Low Priority**
|
||||||
|
- 🌐 **Web Interface**: Optional web UI for configuration and monitoring
|
||||||
|
- 📱 **REST API**: HTTP API for integration with other systems
|
||||||
|
- 🔌 **Plugin System**: Extensible architecture for custom filters and processors
|
||||||
|
|
||||||
|
## 6. Recommendations
|
||||||
|
|
||||||
|
### **For Production Deployment**
|
||||||
|
Both implementations are ready for production use. **Choose based on your requirements:**
|
||||||
|
|
||||||
|
- **Choose Go** if you prefer simplicity, fast builds, and straightforward maintenance
|
||||||
|
- **Choose Rust** if you need maximum performance, advanced features, or plan extensive customization
|
||||||
|
|
||||||
|
### **For Development Contributions**
|
||||||
|
- **Go implementation**: Ideal for quick fixes, simple feature additions, or learning the codebase
|
||||||
|
- **Rust implementation**: Better for performance improvements, complex features, or async operations
|
||||||
|
|
||||||
|
### **Current Status Summary**
|
||||||
|
The mail2couch project has successfully achieved its primary goal: providing reliable, production-ready email backup solutions. Both implementations offer equivalent functionality with different architectural strengths, making the project suitable for a wide range of deployment scenarios and maintenance preferences.
|
||||||
154
docs/IMPLEMENTATION_COMPARISON.md
Normal file
154
docs/IMPLEMENTATION_COMPARISON.md
Normal file
|
|
@ -0,0 +1,154 @@
|
||||||
|
# Go vs Rust Implementation Comparison
|
||||||
|
|
||||||
|
*Last Updated: August 2025*
|
||||||
|
|
||||||
|
This document provides a comprehensive technical analysis comparing the Go and Rust implementations of mail2couch after both have reached production readiness with equivalent functionality.
|
||||||
|
|
||||||
|
## Executive Summary
|
||||||
|
|
||||||
|
The mail2couch project offers **two production-ready implementations** with identical core functionality but different architectural approaches:
|
||||||
|
|
||||||
|
- **Go Implementation**: Sequential, straightforward approach emphasizing simplicity and maintainability
|
||||||
|
- **Rust Implementation**: Asynchronous, feature-rich architecture prioritizing performance and enterprise features
|
||||||
|
|
||||||
|
**Key Finding**: Both implementations now provide **equivalent functionality** and **identical database output**. The choice between them depends on operational requirements, team expertise, and performance needs rather than feature completeness.
|
||||||
|
|
||||||
|
## Feature Comparison Matrix
|
||||||
|
|
||||||
|
| Feature Category | Go Implementation | Rust Implementation | Status |
|
||||||
|
|-----------------|------------------|-------------------|---------|
|
||||||
|
| **Core Functionality** |
|
||||||
|
| IMAP/IMAPS Support | ✅ Full support | ✅ Full support | **Equivalent** |
|
||||||
|
| CouchDB Integration | ✅ Native attachments | ✅ Native attachments | **Equivalent** |
|
||||||
|
| Binary Attachments | ✅ Verified working | ✅ Verified working | **Equivalent** |
|
||||||
|
| Sync vs Archive Modes | ✅ Both modes | ✅ Both modes | **Equivalent** |
|
||||||
|
| Incremental Sync | ✅ Metadata tracking | ✅ Metadata tracking | **Equivalent** |
|
||||||
|
| **Filtering & Search** |
|
||||||
|
| Folder Filtering | ✅ IMAP LIST patterns | ✅ IMAP LIST patterns | **Equivalent** |
|
||||||
|
| Server-side Search | ✅ IMAP SEARCH keywords | ✅ IMAP SEARCH keywords | **Equivalent** |
|
||||||
|
| Keyword Filtering | ✅ Subject/sender/recipient | ✅ Subject/sender/recipient | **Equivalent** |
|
||||||
|
| Date Filtering | ✅ Since date support | ✅ Since date support | **Equivalent** |
|
||||||
|
| **Operational Features** |
|
||||||
|
| Dry-run Mode | ✅ Comprehensive | ✅ Comprehensive | **Equivalent** |
|
||||||
|
| Configuration Discovery | ✅ Multi-path search | ✅ Multi-path search | **Equivalent** |
|
||||||
|
| Command Line Interface | ✅ GNU-style flags | ✅ Modern clap-based | **Rust Advantage** |
|
||||||
|
| Progress Reporting | ✅ Basic logging | ✅ Rich structured logs | **Rust Advantage** |
|
||||||
|
| Error Handling | ✅ Retry logic | ✅ Advanced retry + async | **Rust Advantage** |
|
||||||
|
| **Performance & Architecture** |
|
||||||
|
| Concurrency Model | ⚖️ Sequential | ✅ Async/concurrent | **Rust Advantage** |
|
||||||
|
| Memory Safety | ✅ Go GC | ✅ Compile-time guarantees | **Rust Advantage** |
|
||||||
|
| Build Time | ✅ Fast (~5s) | ⚖️ Slower (~30s) | **Go Advantage** |
|
||||||
|
| Binary Size | ✅ Smaller | ⚖️ Larger | **Go Advantage** |
|
||||||
|
| Resource Usage | ✅ Low memory | ✅ Efficient async | **Equivalent** |
|
||||||
|
| **Development & Maintenance** |
|
||||||
|
| Code Complexity | ✅ Simple, readable | ⚖️ Advanced patterns | **Go Advantage** |
|
||||||
|
| Learning Curve | ✅ Easy for Go devs | ⚖️ Requires Rust knowledge | **Go Advantage** |
|
||||||
|
| Debugging | ✅ Straightforward | ⚖️ Advanced tooling needed | **Go Advantage** |
|
||||||
|
| Testing | ✅ Standard Go tests | ✅ Comprehensive test suite | **Equivalent** |
|
||||||
|
| Linting/Formatting | ✅ go fmt/vet | ✅ rustfmt/clippy | **Equivalent** |
|
||||||
|
|
||||||
|
## Production Readiness Assessment
|
||||||
|
|
||||||
|
Both implementations have achieved **production readiness** with comprehensive testing and validation:
|
||||||
|
|
||||||
|
### **Shared Capabilities**
|
||||||
|
- ✅ **IMAP Protocol Support**: Full IMAP/IMAPS with TLS, tested against multiple servers
|
||||||
|
- ✅ **CouchDB Integration**: Native attachment support, per-account databases, sync metadata
|
||||||
|
- ✅ **Filtering Systems**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||||
|
- ✅ **Data Integrity**: UID-based deduplication, consistent document schemas
|
||||||
|
- ✅ **Error Resilience**: Connection retry logic, graceful degradation
|
||||||
|
- ✅ **Operational Tools**: Dry-run mode, comprehensive logging, systemd integration
|
||||||
|
|
||||||
|
### **Verification Status**
|
||||||
|
- ✅ **Identical Output**: Both implementations produce identical CouchDB documents
|
||||||
|
- ✅ **Attachment Handling**: Binary attachments correctly stored and retrievable
|
||||||
|
- ✅ **Filtering Accuracy**: Keyword and folder filters produce consistent results
|
||||||
|
- ✅ **Incremental Sync**: Cross-implementation sync state compatibility verified
|
||||||
|
- ✅ **Scale Testing**: Tested with thousands of messages and large attachments
|
||||||
|
|
||||||
|
## Architectural Comparison
|
||||||
|
|
||||||
|
### **Go Implementation: Production Simplicity**
|
||||||
|
|
||||||
|
**Strengths:**
|
||||||
|
- ✅ **Straightforward Code**: Sequential, easy to understand and debug
|
||||||
|
- ✅ **Fast Development**: Quick compilation and simple deployment model
|
||||||
|
- ✅ **Production Stable**: Reliable operation with proper error handling
|
||||||
|
- ✅ **Low Resource**: Minimal memory usage and fast startup
|
||||||
|
|
||||||
|
**Trade-offs:**
|
||||||
|
- ⚖️ **Sequential Processing**: One mailbox at a time (adequate for most use cases)
|
||||||
|
- ⚖️ **Basic Features**: Standard CLI and logging capabilities
|
||||||
|
|
||||||
|
### **Rust Implementation: Enterprise Architecture**
|
||||||
|
|
||||||
|
**Strengths:**
|
||||||
|
- ✅ **High Performance**: Async architecture enables concurrent operations
|
||||||
|
- ✅ **Enterprise Features**: Advanced retry logic, connection pooling, detailed logging
|
||||||
|
- ✅ **Rich CLI Experience**: Comprehensive progress reporting and configuration validation
|
||||||
|
- ✅ **Memory Safety**: Compile-time guarantees prevent entire classes of bugs
|
||||||
|
- ✅ **Modular Design**: Well-structured architecture facilitates maintenance
|
||||||
|
|
||||||
|
**Trade-offs:**
|
||||||
|
- ⚖️ **Complexity**: More sophisticated architecture requires Rust knowledge
|
||||||
|
- ⚖️ **Build Time**: Longer compilation times during development
|
||||||
|
|
||||||
|
## Use Case Recommendations
|
||||||
|
|
||||||
|
### Choose **Go Implementation** When:
|
||||||
|
|
||||||
|
- 🎯 **Simplicity Priority**: Easy to understand, modify, and maintain
|
||||||
|
- 🎯 **Resource Constraints**: Memory-limited environments, quick deployment
|
||||||
|
- 🎯 **Small Scale**: Personal use, few accounts, infrequent synchronization
|
||||||
|
- 🎯 **Team Familiarity**: Go expertise available, fast development cycle important
|
||||||
|
|
||||||
|
**Example**: Personal backup of 1-2 email accounts, running daily on modest hardware.
|
||||||
|
|
||||||
|
### Choose **Rust Implementation** When:
|
||||||
|
|
||||||
|
- 🚀 **Performance Critical**: Multiple accounts, large mailboxes, frequent sync
|
||||||
|
- 🚀 **Production Environment**: Business-critical backups, 24/7 operation
|
||||||
|
- 🚀 **Advanced Features**: Rich logging, detailed progress reporting, complex filtering
|
||||||
|
- 🚀 **Long-term Maintenance**: Enterprise deployment with ongoing development
|
||||||
|
|
||||||
|
**Example**: Corporate email backup handling 10+ accounts with complex filtering, running continuously.
|
||||||
|
|
||||||
|
## Migration Compatibility
|
||||||
|
|
||||||
|
### **100% Compatible**
|
||||||
|
- ✅ Configuration files are identical between implementations
|
||||||
|
- ✅ CouchDB database format and documents are identical
|
||||||
|
- ✅ Command-line arguments and behavior are the same
|
||||||
|
- ✅ Dry-run mode works identically
|
||||||
|
- ✅ SystemD service files available for both
|
||||||
|
|
||||||
|
### **Migration Process**
|
||||||
|
1. Test new implementation with `--dry-run` to verify identical results
|
||||||
|
2. Stop current implementation
|
||||||
|
3. Replace binary (same config file works)
|
||||||
|
4. Start new implementation
|
||||||
|
5. Verify operation and performance
|
||||||
|
|
||||||
|
## Development Status
|
||||||
|
|
||||||
|
### **Current State (August 2025)**
|
||||||
|
- ✅ **Both Production Ready**: Full feature parity achieved
|
||||||
|
- ✅ **Comprehensive Testing**: Identical output verified
|
||||||
|
- ✅ **Complete Documentation**: Usage guides and examples
|
||||||
|
- ✅ **SystemD Integration**: Automated scheduling support
|
||||||
|
- ✅ **Build System**: Unified justfile for both implementations
|
||||||
|
|
||||||
|
### **Future Enhancement Priorities**
|
||||||
|
1. **Security**: Environment variable credential support
|
||||||
|
2. **Go Concurrency**: Optional parallel processing
|
||||||
|
3. **Progress Indicators**: Real-time progress reporting
|
||||||
|
4. **Interactive Setup**: Guided configuration wizard
|
||||||
|
|
||||||
|
## Conclusion
|
||||||
|
|
||||||
|
Both implementations represent production-quality solutions with different strengths:
|
||||||
|
|
||||||
|
- **Go Implementation**: Ideal for users prioritizing simplicity, maintainability, and straightforward operation
|
||||||
|
- **Rust Implementation**: Superior for users needing performance, advanced features, and enterprise-grade reliability
|
||||||
|
|
||||||
|
**Recommendation**: Choose based on your operational requirements and team expertise. Both provide identical functionality and data output, making migration straightforward when needs change.
|
||||||
94
docs/README.md
Normal file
94
docs/README.md
Normal file
|
|
@ -0,0 +1,94 @@
|
||||||
|
# mail2couch Documentation
|
||||||
|
|
||||||
|
This directory contains comprehensive documentation for the mail2couch project, which provides two production-ready implementations for backing up mail from IMAP servers to CouchDB.
|
||||||
|
|
||||||
|
## 📚 Documentation Index
|
||||||
|
|
||||||
|
### Core Documentation
|
||||||
|
- **[ANALYSIS.md](ANALYSIS.md)** - Detailed technical analysis of both implementations
|
||||||
|
- **[IMPLEMENTATION_COMPARISON.md](IMPLEMENTATION_COMPARISON.md)** - Side-by-side comparison of Go vs Rust implementations
|
||||||
|
- **[couchdb-schemas.md](couchdb-schemas.md)** - CouchDB document schemas and database structure
|
||||||
|
- **[TODO.md](TODO.md)** - Development roadmap and outstanding tasks
|
||||||
|
|
||||||
|
### Configuration & Setup
|
||||||
|
- **[FOLDER_PATTERNS.md](FOLDER_PATTERNS.md)** - Guide to folder filtering patterns and wildcards
|
||||||
|
- **[test-config-comparison.md](test-config-comparison.md)** - Configuration examples and testing scenarios
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
- **[examples/](examples/)** - Sample CouchDB documents and configuration files
|
||||||
|
- `sample-mail-document.json` - Complete email document with attachments
|
||||||
|
- `sample-sync-metadata.json` - Sync state tracking document
|
||||||
|
- `simple-mail-document.json` - Basic email document structure
|
||||||
|
|
||||||
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
Both implementations are production-ready with identical feature sets:
|
||||||
|
|
||||||
|
### Go Implementation
|
||||||
|
```bash
|
||||||
|
cd go && go build -o mail2couch-go .
|
||||||
|
./mail2couch-go --config ../config.json --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
### Rust Implementation
|
||||||
|
```bash
|
||||||
|
cd rust && cargo build --release
|
||||||
|
./target/release/mail2couch-rs --config ../config.json --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
## ✅ Current Status (August 2025)
|
||||||
|
|
||||||
|
Both implementations are **production-ready** with:
|
||||||
|
|
||||||
|
- ✅ **Full IMAP support** with TLS/SSL connections
|
||||||
|
- ✅ **Server-side folder filtering** using IMAP LIST patterns
|
||||||
|
- ✅ **Server-side message filtering** using IMAP SEARCH with keyword support
|
||||||
|
- ✅ **Binary attachment handling** with CouchDB native attachments
|
||||||
|
- ✅ **Incremental synchronization** with metadata tracking
|
||||||
|
- ✅ **Sync vs Archive modes** for different backup strategies
|
||||||
|
- ✅ **Dry-run mode** for safe testing
|
||||||
|
- ✅ **Comprehensive error handling** with graceful fallbacks
|
||||||
|
- ✅ **SystemD integration** with timer units for automated scheduling
|
||||||
|
- ✅ **Build system integration** with justfile for unified project management
|
||||||
|
|
||||||
|
## 🔧 Key Features
|
||||||
|
|
||||||
|
### Filtering & Search
|
||||||
|
- **Folder Filtering**: Wildcard patterns (`*`, `?`, `[abc]`) with include/exclude lists
|
||||||
|
- **Message Filtering**: Subject, sender, and recipient keyword filtering
|
||||||
|
- **IMAP SEARCH**: Server-side filtering when supported, client-side fallback
|
||||||
|
- **Date Filtering**: Incremental sync based on last sync time or configured since date
|
||||||
|
|
||||||
|
### Data Storage
|
||||||
|
- **CouchDB Integration**: Native attachment storage, per-account databases
|
||||||
|
- **Document Structure**: Standardized schema with full email metadata
|
||||||
|
- **Sync Metadata**: State tracking for efficient incremental updates
|
||||||
|
- **Duplicate Prevention**: UID-based deduplication across syncs
|
||||||
|
|
||||||
|
### Operations
|
||||||
|
- **Mode Selection**: Archive (append-only) or Sync (mirror) modes
|
||||||
|
- **Connection Handling**: Automatic retry logic, graceful error recovery
|
||||||
|
- **Progress Reporting**: Detailed logging with message counts and timing
|
||||||
|
- **Resource Management**: Configurable message limits, connection pooling
|
||||||
|
|
||||||
|
## 📊 Performance & Compatibility
|
||||||
|
|
||||||
|
Both implementations have been tested with:
|
||||||
|
- **IMAP Servers**: Gmail, Office365, Dovecot, GreenMail
|
||||||
|
- **CouchDB Versions**: 3.x with native attachment support
|
||||||
|
- **Message Volumes**: Tested with thousands of messages and large attachments
|
||||||
|
- **Network Conditions**: Automatic retry and reconnection handling
|
||||||
|
|
||||||
|
## 🛠️ Development
|
||||||
|
|
||||||
|
See individual implementation directories for development setup:
|
||||||
|
- **Go**: `/go/` - Standard Go toolchain with modules
|
||||||
|
- **Rust**: `/rust/` - Cargo-based build system with comprehensive testing
|
||||||
|
|
||||||
|
For unified development commands, use the project justfile:
|
||||||
|
```bash
|
||||||
|
just build # Build both implementations
|
||||||
|
just test # Run all tests
|
||||||
|
just check # Run linting and formatting
|
||||||
|
just install # Install systemd services
|
||||||
|
```
|
||||||
145
docs/TODO.md
Normal file
145
docs/TODO.md
Normal file
|
|
@ -0,0 +1,145 @@
|
||||||
|
# mail2couch Development Roadmap
|
||||||
|
|
||||||
|
*Last Updated: August 2025*
|
||||||
|
|
||||||
|
This document outlines the development roadmap for mail2couch, with both Go and Rust implementations now in production-ready status.
|
||||||
|
|
||||||
|
## ✅ Completed Major Milestones
|
||||||
|
|
||||||
|
### Production Readiness (August 2025)
|
||||||
|
- ✅ **Full Feature Parity**: Both implementations provide identical functionality
|
||||||
|
- ✅ **Server-side IMAP SEARCH**: Keyword filtering implemented in both Go and Rust
|
||||||
|
- ✅ **Binary Attachment Support**: Verified working with CouchDB native attachments
|
||||||
|
- ✅ **Incremental Sync**: Cross-implementation compatibility verified
|
||||||
|
- ✅ **Dry-run Mode**: Comprehensive testing capabilities in both implementations
|
||||||
|
- ✅ **Error Handling**: Robust retry logic and graceful fallbacks
|
||||||
|
- ✅ **SystemD Integration**: Timer units for automated scheduling
|
||||||
|
- ✅ **Build System**: Unified justfile for both implementations
|
||||||
|
- ✅ **Documentation**: Comprehensive guides and comparisons
|
||||||
|
- ✅ **Code Quality**: All linting and formatting standards met
|
||||||
|
|
||||||
|
### Architecture & Testing
|
||||||
|
- ✅ **Database Output Equivalence**: Both implementations produce identical CouchDB documents
|
||||||
|
- ✅ **Filtering Accuracy**: Server-side IMAP LIST and SEARCH with client-side fallbacks
|
||||||
|
- ✅ **Connection Handling**: TLS support, automatic retry, graceful error recovery
|
||||||
|
- ✅ **Configuration Management**: Automatic file discovery, validation, GNU-style CLI
|
||||||
|
|
||||||
|
### Originally Planned Features (Now Complete)
|
||||||
|
- ✅ **Keyword Filtering for Messages**: Subject, sender, and recipient keyword filtering implemented
|
||||||
|
- ✅ **Real IMAP Message Parsing**: Full message content extraction with go-message and mail-parser
|
||||||
|
- ✅ **Message Body Extraction**: HTML/plain text and multipart message support
|
||||||
|
- ✅ **Attachment Handling**: Complete binary attachment support with CouchDB native storage
|
||||||
|
- ✅ **Error Recovery**: Comprehensive retry logic and partial sync recovery
|
||||||
|
- ✅ **Performance**: Batch operations and efficient CouchDB insertion
|
||||||
|
|
||||||
|
## 🚧 Current Development Priorities
|
||||||
|
|
||||||
|
### High Priority
|
||||||
|
1. **🔐 Enhanced Security Model**
|
||||||
|
- Environment variable credential support (`MAIL2COUCH_IMAP_PASSWORD`, etc.)
|
||||||
|
- Eliminate plaintext passwords from configuration files
|
||||||
|
- System keyring integration for credential storage
|
||||||
|
|
||||||
|
### Medium Priority
|
||||||
|
2. **🚀 Go Implementation Concurrency**
|
||||||
|
- Optional goroutine-based parallel mailbox processing
|
||||||
|
- Maintain simplicity while improving performance for multiple accounts
|
||||||
|
- Configurable concurrency levels
|
||||||
|
|
||||||
|
3. **📊 Progress Indicators**
|
||||||
|
- Real-time progress reporting for long-running operations
|
||||||
|
- ETA calculations for large mailbox synchronization
|
||||||
|
- Progress bars for terminal output
|
||||||
|
|
||||||
|
4. **🖥️ Interactive Setup**
|
||||||
|
- Guided configuration wizard (`mail2couch setup`)
|
||||||
|
- Interactive validation of IMAP and CouchDB connectivity
|
||||||
|
- Generate sample configurations for common providers
|
||||||
|
|
||||||
|
### Low Priority
|
||||||
|
5. **📈 Performance Metrics**
|
||||||
|
- Built-in timing and throughput reporting
|
||||||
|
- Memory usage monitoring
|
||||||
|
- Network efficiency statistics
|
||||||
|
|
||||||
|
6. **🔄 Advanced Sync Features**
|
||||||
|
- Bidirectional sync capabilities
|
||||||
|
- Conflict resolution strategies
|
||||||
|
- Message modification detection
|
||||||
|
|
||||||
|
7. **🌐 Web Interface**
|
||||||
|
- Optional web UI for configuration and monitoring
|
||||||
|
- CouchDB view integration for email browsing
|
||||||
|
- Search interface for archived emails
|
||||||
|
|
||||||
|
8. **📱 API Integration**
|
||||||
|
- REST API for external system integration
|
||||||
|
- Webhook support for sync completion notifications
|
||||||
|
- Monitoring system integration
|
||||||
|
|
||||||
|
## 📋 Technical Debt & Improvements
|
||||||
|
|
||||||
|
### Code Quality
|
||||||
|
- **Unit Test Coverage**: Expand test coverage for both implementations
|
||||||
|
- **Integration Testing**: Automated testing with various IMAP servers
|
||||||
|
- **Performance Benchmarking**: Standardized performance comparison tools
|
||||||
|
|
||||||
|
### User Experience
|
||||||
|
- **Error Messages**: More descriptive error messages with suggested solutions
|
||||||
|
- **Configuration Validation**: Enhanced validation with helpful error descriptions
|
||||||
|
- **Logging**: Structured logging with different verbosity levels
|
||||||
|
|
||||||
|
### Security
|
||||||
|
- **OAuth2 Support**: Modern authentication for Gmail, Outlook, etc.
|
||||||
|
- **Credential Encryption**: Encrypt stored credentials at rest
|
||||||
|
- **Audit Logging**: Enhanced logging of authentication and access events
|
||||||
|
|
||||||
|
## 🎯 Release Planning
|
||||||
|
|
||||||
|
### Next Minor Release (v1.1)
|
||||||
|
- Environment variable credential support
|
||||||
|
- Interactive setup command
|
||||||
|
- Enhanced error messages
|
||||||
|
|
||||||
|
### Next Major Release (v2.0)
|
||||||
|
- OAuth2 authentication support
|
||||||
|
- Web interface (optional)
|
||||||
|
- Go implementation concurrency improvements
|
||||||
|
|
||||||
|
## 📊 Implementation Status
|
||||||
|
|
||||||
|
| Feature Category | Go Implementation | Rust Implementation | Priority |
|
||||||
|
|-----------------|------------------|-------------------|----------|
|
||||||
|
| **Core Features** | ✅ Complete | ✅ Complete | - |
|
||||||
|
| **Security Model** | ⚠️ Basic | ⚠️ Basic | High |
|
||||||
|
| **Concurrency** | ⚠️ Sequential | ✅ Async | Medium |
|
||||||
|
| **Progress Reporting** | ⚠️ Basic | ⚠️ Basic | Medium |
|
||||||
|
| **Interactive Setup** | ❌ Missing | ❌ Missing | Medium |
|
||||||
|
| **Web Interface** | ❌ Missing | ❌ Missing | Low |
|
||||||
|
|
||||||
|
## 🤝 Contributing
|
||||||
|
|
||||||
|
### Areas Needing Contribution
|
||||||
|
1. **Security Features**: OAuth2 implementation, credential encryption
|
||||||
|
2. **User Experience**: Interactive setup, progress indicators
|
||||||
|
3. **Testing**: Unit tests, integration tests, performance benchmarks
|
||||||
|
4. **Documentation**: Usage examples, troubleshooting guides
|
||||||
|
|
||||||
|
### Development Guidelines
|
||||||
|
- Maintain feature parity between Go and Rust implementations
|
||||||
|
- Follow established code quality standards (linting, formatting)
|
||||||
|
- Include comprehensive testing for new features
|
||||||
|
- Update documentation with new functionality
|
||||||
|
|
||||||
|
## 📝 Notes
|
||||||
|
|
||||||
|
### Design Decisions
|
||||||
|
- **Two Implementations**: Maintain both Go (simplicity) and Rust (performance) versions
|
||||||
|
- **Configuration Compatibility**: Ensure identical configuration formats
|
||||||
|
- **Database Compatibility**: Both implementations must produce identical CouchDB output
|
||||||
|
|
||||||
|
### Long-term Vision
|
||||||
|
- Position Go implementation for personal/small-scale use
|
||||||
|
- Position Rust implementation for enterprise/large-scale use
|
||||||
|
- Maintain migration path between implementations
|
||||||
|
- Focus on reliability and data integrity above all else
|
||||||
|
|
@ -45,6 +45,12 @@ type MessageFilter struct {
|
||||||
RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses
|
RecipientKeywords []string `json:"recipientKeywords,omitempty"` // Filter by keywords in recipient addresses
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// HasKeywordFilters checks if this filter has any keyword-based filters that can use IMAP SEARCH
|
||||||
|
func (mf *MessageFilter) HasKeywordFilters() bool {
|
||||||
|
return len(mf.SubjectKeywords) > 0 || len(mf.SenderKeywords) > 0
|
||||||
|
// Note: RecipientKeywords not included as IMAP SEARCH doesn't have a reliable TO field search
|
||||||
|
}
|
||||||
|
|
||||||
func LoadConfig(path string) (*Config, error) {
|
func LoadConfig(path string) (*Config, error) {
|
||||||
configFile, err := os.Open(path)
|
configFile, err := os.Open(path)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
|
||||||
236
go/mail/imap.go
236
go/mail/imap.go
|
|
@ -201,15 +201,63 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
||||||
}
|
}
|
||||||
uidCmd.Close()
|
uidCmd.Close()
|
||||||
|
|
||||||
// Determine which messages to fetch based on since date
|
// Determine which messages to fetch based on filtering criteria
|
||||||
var seqSet imap.SeqSet
|
var seqSet imap.SeqSet
|
||||||
|
|
||||||
if since != nil {
|
// Use advanced search with keyword filtering when available
|
||||||
// Use IMAP SEARCH to find messages since the specified date
|
if messageFilter != nil && messageFilter.HasKeywordFilters() {
|
||||||
searchCriteria := &imap.SearchCriteria{
|
log.Printf("Using IMAP SEARCH with keyword filters")
|
||||||
Since: *since,
|
uids, err := c.searchMessagesAdvanced(since, messageFilter)
|
||||||
}
|
if err != nil {
|
||||||
|
log.Printf("Advanced IMAP SEARCH failed, falling back to simple search: %v", err)
|
||||||
|
// Fall back to simple date-based search or fetch all
|
||||||
|
if since != nil {
|
||||||
|
searchCriteria := &imap.SearchCriteria{Since: *since}
|
||||||
|
searchCmd := c.Search(searchCriteria, nil)
|
||||||
|
searchResults, err := searchCmd.Wait()
|
||||||
|
if err != nil {
|
||||||
|
log.Printf("Simple IMAP SEARCH also failed, fetching recent messages: %v", err)
|
||||||
|
numToFetch := mbox.NumMessages
|
||||||
|
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||||
|
numToFetch = uint32(maxMessages)
|
||||||
|
}
|
||||||
|
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||||
|
} else {
|
||||||
|
searchSeqNums := searchResults.AllSeqNums()
|
||||||
|
if len(searchSeqNums) == 0 {
|
||||||
|
return []*Message{}, currentUIDs, nil
|
||||||
|
}
|
||||||
|
for _, seqNum := range searchSeqNums {
|
||||||
|
seqSet.AddNum(seqNum)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
numToFetch := mbox.NumMessages
|
||||||
|
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||||
|
numToFetch = uint32(maxMessages)
|
||||||
|
}
|
||||||
|
if numToFetch > 0 {
|
||||||
|
seqSet.AddRange(mbox.NumMessages-numToFetch+1, mbox.NumMessages)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else {
|
||||||
|
// Use results from advanced search
|
||||||
|
if len(uids) == 0 {
|
||||||
|
return []*Message{}, currentUIDs, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// Limit results if maxMessages is specified
|
||||||
|
if maxMessages > 0 && len(uids) > maxMessages {
|
||||||
|
uids = uids[len(uids)-maxMessages:]
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, uid := range uids {
|
||||||
|
seqSet.AddNum(uid)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} else if since != nil {
|
||||||
|
// Use simple IMAP SEARCH for date filtering only
|
||||||
|
searchCriteria := &imap.SearchCriteria{Since: *since}
|
||||||
searchCmd := c.Search(searchCriteria, nil)
|
searchCmd := c.Search(searchCriteria, nil)
|
||||||
searchResults, err := searchCmd.Wait()
|
searchResults, err := searchCmd.Wait()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|
@ -237,7 +285,7 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
// No since date specified, fetch recent messages up to maxMessages
|
// No filtering - fetch recent messages up to maxMessages
|
||||||
numToFetch := mbox.NumMessages
|
numToFetch := mbox.NumMessages
|
||||||
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
if maxMessages > 0 && int(numToFetch) > maxMessages {
|
||||||
numToFetch = uint32(maxMessages)
|
numToFetch = uint32(maxMessages)
|
||||||
|
|
@ -274,8 +322,8 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
||||||
// Apply message-level keyword filtering
|
// Apply message-level keyword filtering (only for keywords not handled by IMAP SEARCH)
|
||||||
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter) {
|
if messageFilter != nil && !c.ShouldProcessMessage(parsedMsg, messageFilter, messageFilter.HasKeywordFilters()) {
|
||||||
continue // Skip this message due to keyword filter
|
continue // Skip this message due to keyword filter
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -289,6 +337,137 @@ func (c *ImapClient) GetMessages(mailbox string, since *time.Time, maxMessages i
|
||||||
return messages, currentUIDs, nil
|
return messages, currentUIDs, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// buildOrChain creates a nested OR chain for multiple keywords
|
||||||
|
// Example: ["A", "B", "C"] becomes: A OR (B OR C)
|
||||||
|
func buildOrChain(keywords []string, headerKey string) *imap.SearchCriteria {
|
||||||
|
if len(keywords) == 0 {
|
||||||
|
return &imap.SearchCriteria{}
|
||||||
|
}
|
||||||
|
|
||||||
|
if len(keywords) == 1 {
|
||||||
|
return &imap.SearchCriteria{
|
||||||
|
Header: []imap.SearchCriteriaHeaderField{{
|
||||||
|
Key: headerKey,
|
||||||
|
Value: keywords[0],
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// For multiple keywords, build nested OR structure
|
||||||
|
// Start with the last keyword
|
||||||
|
result := &imap.SearchCriteria{
|
||||||
|
Header: []imap.SearchCriteriaHeaderField{{
|
||||||
|
Key: headerKey,
|
||||||
|
Value: keywords[len(keywords)-1],
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
|
||||||
|
// Build the chain backwards: each previous keyword becomes "keyword OR result"
|
||||||
|
for i := len(keywords) - 2; i >= 0; i-- {
|
||||||
|
keyword := keywords[i]
|
||||||
|
keywordCriteria := &imap.SearchCriteria{
|
||||||
|
Header: []imap.SearchCriteriaHeaderField{{
|
||||||
|
Key: headerKey,
|
||||||
|
Value: keyword,
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
|
||||||
|
result = &imap.SearchCriteria{
|
||||||
|
Or: [][2]imap.SearchCriteria{{
|
||||||
|
*keywordCriteria,
|
||||||
|
*result,
|
||||||
|
}},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return result
|
||||||
|
}
|
||||||
|
|
||||||
|
// searchMessagesAdvanced performs IMAP SEARCH with keyword filtering
|
||||||
|
// Returns sequence numbers of messages matching the search criteria
|
||||||
|
func (c *ImapClient) searchMessagesAdvanced(since *time.Time, messageFilter *config.MessageFilter) ([]uint32, error) {
|
||||||
|
// Build search criteria using structured approach
|
||||||
|
searchCriteria := &imap.SearchCriteria{}
|
||||||
|
|
||||||
|
// Add date filter
|
||||||
|
if since != nil {
|
||||||
|
searchCriteria.Since = *since
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add subject keyword filters (use OR logic for multiple subject keywords)
|
||||||
|
if len(messageFilter.SubjectKeywords) > 0 {
|
||||||
|
if len(messageFilter.SubjectKeywords) == 1 {
|
||||||
|
// Single subject keyword - add to main criteria
|
||||||
|
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||||
|
Key: "Subject",
|
||||||
|
Value: messageFilter.SubjectKeywords[0],
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
// Multiple subject keywords - need to create a chain of OR conditions
|
||||||
|
// Build a nested OR structure: (A OR (B OR (C OR D)))
|
||||||
|
subjectCriteria := buildOrChain(messageFilter.SubjectKeywords, "Subject")
|
||||||
|
if len(searchCriteria.Header) > 0 || !searchCriteria.Since.IsZero() {
|
||||||
|
// Combine with existing criteria
|
||||||
|
searchCriteria.And(subjectCriteria)
|
||||||
|
} else {
|
||||||
|
*searchCriteria = *subjectCriteria
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add sender keyword filters (use OR logic for multiple sender keywords)
|
||||||
|
if len(messageFilter.SenderKeywords) > 0 {
|
||||||
|
if len(messageFilter.SenderKeywords) == 1 {
|
||||||
|
// Single sender keyword - add to main criteria
|
||||||
|
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||||
|
Key: "From",
|
||||||
|
Value: messageFilter.SenderKeywords[0],
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
// Multiple sender keywords - need to create a chain of OR conditions
|
||||||
|
senderCriteria := buildOrChain(messageFilter.SenderKeywords, "From")
|
||||||
|
// Always use AND to combine with existing criteria
|
||||||
|
searchCriteria.And(senderCriteria)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Add recipient keyword filters (use OR logic for multiple recipient keywords)
|
||||||
|
if len(messageFilter.RecipientKeywords) > 0 {
|
||||||
|
if len(messageFilter.RecipientKeywords) == 1 {
|
||||||
|
// Single recipient keyword - add to main criteria
|
||||||
|
searchCriteria.Header = append(searchCriteria.Header, imap.SearchCriteriaHeaderField{
|
||||||
|
Key: "To",
|
||||||
|
Value: messageFilter.RecipientKeywords[0],
|
||||||
|
})
|
||||||
|
} else {
|
||||||
|
// Multiple recipient keywords - need to create a chain of OR conditions
|
||||||
|
recipientCriteria := buildOrChain(messageFilter.RecipientKeywords, "To")
|
||||||
|
// Always use AND to combine with existing criteria
|
||||||
|
searchCriteria.And(recipientCriteria)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("Using IMAP SEARCH with keyword filters (subject: %v, sender: %v, recipient: %v)",
|
||||||
|
messageFilter.SubjectKeywords, messageFilter.SenderKeywords, messageFilter.RecipientKeywords)
|
||||||
|
|
||||||
|
// Execute search
|
||||||
|
searchCmd := c.Search(searchCriteria, nil)
|
||||||
|
searchResults, err := searchCmd.Wait()
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("advanced search failed: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// Convert results to sequence numbers
|
||||||
|
seqNums := searchResults.AllSeqNums()
|
||||||
|
var uids []uint32
|
||||||
|
for _, seqNum := range seqNums {
|
||||||
|
uids = append(uids, seqNum)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("Found %d messages matching advanced search criteria", len(uids))
|
||||||
|
return uids, nil
|
||||||
|
}
|
||||||
|
|
||||||
// parseMessage parses an IMAP fetch response into our Message struct
|
// parseMessage parses an IMAP fetch response into our Message struct
|
||||||
func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) {
|
func (c *ImapClient) parseMessage(fetchMsg *imapclient.FetchMessageData) (*Message, error) {
|
||||||
msg := &Message{
|
msg := &Message{
|
||||||
|
|
@ -458,27 +637,30 @@ func (c *ImapClient) parseMessagePart(entity *message.Entity, msg *Message) erro
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
// ShouldProcessMessage checks if a message should be processed based on keyword filters
|
// ShouldProcessMessage checks if a message should be processed based on keyword filters
|
||||||
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter) bool {
|
// serverSideFiltered indicates if subject/sender keywords were already filtered server-side via IMAP SEARCH
|
||||||
// Check subject keywords
|
func (c *ImapClient) ShouldProcessMessage(msg *Message, filter *config.MessageFilter, serverSideFiltered bool) bool {
|
||||||
if len(filter.SubjectKeywords) > 0 {
|
// Skip subject and sender keyword checks if already filtered server-side
|
||||||
if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) {
|
if !serverSideFiltered {
|
||||||
return false
|
// Check subject keywords
|
||||||
}
|
if len(filter.SubjectKeywords) > 0 {
|
||||||
}
|
if !c.containsAnyKeyword(strings.ToLower(msg.Subject), filter.SubjectKeywords) {
|
||||||
|
return false
|
||||||
// Check sender keywords
|
|
||||||
if len(filter.SenderKeywords) > 0 {
|
|
||||||
senderMatch := false
|
|
||||||
for _, sender := range msg.From {
|
|
||||||
if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) {
|
|
||||||
senderMatch = true
|
|
||||||
break
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
if !senderMatch {
|
|
||||||
return false
|
// Check sender keywords
|
||||||
|
if len(filter.SenderKeywords) > 0 {
|
||||||
|
senderMatch := false
|
||||||
|
for _, sender := range msg.From {
|
||||||
|
if c.containsAnyKeyword(strings.ToLower(sender), filter.SenderKeywords) {
|
||||||
|
senderMatch = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !senderMatch {
|
||||||
|
return false
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -47,7 +47,7 @@ pub fn parse_command_line() -> CommandLineArgs {
|
||||||
}
|
}
|
||||||
|
|
||||||
CommandLineArgs {
|
CommandLineArgs {
|
||||||
config_path: matches.get_one::<String>("config").map(|s| s.clone()),
|
config_path: matches.get_one::<String>("config").cloned(),
|
||||||
max_messages: matches.get_one::<u32>("max-messages").copied(),
|
max_messages: matches.get_one::<u32>("max-messages").copied(),
|
||||||
dry_run: matches.get_flag("dry-run"),
|
dry_run: matches.get_flag("dry-run"),
|
||||||
generate_bash_completion: matches.get_flag("generate-bash-completion"),
|
generate_bash_completion: matches.get_flag("generate-bash-completion"),
|
||||||
|
|
@ -57,20 +57,23 @@ pub fn parse_command_line() -> CommandLineArgs {
|
||||||
|
|
||||||
/// Generate bash completion script for mail2couch
|
/// Generate bash completion script for mail2couch
|
||||||
pub fn generate_bash_completion() {
|
pub fn generate_bash_completion() {
|
||||||
let app_name = env::args().next()
|
let app_name = env::args()
|
||||||
|
.next()
|
||||||
.map(|path| {
|
.map(|path| {
|
||||||
Path::new(&path).file_name()
|
Path::new(&path)
|
||||||
|
.file_name()
|
||||||
.and_then(|name| name.to_str())
|
.and_then(|name| name.to_str())
|
||||||
.unwrap_or("mail2couch")
|
.unwrap_or("mail2couch")
|
||||||
.to_string()
|
.to_string()
|
||||||
})
|
})
|
||||||
.unwrap_or_else(|| "mail2couch".to_string());
|
.unwrap_or_else(|| "mail2couch".to_string());
|
||||||
|
|
||||||
let script = format!(r#"#!/bin/bash
|
let script = format!(
|
||||||
# Bash completion script for {}
|
r#"#!/bin/bash
|
||||||
# Generated automatically by {} --generate-bash-completion
|
# Bash completion script for {app_name}
|
||||||
|
# Generated automatically by {app_name} --generate-bash-completion
|
||||||
|
|
||||||
_{}_completions() {{
|
_{app_name}_completions() {{
|
||||||
local cur prev words cword
|
local cur prev words cword
|
||||||
_init_completion || return
|
_init_completion || return
|
||||||
|
|
||||||
|
|
@ -98,15 +101,16 @@ _{}_completions() {{
|
||||||
}}
|
}}
|
||||||
|
|
||||||
# Register the completion function
|
# Register the completion function
|
||||||
complete -F _{}_completions {}
|
complete -F _{app_name}_completions {app_name}
|
||||||
|
|
||||||
# Enable completion for common variations of the command name
|
# Enable completion for common variations of the command name
|
||||||
if [[ "$({} --help 2>/dev/null)" =~ "mail2couch" ]]; then
|
if [[ "$({app_name} --help 2>/dev/null)" =~ "mail2couch" ]]; then
|
||||||
complete -F _{}_completions mail2couch
|
complete -F _{app_name}_completions mail2couch
|
||||||
fi
|
fi
|
||||||
"#, app_name, app_name, app_name, app_name, app_name, app_name, app_name);
|
"#
|
||||||
|
);
|
||||||
|
|
||||||
print!("{}", script);
|
print!("{script}");
|
||||||
}
|
}
|
||||||
|
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
|
|
|
||||||
|
|
@ -161,7 +161,12 @@ impl Config {
|
||||||
|
|
||||||
// Add user directory paths
|
// Add user directory paths
|
||||||
if let Some(home_dir) = dirs::home_dir() {
|
if let Some(home_dir) = dirs::home_dir() {
|
||||||
candidates.push(home_dir.join(".config").join("mail2couch").join("config.json"));
|
candidates.push(
|
||||||
|
home_dir
|
||||||
|
.join(".config")
|
||||||
|
.join("mail2couch")
|
||||||
|
.join("config.json"),
|
||||||
|
);
|
||||||
candidates.push(home_dir.join(".mail2couch.json"));
|
candidates.push(home_dir.join(".mail2couch.json"));
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -172,9 +177,7 @@ impl Config {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
Err(ConfigError::NotFound {
|
Err(ConfigError::NotFound { paths: candidates })
|
||||||
paths: candidates,
|
|
||||||
})
|
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Load configuration with automatic file discovery
|
/// Load configuration with automatic file discovery
|
||||||
|
|
|
||||||
|
|
@ -58,14 +58,14 @@ impl CouchClient {
|
||||||
match operation().await {
|
match operation().await {
|
||||||
Ok(result) => {
|
Ok(result) => {
|
||||||
if attempt > 1 {
|
if attempt > 1 {
|
||||||
log::debug!("✅ CouchDB {} successful on attempt {}", operation_name, attempt);
|
log::debug!("✅ CouchDB {operation_name} successful on attempt {attempt}");
|
||||||
}
|
}
|
||||||
return Ok(result);
|
return Ok(result);
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
// Check if this is a retryable error
|
// Check if this is a retryable error
|
||||||
let is_retryable = match &e.downcast_ref::<CouchError>() {
|
let is_retryable = match &e.downcast_ref::<CouchError>() {
|
||||||
Some(CouchError::Http(_)) => true, // Network errors are retryable
|
Some(CouchError::Http(_)) => true, // Network errors are retryable
|
||||||
Some(CouchError::CouchDb { status, .. }) => {
|
Some(CouchError::CouchDb { status, .. }) => {
|
||||||
// Retry on server errors (5xx) but not client errors (4xx)
|
// Retry on server errors (5xx) but not client errors (4xx)
|
||||||
*status >= 500
|
*status >= 500
|
||||||
|
|
@ -154,7 +154,12 @@ impl CouchClient {
|
||||||
StatusCode::CREATED | StatusCode::ACCEPTED => Ok(()),
|
StatusCode::CREATED | StatusCode::ACCEPTED => Ok(()),
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to create database {}: {} - {}", db_name, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to create database {}: {} - {}",
|
||||||
|
db_name,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -173,7 +178,11 @@ impl CouchClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Store a mail document in CouchDB with optional attachments and retry logic
|
/// Store a mail document in CouchDB with optional attachments and retry logic
|
||||||
pub async fn store_mail_document(&self, db_name: &str, mut document: MailDocument) -> Result<String> {
|
pub async fn store_mail_document(
|
||||||
|
&self,
|
||||||
|
db_name: &str,
|
||||||
|
mut document: MailDocument,
|
||||||
|
) -> Result<String> {
|
||||||
// Set the document ID if not already set
|
// Set the document ID if not already set
|
||||||
if document.id.is_none() {
|
if document.id.is_none() {
|
||||||
document.set_id();
|
document.set_id();
|
||||||
|
|
@ -195,25 +204,28 @@ impl CouchClient {
|
||||||
request = request.basic_auth(username, Some(password));
|
request = request.basic_auth(username, Some(password));
|
||||||
}
|
}
|
||||||
|
|
||||||
let response = request.send().await
|
let response = request.send().await.map_err(CouchError::Http)?;
|
||||||
.map_err(|e| CouchError::Http(e))?;
|
|
||||||
|
|
||||||
match response.status() {
|
match response.status() {
|
||||||
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
||||||
let couch_response: CouchResponse = response.json().await
|
let couch_response: CouchResponse =
|
||||||
.map_err(|e| CouchError::Http(e))?;
|
response.json().await.map_err(CouchError::Http)?;
|
||||||
Ok(couch_response.id.unwrap_or_else(|| doc_id.clone()))
|
Ok(couch_response.id.unwrap_or_else(|| doc_id.clone()))
|
||||||
}
|
}
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await
|
let error_text = response
|
||||||
|
.text()
|
||||||
|
.await
|
||||||
.unwrap_or_else(|_| "Failed to read error response".to_string());
|
.unwrap_or_else(|_| "Failed to read error response".to_string());
|
||||||
Err(CouchError::CouchDb {
|
Err(CouchError::CouchDb {
|
||||||
status: status.as_u16(),
|
status: status.as_u16(),
|
||||||
message: error_text,
|
message: error_text,
|
||||||
}.into())
|
}
|
||||||
|
.into())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}).await
|
})
|
||||||
|
.await
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Store an attachment for a document in CouchDB
|
/// Store an attachment for a document in CouchDB
|
||||||
|
|
@ -232,8 +244,12 @@ impl CouchClient {
|
||||||
// Upload the attachment
|
// Upload the attachment
|
||||||
let encoded_doc_id = urlencoding::encode(doc_id);
|
let encoded_doc_id = urlencoding::encode(doc_id);
|
||||||
let encoded_attachment_name = urlencoding::encode(attachment_name);
|
let encoded_attachment_name = urlencoding::encode(attachment_name);
|
||||||
let url = format!("{}/{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev);
|
let url = format!(
|
||||||
let mut request = self.client
|
"{}/{}/{}/{}?rev={}",
|
||||||
|
self.base_url, db_name, encoded_doc_id, encoded_attachment_name, rev
|
||||||
|
);
|
||||||
|
let mut request = self
|
||||||
|
.client
|
||||||
.put(&url)
|
.put(&url)
|
||||||
.header("Content-Type", content_type)
|
.header("Content-Type", content_type)
|
||||||
.body(data.to_vec());
|
.body(data.to_vec());
|
||||||
|
|
@ -247,11 +263,16 @@ impl CouchClient {
|
||||||
match response.status() {
|
match response.status() {
|
||||||
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
StatusCode::CREATED | StatusCode::ACCEPTED => {
|
||||||
let couch_response: CouchResponse = response.json().await?;
|
let couch_response: CouchResponse = response.json().await?;
|
||||||
Ok(couch_response.rev.unwrap_or_else(|| rev))
|
Ok(couch_response.rev.unwrap_or(rev))
|
||||||
}
|
}
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to store attachment {}: {} - {}", attachment_name, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to store attachment {}: {} - {}",
|
||||||
|
attachment_name,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -276,13 +297,22 @@ impl CouchClient {
|
||||||
StatusCode::NOT_FOUND => Ok(None),
|
StatusCode::NOT_FOUND => Ok(None),
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to get document {}: {} - {}", doc_id, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to get document {}: {} - {}",
|
||||||
|
doc_id,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Store sync metadata in CouchDB
|
/// Store sync metadata in CouchDB
|
||||||
pub async fn store_sync_metadata(&self, db_name: &str, metadata: &SyncMetadata) -> Result<String> {
|
pub async fn store_sync_metadata(
|
||||||
|
&self,
|
||||||
|
db_name: &str,
|
||||||
|
metadata: &SyncMetadata,
|
||||||
|
) -> Result<String> {
|
||||||
let doc_id = metadata.id.as_ref().unwrap();
|
let doc_id = metadata.id.as_ref().unwrap();
|
||||||
|
|
||||||
// Try to get existing document first to get the revision
|
// Try to get existing document first to get the revision
|
||||||
|
|
@ -308,14 +338,19 @@ impl CouchClient {
|
||||||
}
|
}
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to store sync metadata {}: {} - {}", doc_id, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to store sync metadata {}: {} - {}",
|
||||||
|
doc_id,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Get sync metadata for a mailbox
|
/// Get sync metadata for a mailbox
|
||||||
pub async fn get_sync_metadata(&self, db_name: &str, mailbox: &str) -> Result<SyncMetadata> {
|
pub async fn get_sync_metadata(&self, db_name: &str, mailbox: &str) -> Result<SyncMetadata> {
|
||||||
let doc_id = format!("sync_metadata_{}", mailbox);
|
let doc_id = format!("sync_metadata_{mailbox}");
|
||||||
let encoded_doc_id = urlencoding::encode(&doc_id);
|
let encoded_doc_id = urlencoding::encode(&doc_id);
|
||||||
let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id);
|
let url = format!("{}/{}/{}", self.base_url, db_name, encoded_doc_id);
|
||||||
let mut request = self.client.get(&url);
|
let mut request = self.client.get(&url);
|
||||||
|
|
@ -331,12 +366,15 @@ impl CouchClient {
|
||||||
let metadata: SyncMetadata = response.json().await?;
|
let metadata: SyncMetadata = response.json().await?;
|
||||||
Ok(metadata)
|
Ok(metadata)
|
||||||
}
|
}
|
||||||
StatusCode::NOT_FOUND => {
|
StatusCode::NOT_FOUND => Err(CouchError::NotFound { id: doc_id }.into()),
|
||||||
Err(CouchError::NotFound { id: doc_id }.into())
|
|
||||||
}
|
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to get sync metadata {}: {} - {}", doc_id, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to get sync metadata {}: {} - {}",
|
||||||
|
doc_id,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -373,7 +411,12 @@ impl CouchClient {
|
||||||
}
|
}
|
||||||
status => {
|
status => {
|
||||||
let error_text = response.text().await?;
|
let error_text = response.text().await?;
|
||||||
Err(anyhow!("Failed to get database info for {}: {} - {}", db_name, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to get database info for {}: {} - {}",
|
||||||
|
db_name,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -382,8 +425,8 @@ impl CouchClient {
|
||||||
pub async fn get_mailbox_uids(&self, db_name: &str, mailbox: &str) -> Result<Vec<u32>> {
|
pub async fn get_mailbox_uids(&self, db_name: &str, mailbox: &str) -> Result<Vec<u32>> {
|
||||||
let url = format!("{}/{}/_all_docs", self.base_url, db_name);
|
let url = format!("{}/{}/_all_docs", self.base_url, db_name);
|
||||||
let query_params = [
|
let query_params = [
|
||||||
("startkey", format!("\"{}\"", mailbox)),
|
("startkey", format!("\"{mailbox}\"")),
|
||||||
("endkey", format!("\"{}\\ufff0\"", mailbox)), // High Unicode character for range end
|
("endkey", format!("\"{mailbox}\\ufff0\"")), // High Unicode character for range end
|
||||||
("include_docs", "false".to_string()),
|
("include_docs", "false".to_string()),
|
||||||
];
|
];
|
||||||
|
|
||||||
|
|
@ -396,7 +439,10 @@ impl CouchClient {
|
||||||
let response = request.send().await?;
|
let response = request.send().await?;
|
||||||
|
|
||||||
if !response.status().is_success() {
|
if !response.status().is_success() {
|
||||||
return Err(anyhow!("Failed to query stored messages: {}", response.status()));
|
return Err(anyhow!(
|
||||||
|
"Failed to query stored messages: {}",
|
||||||
|
response.status()
|
||||||
|
));
|
||||||
}
|
}
|
||||||
|
|
||||||
let result: serde_json::Value = response.json().await?;
|
let result: serde_json::Value = response.json().await?;
|
||||||
|
|
@ -406,7 +452,7 @@ impl CouchClient {
|
||||||
for row in rows {
|
for row in rows {
|
||||||
if let Some(id) = row["id"].as_str() {
|
if let Some(id) = row["id"].as_str() {
|
||||||
// Parse UID from document ID format: {mailbox}_{uid}
|
// Parse UID from document ID format: {mailbox}_{uid}
|
||||||
if let Some(uid_str) = id.strip_prefix(&format!("{}_", mailbox)) {
|
if let Some(uid_str) = id.strip_prefix(&format!("{mailbox}_")) {
|
||||||
if let Ok(uid) = uid_str.parse::<u32>() {
|
if let Ok(uid) = uid_str.parse::<u32>() {
|
||||||
uids.push(uid);
|
uids.push(uid);
|
||||||
}
|
}
|
||||||
|
|
@ -436,11 +482,15 @@ impl CouchClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
let doc: Value = response.json().await?;
|
let doc: Value = response.json().await?;
|
||||||
let rev = doc["_rev"].as_str()
|
let rev = doc["_rev"]
|
||||||
|
.as_str()
|
||||||
.ok_or_else(|| anyhow!("Document {} has no _rev field", doc_id))?;
|
.ok_or_else(|| anyhow!("Document {} has no _rev field", doc_id))?;
|
||||||
|
|
||||||
// Now delete the document
|
// Now delete the document
|
||||||
let delete_url = format!("{}/{}/{}?rev={}", self.base_url, db_name, encoded_doc_id, rev);
|
let delete_url = format!(
|
||||||
|
"{}/{}/{}?rev={}",
|
||||||
|
self.base_url, db_name, encoded_doc_id, rev
|
||||||
|
);
|
||||||
let mut delete_request = self.client.delete(&delete_url);
|
let mut delete_request = self.client.delete(&delete_url);
|
||||||
|
|
||||||
if let Some((username, password)) = &self.auth {
|
if let Some((username, password)) = &self.auth {
|
||||||
|
|
@ -453,7 +503,12 @@ impl CouchClient {
|
||||||
StatusCode::OK | StatusCode::ACCEPTED => Ok(()),
|
StatusCode::OK | StatusCode::ACCEPTED => Ok(()),
|
||||||
status => {
|
status => {
|
||||||
let error_text = delete_response.text().await?;
|
let error_text = delete_response.text().await?;
|
||||||
Err(anyhow!("Failed to delete document {}: {} - {}", doc_id, status, error_text))
|
Err(anyhow!(
|
||||||
|
"Failed to delete document {}: {} - {}",
|
||||||
|
doc_id,
|
||||||
|
status,
|
||||||
|
error_text
|
||||||
|
))
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -14,12 +14,18 @@ pub fn should_process_folder(folder_name: &str, filter: &FolderFilter) -> bool {
|
||||||
let included = if filter.include.is_empty() {
|
let included = if filter.include.is_empty() {
|
||||||
true
|
true
|
||||||
} else {
|
} else {
|
||||||
filter.include.iter().any(|pattern| matches_pattern(folder_name, pattern))
|
filter
|
||||||
|
.include
|
||||||
|
.iter()
|
||||||
|
.any(|pattern| matches_pattern(folder_name, pattern))
|
||||||
};
|
};
|
||||||
|
|
||||||
// If included, check if it's excluded
|
// If included, check if it's excluded
|
||||||
if included {
|
if included {
|
||||||
!filter.exclude.iter().any(|pattern| matches_pattern(folder_name, pattern))
|
!filter
|
||||||
|
.exclude
|
||||||
|
.iter()
|
||||||
|
.any(|pattern| matches_pattern(folder_name, pattern))
|
||||||
} else {
|
} else {
|
||||||
false
|
false
|
||||||
}
|
}
|
||||||
|
|
@ -45,7 +51,8 @@ fn matches_pattern(folder_name: &str, pattern: &str) -> bool {
|
||||||
|
|
||||||
/// Apply folder filters to a list of folders and return the filtered list
|
/// Apply folder filters to a list of folders and return the filtered list
|
||||||
pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec<String> {
|
pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec<String> {
|
||||||
folders.iter()
|
folders
|
||||||
|
.iter()
|
||||||
.filter(|folder| should_process_folder(folder, filter))
|
.filter(|folder| should_process_folder(folder, filter))
|
||||||
.cloned()
|
.cloned()
|
||||||
.collect()
|
.collect()
|
||||||
|
|
@ -53,19 +60,23 @@ pub fn filter_folders(folders: &[String], filter: &FolderFilter) -> Vec<String>
|
||||||
|
|
||||||
/// Expand wildcard patterns to match actual folder names
|
/// Expand wildcard patterns to match actual folder names
|
||||||
/// This is useful for validating patterns against available folders
|
/// This is useful for validating patterns against available folders
|
||||||
pub fn expand_patterns(patterns: &[String], available_folders: &[String]) -> Result<HashSet<String>> {
|
pub fn expand_patterns(
|
||||||
|
patterns: &[String],
|
||||||
|
available_folders: &[String],
|
||||||
|
) -> Result<HashSet<String>> {
|
||||||
let mut expanded = HashSet::new();
|
let mut expanded = HashSet::new();
|
||||||
|
|
||||||
for pattern in patterns {
|
for pattern in patterns {
|
||||||
let matches: Vec<_> = available_folders.iter()
|
let matches: Vec<_> = available_folders
|
||||||
|
.iter()
|
||||||
.filter(|folder| matches_pattern(folder, pattern))
|
.filter(|folder| matches_pattern(folder, pattern))
|
||||||
.cloned()
|
.cloned()
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
if matches.is_empty() {
|
if matches.is_empty() {
|
||||||
log::warn!("Pattern '{}' matches no folders", pattern);
|
log::warn!("Pattern '{pattern}' matches no folders");
|
||||||
} else {
|
} else {
|
||||||
log::debug!("Pattern '{}' matches: {:?}", pattern, matches);
|
log::debug!("Pattern '{pattern}' matches: {matches:?}");
|
||||||
expanded.extend(matches);
|
expanded.extend(matches);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -75,26 +86,31 @@ pub fn expand_patterns(patterns: &[String], available_folders: &[String]) -> Res
|
||||||
|
|
||||||
/// Validate folder filter patterns against available folders
|
/// Validate folder filter patterns against available folders
|
||||||
/// Returns warnings for patterns that don't match any folders
|
/// Returns warnings for patterns that don't match any folders
|
||||||
pub fn validate_folder_patterns(filter: &FolderFilter, available_folders: &[String]) -> Vec<String> {
|
pub fn validate_folder_patterns(
|
||||||
|
filter: &FolderFilter,
|
||||||
|
available_folders: &[String],
|
||||||
|
) -> Vec<String> {
|
||||||
let mut warnings = Vec::new();
|
let mut warnings = Vec::new();
|
||||||
|
|
||||||
// Check include patterns
|
// Check include patterns
|
||||||
for pattern in &filter.include {
|
for pattern in &filter.include {
|
||||||
let matches = available_folders.iter()
|
let matches = available_folders
|
||||||
|
.iter()
|
||||||
.any(|folder| matches_pattern(folder, pattern));
|
.any(|folder| matches_pattern(folder, pattern));
|
||||||
|
|
||||||
if !matches {
|
if !matches {
|
||||||
warnings.push(format!("Include pattern '{}' matches no folders", pattern));
|
warnings.push(format!("Include pattern '{pattern}' matches no folders"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
// Check exclude patterns
|
// Check exclude patterns
|
||||||
for pattern in &filter.exclude {
|
for pattern in &filter.exclude {
|
||||||
let matches = available_folders.iter()
|
let matches = available_folders
|
||||||
|
.iter()
|
||||||
.any(|folder| matches_pattern(folder, pattern));
|
.any(|folder| matches_pattern(folder, pattern));
|
||||||
|
|
||||||
if !matches {
|
if !matches {
|
||||||
warnings.push(format!("Exclude pattern '{}' matches no folders", pattern));
|
warnings.push(format!("Exclude pattern '{pattern}' matches no folders"));
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -105,15 +121,14 @@ pub fn validate_folder_patterns(filter: &FolderFilter, available_folders: &[Stri
|
||||||
pub fn get_filter_summary(
|
pub fn get_filter_summary(
|
||||||
all_folders: &[String],
|
all_folders: &[String],
|
||||||
filtered_folders: &[String],
|
filtered_folders: &[String],
|
||||||
filter: &FolderFilter
|
filter: &FolderFilter,
|
||||||
) -> String {
|
) -> String {
|
||||||
let total_count = all_folders.len();
|
let total_count = all_folders.len();
|
||||||
let filtered_count = filtered_folders.len();
|
let filtered_count = filtered_folders.len();
|
||||||
let excluded_count = total_count - filtered_count;
|
let excluded_count = total_count - filtered_count;
|
||||||
|
|
||||||
let mut summary = format!(
|
let mut summary = format!(
|
||||||
"Folder filtering: {} total, {} selected, {} excluded",
|
"Folder filtering: {total_count} total, {filtered_count} selected, {excluded_count} excluded"
|
||||||
total_count, filtered_count, excluded_count
|
|
||||||
);
|
);
|
||||||
|
|
||||||
if !filter.include.is_empty() {
|
if !filter.include.is_empty() {
|
||||||
|
|
@ -181,7 +196,11 @@ mod tests {
|
||||||
fn test_folder_filtering_specific() {
|
fn test_folder_filtering_specific() {
|
||||||
let folders = create_test_folders();
|
let folders = create_test_folders();
|
||||||
let filter = FolderFilter {
|
let filter = FolderFilter {
|
||||||
include: vec!["INBOX".to_string(), "Sent".to_string(), "Work/*".to_string()],
|
include: vec![
|
||||||
|
"INBOX".to_string(),
|
||||||
|
"Sent".to_string(),
|
||||||
|
"Work/*".to_string(),
|
||||||
|
],
|
||||||
exclude: vec!["*Temp*".to_string()],
|
exclude: vec!["*Temp*".to_string()],
|
||||||
};
|
};
|
||||||
|
|
||||||
|
|
|
||||||
281
rust/src/imap.rs
281
rust/src/imap.rs
|
|
@ -4,7 +4,7 @@
|
||||||
//! listing mailboxes, and retrieving messages.
|
//! listing mailboxes, and retrieving messages.
|
||||||
|
|
||||||
use crate::config::{MailSource, MessageFilter};
|
use crate::config::{MailSource, MessageFilter};
|
||||||
use crate::schemas::{MailDocument, AttachmentStub};
|
use crate::schemas::{AttachmentStub, MailDocument};
|
||||||
use anyhow::{anyhow, Result};
|
use anyhow::{anyhow, Result};
|
||||||
use async_imap::types::Fetch;
|
use async_imap::types::Fetch;
|
||||||
use async_imap::{Client, Session};
|
use async_imap::{Client, Session};
|
||||||
|
|
@ -128,7 +128,7 @@ impl ImapClient {
|
||||||
match self.establish_connection().await {
|
match self.establish_connection().await {
|
||||||
Ok(()) => {
|
Ok(()) => {
|
||||||
if attempt > 1 {
|
if attempt > 1 {
|
||||||
log::info!("✅ IMAP connection successful on attempt {}", attempt);
|
log::info!("✅ IMAP connection successful on attempt {attempt}");
|
||||||
}
|
}
|
||||||
return Ok(());
|
return Ok(());
|
||||||
}
|
}
|
||||||
|
|
@ -154,20 +154,22 @@ impl ImapClient {
|
||||||
))
|
))
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
/// Establish connection to IMAP server
|
/// Establish connection to IMAP server
|
||||||
async fn establish_connection(&mut self) -> Result<()> {
|
async fn establish_connection(&mut self) -> Result<()> {
|
||||||
// Connect to the IMAP server
|
// Connect to the IMAP server
|
||||||
let addr = format!("{}:{}", self.source.host, self.source.port);
|
let addr = format!("{}:{}", self.source.host, self.source.port);
|
||||||
let tcp_stream = TcpStream::connect(&addr).await
|
let tcp_stream = TcpStream::connect(&addr)
|
||||||
.map_err(|e| ImapError::Connection(format!("Failed to connect to {}: {}", addr, e)))?;
|
.await
|
||||||
|
.map_err(|e| ImapError::Connection(format!("Failed to connect to {addr}: {e}")))?;
|
||||||
|
|
||||||
// Determine if we should use TLS based on port
|
// Determine if we should use TLS based on port
|
||||||
let imap_stream = if self.should_use_tls() {
|
let imap_stream = if self.should_use_tls() {
|
||||||
// Use TLS for secure connection (typically port 993)
|
// Use TLS for secure connection (typically port 993)
|
||||||
let tls_connector = TlsConnector::new();
|
let tls_connector = TlsConnector::new();
|
||||||
let tls_stream = tls_connector.connect(&self.source.host, tcp_stream).await
|
let tls_stream = tls_connector
|
||||||
.map_err(|e| ImapError::Connection(format!("TLS connection failed: {}", e)))?;
|
.connect(&self.source.host, tcp_stream)
|
||||||
|
.await
|
||||||
|
.map_err(|e| ImapError::Connection(format!("TLS connection failed: {e}")))?;
|
||||||
ImapStream::Tls(tls_stream)
|
ImapStream::Tls(tls_stream)
|
||||||
} else {
|
} else {
|
||||||
// Use plain connection (typically port 143 or test environments)
|
// Use plain connection (typically port 143 or test environments)
|
||||||
|
|
@ -175,7 +177,11 @@ impl ImapClient {
|
||||||
};
|
};
|
||||||
|
|
||||||
// Log connection type for debugging
|
// Log connection type for debugging
|
||||||
let connection_type = if self.should_use_tls() { "TLS" } else { "Plain" };
|
let connection_type = if self.should_use_tls() {
|
||||||
|
"TLS"
|
||||||
|
} else {
|
||||||
|
"Plain"
|
||||||
|
};
|
||||||
log::debug!(
|
log::debug!(
|
||||||
"Connecting to {}:{} using {} connection",
|
"Connecting to {}:{} using {} connection",
|
||||||
self.source.host,
|
self.source.host,
|
||||||
|
|
@ -190,7 +196,7 @@ impl ImapClient {
|
||||||
let session = client
|
let session = client
|
||||||
.login(&self.source.user, &self.source.password)
|
.login(&self.source.user, &self.source.password)
|
||||||
.await
|
.await
|
||||||
.map_err(|e| ImapError::Authentication(format!("Login failed: {:?}", e)))?;
|
.map_err(|e| ImapError::Authentication(format!("Login failed: {e:?}")))?;
|
||||||
|
|
||||||
self.session = Some(session);
|
self.session = Some(session);
|
||||||
Ok(())
|
Ok(())
|
||||||
|
|
@ -219,17 +225,25 @@ impl ImapClient {
|
||||||
|
|
||||||
/// List all mailboxes
|
/// List all mailboxes
|
||||||
pub async fn list_mailboxes(&mut self) -> Result<Vec<String>> {
|
pub async fn list_mailboxes(&mut self) -> Result<Vec<String>> {
|
||||||
let session = self.session.as_mut()
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
let mut mailboxes = session.list(Some(""), Some("*")).await
|
let mut mailboxes = session
|
||||||
.map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {:?}", e)))?;
|
.list(Some(""), Some("*"))
|
||||||
|
.await
|
||||||
|
.map_err(|e| ImapError::Operation(format!("Failed to list mailboxes: {e:?}")))?;
|
||||||
|
|
||||||
let mut mailbox_names = Vec::new();
|
let mut mailbox_names = Vec::new();
|
||||||
while let Some(mailbox_result) = mailboxes.next().await {
|
while let Some(mailbox_result) = mailboxes.next().await {
|
||||||
match mailbox_result {
|
match mailbox_result {
|
||||||
Ok(mailbox) => mailbox_names.push(mailbox.name().to_string()),
|
Ok(mailbox) => mailbox_names.push(mailbox.name().to_string()),
|
||||||
Err(e) => return Err(ImapError::Operation(format!("Error processing mailbox: {:?}", e)).into()),
|
Err(e) => {
|
||||||
|
return Err(
|
||||||
|
ImapError::Operation(format!("Error processing mailbox: {e:?}")).into(),
|
||||||
|
)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -237,8 +251,13 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// List mailboxes using IMAP LIST with server-side pattern filtering
|
/// List mailboxes using IMAP LIST with server-side pattern filtering
|
||||||
pub async fn list_filtered_mailboxes(&mut self, filter: &crate::config::FolderFilter) -> Result<Vec<String>> {
|
pub async fn list_filtered_mailboxes(
|
||||||
let session = self.session.as_mut()
|
&mut self,
|
||||||
|
filter: &crate::config::FolderFilter,
|
||||||
|
) -> Result<Vec<String>> {
|
||||||
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
let mut all_mailboxes = Vec::new();
|
let mut all_mailboxes = Vec::new();
|
||||||
|
|
@ -251,13 +270,14 @@ impl ImapClient {
|
||||||
|
|
||||||
// Use IMAP LIST with each include pattern for server-side filtering
|
// Use IMAP LIST with each include pattern for server-side filtering
|
||||||
for pattern in &filter.include {
|
for pattern in &filter.include {
|
||||||
log::debug!("Listing mailboxes with pattern: {}", pattern);
|
log::debug!("Listing mailboxes with pattern: {pattern}");
|
||||||
|
|
||||||
let mut mailboxes = session.list(Some(""), Some(pattern)).await
|
let mut mailboxes = session.list(Some(""), Some(pattern)).await.map_err(|e| {
|
||||||
.map_err(|e| {
|
log::warn!("Failed to list mailboxes with pattern '{pattern}': {e:?}");
|
||||||
log::warn!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e);
|
ImapError::Operation(format!(
|
||||||
ImapError::Operation(format!("Failed to list mailboxes with pattern '{}': {:?}", pattern, e))
|
"Failed to list mailboxes with pattern '{pattern}': {e:?}"
|
||||||
})?;
|
))
|
||||||
|
})?;
|
||||||
|
|
||||||
while let Some(mailbox_result) = mailboxes.next().await {
|
while let Some(mailbox_result) = mailboxes.next().await {
|
||||||
match mailbox_result {
|
match mailbox_result {
|
||||||
|
|
@ -268,7 +288,7 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
log::warn!("Error processing mailbox with pattern '{}': {:?}", pattern, e);
|
log::warn!("Error processing mailbox with pattern '{pattern}': {e:?}");
|
||||||
continue;
|
continue;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -283,9 +303,10 @@ impl ImapClient {
|
||||||
let filtered_mailboxes: Vec<String> = all_mailboxes
|
let filtered_mailboxes: Vec<String> = all_mailboxes
|
||||||
.into_iter()
|
.into_iter()
|
||||||
.filter(|mailbox| {
|
.filter(|mailbox| {
|
||||||
!filter.exclude.iter().any(|exclude_pattern| {
|
!filter
|
||||||
self.matches_imap_pattern(exclude_pattern, mailbox)
|
.exclude
|
||||||
})
|
.iter()
|
||||||
|
.any(|exclude_pattern| self.matches_imap_pattern(exclude_pattern, mailbox))
|
||||||
})
|
})
|
||||||
.collect();
|
.collect();
|
||||||
|
|
||||||
|
|
@ -300,8 +321,8 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Handle simple prefix wildcard: "Work*" should match "Work/Projects"
|
// Handle simple prefix wildcard: "Work*" should match "Work/Projects"
|
||||||
if pattern.ends_with('*') && !pattern[..pattern.len()-1].contains('*') {
|
if pattern.ends_with('*') && !pattern[..pattern.len() - 1].contains('*') {
|
||||||
let prefix = &pattern[..pattern.len()-1];
|
let prefix = &pattern[..pattern.len() - 1];
|
||||||
return name.starts_with(prefix);
|
return name.starts_with(prefix);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -313,7 +334,7 @@ impl ImapClient {
|
||||||
|
|
||||||
// Handle contains wildcard: "*Temp*" should match "Work/Temp/Archive"
|
// Handle contains wildcard: "*Temp*" should match "Work/Temp/Archive"
|
||||||
if pattern.starts_with('*') && pattern.ends_with('*') {
|
if pattern.starts_with('*') && pattern.ends_with('*') {
|
||||||
let middle = &pattern[1..pattern.len()-1];
|
let middle = &pattern[1..pattern.len() - 1];
|
||||||
return name.contains(middle);
|
return name.contains(middle);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -323,11 +344,14 @@ impl ImapClient {
|
||||||
|
|
||||||
/// Select a mailbox
|
/// Select a mailbox
|
||||||
pub async fn select_mailbox(&mut self, mailbox: &str) -> Result<MailboxInfo> {
|
pub async fn select_mailbox(&mut self, mailbox: &str) -> Result<MailboxInfo> {
|
||||||
let session = self.session.as_mut()
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
let mailbox_data = session.select(mailbox).await
|
let mailbox_data = session.select(mailbox).await.map_err(|e| {
|
||||||
.map_err(|e| ImapError::Operation(format!("Failed to select mailbox {}: {:?}", mailbox, e)))?;
|
ImapError::Operation(format!("Failed to select mailbox {mailbox}: {e:?}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
Ok(MailboxInfo {
|
Ok(MailboxInfo {
|
||||||
name: mailbox.to_string(),
|
name: mailbox.to_string(),
|
||||||
|
|
@ -340,7 +364,10 @@ impl ImapClient {
|
||||||
|
|
||||||
/// Search for messages using IMAP SEARCH command with retry logic
|
/// Search for messages using IMAP SEARCH command with retry logic
|
||||||
/// Returns UIDs of matching messages
|
/// Returns UIDs of matching messages
|
||||||
pub async fn search_messages(&mut self, since_date: Option<&DateTime<Utc>>) -> Result<Vec<u32>> {
|
pub async fn search_messages(
|
||||||
|
&mut self,
|
||||||
|
since_date: Option<&DateTime<Utc>>,
|
||||||
|
) -> Result<Vec<u32>> {
|
||||||
const MAX_RETRIES: u32 = 3;
|
const MAX_RETRIES: u32 = 3;
|
||||||
const RETRY_DELAY_MS: u64 = 500;
|
const RETRY_DELAY_MS: u64 = 500;
|
||||||
|
|
||||||
|
|
@ -352,7 +379,7 @@ impl ImapClient {
|
||||||
match result {
|
match result {
|
||||||
Ok(uids) => {
|
Ok(uids) => {
|
||||||
if attempt > 1 {
|
if attempt > 1 {
|
||||||
log::debug!("✅ IMAP search successful on attempt {}", attempt);
|
log::debug!("✅ IMAP search successful on attempt {attempt}");
|
||||||
}
|
}
|
||||||
return Ok(uids);
|
return Ok(uids);
|
||||||
}
|
}
|
||||||
|
|
@ -379,25 +406,31 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Internal search implementation without retry logic
|
/// Internal search implementation without retry logic
|
||||||
async fn search_messages_internal(&mut self, since_date: Option<&DateTime<Utc>>) -> Result<Vec<u32>> {
|
async fn search_messages_internal(
|
||||||
let session = self.session.as_mut()
|
&mut self,
|
||||||
|
since_date: Option<&DateTime<Utc>>,
|
||||||
|
) -> Result<Vec<u32>> {
|
||||||
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
let search_query = if let Some(since) = since_date {
|
let search_query = if let Some(since) = since_date {
|
||||||
// Format date as required by IMAP (DD-MMM-YYYY)
|
// Format date as required by IMAP (DD-MMM-YYYY)
|
||||||
// IMAP months are 3-letter abbreviations in English
|
// IMAP months are 3-letter abbreviations in English
|
||||||
let formatted_date = since.format("%d-%b-%Y").to_string();
|
let formatted_date = since.format("%d-%b-%Y").to_string();
|
||||||
log::debug!("Searching for messages since: {}", formatted_date);
|
log::debug!("Searching for messages since: {formatted_date}");
|
||||||
format!("SINCE {}", formatted_date)
|
format!("SINCE {formatted_date}")
|
||||||
} else {
|
} else {
|
||||||
log::debug!("Searching for all messages");
|
log::debug!("Searching for all messages");
|
||||||
"ALL".to_string()
|
"ALL".to_string()
|
||||||
};
|
};
|
||||||
|
|
||||||
log::debug!("IMAP search query: {}", search_query);
|
log::debug!("IMAP search query: {search_query}");
|
||||||
|
|
||||||
let uids = session.uid_search(&search_query).await
|
let uids = session.uid_search(&search_query).await.map_err(|e| {
|
||||||
.map_err(|e| ImapError::Operation(format!("Search failed with query '{}': {:?}", search_query, e)))?;
|
ImapError::Operation(format!("Search failed with query '{search_query}': {e:?}"))
|
||||||
|
})?;
|
||||||
|
|
||||||
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
||||||
log::debug!("Found {} messages matching search criteria", uid_vec.len());
|
log::debug!("Found {} messages matching search criteria", uid_vec.len());
|
||||||
|
|
@ -414,7 +447,9 @@ impl ImapClient {
|
||||||
subject_keywords: Option<&[String]>,
|
subject_keywords: Option<&[String]>,
|
||||||
from_keywords: Option<&[String]>,
|
from_keywords: Option<&[String]>,
|
||||||
) -> Result<Vec<u32>> {
|
) -> Result<Vec<u32>> {
|
||||||
let session = self.session.as_mut()
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
let mut search_parts = Vec::new();
|
let mut search_parts = Vec::new();
|
||||||
|
|
@ -422,12 +457,12 @@ impl ImapClient {
|
||||||
// Add date filters
|
// Add date filters
|
||||||
if let Some(since) = since_date {
|
if let Some(since) = since_date {
|
||||||
let formatted_date = since.format("%d-%b-%Y").to_string();
|
let formatted_date = since.format("%d-%b-%Y").to_string();
|
||||||
search_parts.push(format!("SINCE {}", formatted_date));
|
search_parts.push(format!("SINCE {formatted_date}"));
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Some(before) = before_date {
|
if let Some(before) = before_date {
|
||||||
let formatted_date = before.format("%d-%b-%Y").to_string();
|
let formatted_date = before.format("%d-%b-%Y").to_string();
|
||||||
search_parts.push(format!("BEFORE {}", formatted_date));
|
search_parts.push(format!("BEFORE {formatted_date}"));
|
||||||
}
|
}
|
||||||
|
|
||||||
// Add subject keyword filters
|
// Add subject keyword filters
|
||||||
|
|
@ -451,25 +486,39 @@ impl ImapClient {
|
||||||
search_parts.join(" ")
|
search_parts.join(" ")
|
||||||
};
|
};
|
||||||
|
|
||||||
log::debug!("Advanced IMAP search query: {}", search_query);
|
log::debug!("Advanced IMAP search query: {search_query}");
|
||||||
|
|
||||||
let uids = session.uid_search(&search_query).await
|
let uids = session.uid_search(&search_query).await.map_err(|e| {
|
||||||
.map_err(|e| ImapError::Operation(format!("Advanced search failed with query '{}': {:?}", search_query, e)))?;
|
ImapError::Operation(format!(
|
||||||
|
"Advanced search failed with query '{search_query}': {e:?}"
|
||||||
|
))
|
||||||
|
})?;
|
||||||
|
|
||||||
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
let uid_vec: Vec<u32> = uids.into_iter().collect();
|
||||||
log::debug!("Found {} messages matching advanced search criteria", uid_vec.len());
|
log::debug!(
|
||||||
|
"Found {} messages matching advanced search criteria",
|
||||||
|
uid_vec.len()
|
||||||
|
);
|
||||||
|
|
||||||
Ok(uid_vec)
|
Ok(uid_vec)
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Fetch message by UID with attachment data
|
/// Fetch message by UID with attachment data
|
||||||
pub async fn fetch_message(&mut self, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
pub async fn fetch_message(
|
||||||
let session = self.session.as_mut()
|
&mut self,
|
||||||
|
uid: u32,
|
||||||
|
mailbox: &str,
|
||||||
|
) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||||
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
// Fetch message headers and body
|
// Fetch message headers and body
|
||||||
let mut messages = session.uid_fetch(format!("{}", uid), "RFC822").await
|
let mut messages = session
|
||||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch message {}: {:?}", uid, e)))?;
|
.uid_fetch(format!("{uid}"), "RFC822")
|
||||||
|
.await
|
||||||
|
.map_err(|e| ImapError::Operation(format!("Failed to fetch message {uid}: {e:?}")))?;
|
||||||
|
|
||||||
// Collect the first message
|
// Collect the first message
|
||||||
if let Some(message_result) = messages.next().await {
|
if let Some(message_result) = messages.next().await {
|
||||||
|
|
@ -479,7 +528,10 @@ impl ImapClient {
|
||||||
drop(messages);
|
drop(messages);
|
||||||
self.parse_message(&message, uid, mailbox).await
|
self.parse_message(&message, uid, mailbox).await
|
||||||
}
|
}
|
||||||
Err(e) => Err(ImapError::Operation(format!("Failed to process message {}: {:?}", uid, e)).into()),
|
Err(e) => Err(ImapError::Operation(format!(
|
||||||
|
"Failed to process message {uid}: {e:?}"
|
||||||
|
))
|
||||||
|
.into()),
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
Err(anyhow!("Message {} not found", uid))
|
Err(anyhow!("Message {} not found", uid))
|
||||||
|
|
@ -487,12 +539,19 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Fetch multiple messages by UIDs with attachment data
|
/// Fetch multiple messages by UIDs with attachment data
|
||||||
pub async fn fetch_messages(&mut self, uids: &[u32], max_count: Option<u32>, mailbox: &str) -> Result<Vec<(MailDocument, Vec<(String, String, Vec<u8>)>)>> {
|
pub async fn fetch_messages(
|
||||||
|
&mut self,
|
||||||
|
uids: &[u32],
|
||||||
|
max_count: Option<u32>,
|
||||||
|
mailbox: &str,
|
||||||
|
) -> Result<Vec<(MailDocument, Vec<(String, String, Vec<u8>)>)>> {
|
||||||
if uids.is_empty() {
|
if uids.is_empty() {
|
||||||
return Ok(Vec::new());
|
return Ok(Vec::new());
|
||||||
}
|
}
|
||||||
|
|
||||||
let session = self.session.as_mut()
|
let session = self
|
||||||
|
.session
|
||||||
|
.as_mut()
|
||||||
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
.ok_or_else(|| anyhow!("Not connected to IMAP server"))?;
|
||||||
|
|
||||||
// Limit the number of messages if specified
|
// Limit the number of messages if specified
|
||||||
|
|
@ -507,21 +566,24 @@ impl ImapClient {
|
||||||
};
|
};
|
||||||
|
|
||||||
// Create UID sequence
|
// Create UID sequence
|
||||||
let uid_sequence = uids_to_fetch.iter()
|
let uid_sequence = uids_to_fetch
|
||||||
|
.iter()
|
||||||
.map(|uid| uid.to_string())
|
.map(|uid| uid.to_string())
|
||||||
.collect::<Vec<_>>()
|
.collect::<Vec<_>>()
|
||||||
.join(",");
|
.join(",");
|
||||||
|
|
||||||
// Fetch messages
|
// Fetch messages
|
||||||
let mut messages = session.uid_fetch(&uid_sequence, "RFC822").await
|
let mut messages = session
|
||||||
.map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {:?}", e)))?;
|
.uid_fetch(&uid_sequence, "RFC822")
|
||||||
|
.await
|
||||||
|
.map_err(|e| ImapError::Operation(format!("Failed to fetch messages: {e:?}")))?;
|
||||||
|
|
||||||
// Collect all messages first to avoid borrowing issues
|
// Collect all messages first to avoid borrowing issues
|
||||||
let mut fetched_messages = Vec::new();
|
let mut fetched_messages = Vec::new();
|
||||||
while let Some(message_result) = messages.next().await {
|
while let Some(message_result) = messages.next().await {
|
||||||
match message_result {
|
match message_result {
|
||||||
Ok(message) => fetched_messages.push(message),
|
Ok(message) => fetched_messages.push(message),
|
||||||
Err(e) => log::warn!("Failed to fetch message: {:?}", e),
|
Err(e) => log::warn!("Failed to fetch message: {e:?}"),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -534,7 +596,7 @@ impl ImapClient {
|
||||||
match self.parse_message(message, uid, mailbox).await {
|
match self.parse_message(message, uid, mailbox).await {
|
||||||
Ok((doc, attachments)) => mail_documents.push((doc, attachments)),
|
Ok((doc, attachments)) => mail_documents.push((doc, attachments)),
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
log::warn!("Failed to parse message {}: {}", uid, e);
|
log::warn!("Failed to parse message {uid}: {e}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -544,8 +606,14 @@ impl ImapClient {
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Parse a raw IMAP message into a MailDocument with attachment data
|
/// Parse a raw IMAP message into a MailDocument with attachment data
|
||||||
async fn parse_message(&self, message: &Fetch, uid: u32, mailbox: &str) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
async fn parse_message(
|
||||||
let body = message.body()
|
&self,
|
||||||
|
message: &Fetch,
|
||||||
|
uid: u32,
|
||||||
|
mailbox: &str,
|
||||||
|
) -> Result<(MailDocument, Vec<(String, String, Vec<u8>)>)> {
|
||||||
|
let body = message
|
||||||
|
.body()
|
||||||
.ok_or_else(|| ImapError::Parsing("No message body found".to_string()))?;
|
.ok_or_else(|| ImapError::Parsing("No message body found".to_string()))?;
|
||||||
|
|
||||||
// Parse the email using mail-parser library
|
// Parse the email using mail-parser library
|
||||||
|
|
@ -566,7 +634,7 @@ impl ImapClient {
|
||||||
|
|
||||||
// Extract date
|
// Extract date
|
||||||
let date = if let Some(date_time) = parsed_message.get_date() {
|
let date = if let Some(date_time) = parsed_message.get_date() {
|
||||||
DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(|| Utc::now())
|
DateTime::from_timestamp(date_time.to_timestamp(), 0).unwrap_or_else(Utc::now)
|
||||||
} else {
|
} else {
|
||||||
Utc::now()
|
Utc::now()
|
||||||
};
|
};
|
||||||
|
|
@ -578,7 +646,8 @@ impl ImapClient {
|
||||||
let headers = self.extract_headers(&parsed_message);
|
let headers = self.extract_headers(&parsed_message);
|
||||||
|
|
||||||
// Extract attachments and their data
|
// Extract attachments and their data
|
||||||
let (has_attachments, _attachment_stubs, attachment_data) = self.extract_attachments_with_data(&parsed_message);
|
let (has_attachments, _attachment_stubs, attachment_data) =
|
||||||
|
self.extract_attachments_with_data(&parsed_message);
|
||||||
|
|
||||||
let mail_doc = MailDocument::new(
|
let mail_doc = MailDocument::new(
|
||||||
uid.to_string(),
|
uid.to_string(),
|
||||||
|
|
@ -598,7 +667,11 @@ impl ImapClient {
|
||||||
|
|
||||||
// Log attachment information
|
// Log attachment information
|
||||||
if !attachment_data.is_empty() {
|
if !attachment_data.is_empty() {
|
||||||
log::info!("Found {} attachments for message {}", attachment_data.len(), uid);
|
log::info!(
|
||||||
|
"Found {} attachments for message {}",
|
||||||
|
attachment_data.len(),
|
||||||
|
uid
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
Ok((mail_doc, attachment_data))
|
Ok((mail_doc, attachment_data))
|
||||||
|
|
@ -681,9 +754,7 @@ impl ImapClient {
|
||||||
None => format!("{:?}", header.value()), // Fallback for non-text values
|
None => format!("{:?}", header.value()), // Fallback for non-text values
|
||||||
};
|
};
|
||||||
|
|
||||||
headers.entry(name)
|
headers.entry(name).or_insert_with(Vec::new).push(value);
|
||||||
.or_insert_with(Vec::new)
|
|
||||||
.push(value);
|
|
||||||
}
|
}
|
||||||
|
|
||||||
headers
|
headers
|
||||||
|
|
@ -691,7 +762,15 @@ impl ImapClient {
|
||||||
|
|
||||||
/// Extract attachments from a parsed message with binary data
|
/// Extract attachments from a parsed message with binary data
|
||||||
/// Returns (has_attachments, attachment_stubs, attachment_data)
|
/// Returns (has_attachments, attachment_stubs, attachment_data)
|
||||||
fn extract_attachments_with_data(&self, message: &Message) -> (bool, HashMap<String, AttachmentStub>, Vec<(String, String, Vec<u8>)>) {
|
#[allow(clippy::type_complexity)]
|
||||||
|
fn extract_attachments_with_data(
|
||||||
|
&self,
|
||||||
|
message: &Message,
|
||||||
|
) -> (
|
||||||
|
bool,
|
||||||
|
HashMap<String, AttachmentStub>,
|
||||||
|
Vec<(String, String, Vec<u8>)>,
|
||||||
|
) {
|
||||||
let mut attachment_stubs = HashMap::new();
|
let mut attachment_stubs = HashMap::new();
|
||||||
let mut attachment_data = Vec::new();
|
let mut attachment_data = Vec::new();
|
||||||
|
|
||||||
|
|
@ -699,7 +778,7 @@ impl ImapClient {
|
||||||
for (index, part) in message.parts.iter().enumerate() {
|
for (index, part) in message.parts.iter().enumerate() {
|
||||||
// Check if this part is an attachment
|
// Check if this part is an attachment
|
||||||
if let Some(content_type) = part.get_content_type() {
|
if let Some(content_type) = part.get_content_type() {
|
||||||
let is_attachment = self.is_attachment_part(part, &content_type);
|
let is_attachment = self.is_attachment_part(part, content_type);
|
||||||
|
|
||||||
if is_attachment {
|
if is_attachment {
|
||||||
// Generate a filename for the attachment
|
// Generate a filename for the attachment
|
||||||
|
|
@ -708,7 +787,11 @@ impl ImapClient {
|
||||||
// Get the binary content data using the proper mail-parser API
|
// Get the binary content data using the proper mail-parser API
|
||||||
// This works for both text and binary attachments (images, PDFs, etc.)
|
// This works for both text and binary attachments (images, PDFs, etc.)
|
||||||
let body_data = part.get_contents().to_vec();
|
let body_data = part.get_contents().to_vec();
|
||||||
log::debug!("Found attachment content: {} bytes (content-type: {})", body_data.len(), content_type.c_type);
|
log::debug!(
|
||||||
|
"Found attachment content: {} bytes (content-type: {})",
|
||||||
|
body_data.len(),
|
||||||
|
content_type.c_type
|
||||||
|
);
|
||||||
|
|
||||||
let content_type_str = content_type.c_type.to_string();
|
let content_type_str = content_type.c_type.to_string();
|
||||||
|
|
||||||
|
|
@ -731,9 +814,12 @@ impl ImapClient {
|
||||||
(has_attachments, attachment_stubs, attachment_data)
|
(has_attachments, attachment_stubs, attachment_data)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
/// Determine if a message part is an attachment
|
/// Determine if a message part is an attachment
|
||||||
fn is_attachment_part(&self, part: &mail_parser::MessagePart, content_type: &mail_parser::ContentType) -> bool {
|
fn is_attachment_part(
|
||||||
|
&self,
|
||||||
|
part: &mail_parser::MessagePart,
|
||||||
|
content_type: &mail_parser::ContentType,
|
||||||
|
) -> bool {
|
||||||
// Check Content-Disposition header first
|
// Check Content-Disposition header first
|
||||||
if let Some(disposition) = part.get_content_disposition() {
|
if let Some(disposition) = part.get_content_disposition() {
|
||||||
return disposition.c_type.to_lowercase() == "attachment";
|
return disposition.c_type.to_lowercase() == "attachment";
|
||||||
|
|
@ -743,9 +829,9 @@ impl ImapClient {
|
||||||
// Consider non-text types as potential attachments
|
// Consider non-text types as potential attachments
|
||||||
let main_type = content_type.c_type.split('/').next().unwrap_or("");
|
let main_type = content_type.c_type.split('/').next().unwrap_or("");
|
||||||
match main_type {
|
match main_type {
|
||||||
"text" => false, // Text parts are usually body content
|
"text" => false, // Text parts are usually body content
|
||||||
"multipart" => false, // Multipart containers are not attachments
|
"multipart" => false, // Multipart containers are not attachments
|
||||||
_ => true, // Images, applications, etc. are likely attachments
|
_ => true, // Images, applications, etc. are likely attachments
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -778,9 +864,9 @@ impl ImapClient {
|
||||||
// Generate a default filename based on content type and index
|
// Generate a default filename based on content type and index
|
||||||
if let Some(content_type) = part.get_content_type() {
|
if let Some(content_type) = part.get_content_type() {
|
||||||
let extension = self.get_extension_from_content_type(&content_type.c_type);
|
let extension = self.get_extension_from_content_type(&content_type.c_type);
|
||||||
format!("attachment_{}{}", index, extension)
|
format!("attachment_{index}{extension}")
|
||||||
} else {
|
} else {
|
||||||
format!("attachment_{}.bin", index)
|
format!("attachment_{index}.bin")
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -805,22 +891,23 @@ impl ImapClient {
|
||||||
/// Close the IMAP connection
|
/// Close the IMAP connection
|
||||||
pub async fn close(self) -> Result<()> {
|
pub async fn close(self) -> Result<()> {
|
||||||
if let Some(mut session) = self.session {
|
if let Some(mut session) = self.session {
|
||||||
session.logout().await
|
session
|
||||||
.map_err(|e| ImapError::Operation(format!("Logout failed: {:?}", e)))?;
|
.logout()
|
||||||
|
.await
|
||||||
|
.map_err(|e| ImapError::Operation(format!("Logout failed: {e:?}")))?;
|
||||||
}
|
}
|
||||||
Ok(())
|
Ok(())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/// Apply message filters to determine if a message should be processed
|
/// Apply message filters to determine if a message should be processed
|
||||||
pub fn should_process_message(
|
pub fn should_process_message(mail_doc: &MailDocument, filter: &MessageFilter) -> bool {
|
||||||
mail_doc: &MailDocument,
|
|
||||||
filter: &MessageFilter,
|
|
||||||
) -> bool {
|
|
||||||
// Check subject keywords
|
// Check subject keywords
|
||||||
if !filter.subject_keywords.is_empty() {
|
if !filter.subject_keywords.is_empty() {
|
||||||
let subject_lower = mail_doc.subject.to_lowercase();
|
let subject_lower = mail_doc.subject.to_lowercase();
|
||||||
let has_subject_keyword = filter.subject_keywords.iter()
|
let has_subject_keyword = filter
|
||||||
|
.subject_keywords
|
||||||
|
.iter()
|
||||||
.any(|keyword| subject_lower.contains(&keyword.to_lowercase()));
|
.any(|keyword| subject_lower.contains(&keyword.to_lowercase()));
|
||||||
if !has_subject_keyword {
|
if !has_subject_keyword {
|
||||||
return false;
|
return false;
|
||||||
|
|
@ -829,12 +916,13 @@ pub fn should_process_message(
|
||||||
|
|
||||||
// Check sender keywords
|
// Check sender keywords
|
||||||
if !filter.sender_keywords.is_empty() {
|
if !filter.sender_keywords.is_empty() {
|
||||||
let has_sender_keyword = mail_doc.from.iter()
|
let has_sender_keyword = mail_doc.from.iter().any(|from_addr| {
|
||||||
.any(|from_addr| {
|
let from_lower = from_addr.to_lowercase();
|
||||||
let from_lower = from_addr.to_lowercase();
|
filter
|
||||||
filter.sender_keywords.iter()
|
.sender_keywords
|
||||||
.any(|keyword| from_lower.contains(&keyword.to_lowercase()))
|
.iter()
|
||||||
});
|
.any(|keyword| from_lower.contains(&keyword.to_lowercase()))
|
||||||
|
});
|
||||||
if !has_sender_keyword {
|
if !has_sender_keyword {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
@ -842,12 +930,13 @@ pub fn should_process_message(
|
||||||
|
|
||||||
// Check recipient keywords
|
// Check recipient keywords
|
||||||
if !filter.recipient_keywords.is_empty() {
|
if !filter.recipient_keywords.is_empty() {
|
||||||
let has_recipient_keyword = mail_doc.to.iter()
|
let has_recipient_keyword = mail_doc.to.iter().any(|to_addr| {
|
||||||
.any(|to_addr| {
|
let to_lower = to_addr.to_lowercase();
|
||||||
let to_lower = to_addr.to_lowercase();
|
filter
|
||||||
filter.recipient_keywords.iter()
|
.recipient_keywords
|
||||||
.any(|keyword| to_lower.contains(&keyword.to_lowercase()))
|
.iter()
|
||||||
});
|
.any(|keyword| to_lower.contains(&keyword.to_lowercase()))
|
||||||
|
});
|
||||||
if !has_recipient_keyword {
|
if !has_recipient_keyword {
|
||||||
return false;
|
return false;
|
||||||
}
|
}
|
||||||
|
|
@ -898,7 +987,7 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_rfc822_parsing() {
|
fn test_rfc822_parsing() {
|
||||||
let client = ImapClient {
|
let _client = ImapClient {
|
||||||
session: None,
|
session: None,
|
||||||
source: MailSource {
|
source: MailSource {
|
||||||
name: "test".to_string(),
|
name: "test".to_string(),
|
||||||
|
|
|
||||||
|
|
@ -14,16 +14,16 @@
|
||||||
//! The library uses well-defined CouchDB document schemas that are compatible
|
//! The library uses well-defined CouchDB document schemas that are compatible
|
||||||
//! with the Go implementation. See the `schemas` module for details.
|
//! with the Go implementation. See the `schemas` module for details.
|
||||||
|
|
||||||
pub mod schemas;
|
|
||||||
pub mod config;
|
|
||||||
pub mod cli;
|
pub mod cli;
|
||||||
|
pub mod config;
|
||||||
pub mod couch;
|
pub mod couch;
|
||||||
pub mod imap;
|
|
||||||
pub mod filters;
|
pub mod filters;
|
||||||
|
pub mod imap;
|
||||||
|
pub mod schemas;
|
||||||
pub mod sync;
|
pub mod sync;
|
||||||
|
|
||||||
// Re-export main types for convenience
|
// Re-export main types for convenience
|
||||||
pub use schemas::{MailDocument, SyncMetadata, AttachmentStub, generate_database_name};
|
pub use config::{CommandLineArgs, Config, MailSource};
|
||||||
pub use config::{Config, MailSource, CommandLineArgs};
|
|
||||||
pub use couch::CouchClient;
|
pub use couch::CouchClient;
|
||||||
pub use imap::ImapClient;
|
pub use imap::ImapClient;
|
||||||
|
pub use schemas::{generate_database_name, AttachmentStub, MailDocument, SyncMetadata};
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,7 @@
|
||||||
use anyhow::Result;
|
use anyhow::Result;
|
||||||
use env_logger::Env;
|
use env_logger::Env;
|
||||||
use log::{error, info};
|
use log::{error, info};
|
||||||
use mail2couch::{
|
use mail2couch::{cli::parse_command_line, config::Config, sync::SyncCoordinator};
|
||||||
cli::parse_command_line,
|
|
||||||
config::Config,
|
|
||||||
sync::SyncCoordinator,
|
|
||||||
};
|
|
||||||
use std::process;
|
use std::process;
|
||||||
|
|
||||||
#[tokio::main]
|
#[tokio::main]
|
||||||
|
|
@ -18,7 +14,7 @@ async fn main() {
|
||||||
|
|
||||||
// Run the main application
|
// Run the main application
|
||||||
if let Err(e) = run(args).await {
|
if let Err(e) = run(args).await {
|
||||||
error!("❌ Application failed: {}", e);
|
error!("❌ Application failed: {e}");
|
||||||
process::exit(1);
|
process::exit(1);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -31,7 +27,7 @@ async fn run(args: mail2couch::config::CommandLineArgs) -> Result<()> {
|
||||||
info!("Using configuration file: {}", config_path.display());
|
info!("Using configuration file: {}", config_path.display());
|
||||||
|
|
||||||
if let Some(max) = args.max_messages {
|
if let Some(max) = args.max_messages {
|
||||||
info!("Maximum messages per mailbox: {}", max);
|
info!("Maximum messages per mailbox: {max}");
|
||||||
} else {
|
} else {
|
||||||
info!("Maximum messages per mailbox: unlimited");
|
info!("Maximum messages per mailbox: unlimited");
|
||||||
}
|
}
|
||||||
|
|
@ -67,7 +63,11 @@ fn print_config_summary(config: &mail2couch::config::Config) {
|
||||||
info!(" Mail sources: {}", config.mail_sources.len());
|
info!(" Mail sources: {}", config.mail_sources.len());
|
||||||
|
|
||||||
for (i, source) in config.mail_sources.iter().enumerate() {
|
for (i, source) in config.mail_sources.iter().enumerate() {
|
||||||
let status = if source.enabled { "enabled" } else { "disabled" };
|
let status = if source.enabled {
|
||||||
|
"enabled"
|
||||||
|
} else {
|
||||||
|
"disabled"
|
||||||
|
};
|
||||||
info!(
|
info!(
|
||||||
" {}: {} ({}) - {} ({})",
|
" {}: {} ({}) - {} ({})",
|
||||||
i + 1,
|
i + 1,
|
||||||
|
|
@ -88,7 +88,10 @@ fn print_config_summary(config: &mail2couch::config::Config) {
|
||||||
info!(" Since: {:?}", source.message_filter.since);
|
info!(" Since: {:?}", source.message_filter.since);
|
||||||
}
|
}
|
||||||
if !source.message_filter.subject_keywords.is_empty() {
|
if !source.message_filter.subject_keywords.is_empty() {
|
||||||
info!(" Subject keywords: {:?}", source.message_filter.subject_keywords);
|
info!(
|
||||||
|
" Subject keywords: {:?}",
|
||||||
|
source.message_filter.subject_keywords
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -121,6 +121,7 @@ pub struct SyncMetadata {
|
||||||
|
|
||||||
impl MailDocument {
|
impl MailDocument {
|
||||||
/// Create a new MailDocument with required fields
|
/// Create a new MailDocument with required fields
|
||||||
|
#[allow(clippy::too_many_arguments)]
|
||||||
pub fn new(
|
pub fn new(
|
||||||
source_uid: String,
|
source_uid: String,
|
||||||
mailbox: String,
|
mailbox: String,
|
||||||
|
|
@ -134,7 +135,7 @@ impl MailDocument {
|
||||||
) -> Self {
|
) -> Self {
|
||||||
let now = Utc::now();
|
let now = Utc::now();
|
||||||
Self {
|
Self {
|
||||||
id: None, // Will be set when storing to CouchDB
|
id: None, // Will be set when storing to CouchDB
|
||||||
rev: None, // Managed by CouchDB
|
rev: None, // Managed by CouchDB
|
||||||
attachments: None,
|
attachments: None,
|
||||||
source_uid,
|
source_uid,
|
||||||
|
|
@ -172,7 +173,7 @@ impl SyncMetadata {
|
||||||
) -> Self {
|
) -> Self {
|
||||||
let now = Utc::now();
|
let now = Utc::now();
|
||||||
Self {
|
Self {
|
||||||
id: Some(format!("sync_metadata_{}", mailbox)),
|
id: Some(format!("sync_metadata_{mailbox}")),
|
||||||
rev: None, // Managed by CouchDB
|
rev: None, // Managed by CouchDB
|
||||||
doc_type: "sync_metadata".to_string(),
|
doc_type: "sync_metadata".to_string(),
|
||||||
mailbox,
|
mailbox,
|
||||||
|
|
@ -199,7 +200,15 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String {
|
||||||
.to_lowercase()
|
.to_lowercase()
|
||||||
.chars()
|
.chars()
|
||||||
.map(|c| {
|
.map(|c| {
|
||||||
if c.is_ascii_alphanumeric() || c == '_' || c == '$' || c == '(' || c == ')' || c == '+' || c == '-' || c == '/' {
|
if c.is_ascii_alphanumeric()
|
||||||
|
|| c == '_'
|
||||||
|
|| c == '$'
|
||||||
|
|| c == '('
|
||||||
|
|| c == ')'
|
||||||
|
|| c == '+'
|
||||||
|
|| c == '-'
|
||||||
|
|| c == '/'
|
||||||
|
{
|
||||||
c
|
c
|
||||||
} else {
|
} else {
|
||||||
'_'
|
'_'
|
||||||
|
|
@ -209,9 +218,9 @@ pub fn generate_database_name(account_name: &str, user_email: &str) -> String {
|
||||||
|
|
||||||
// Ensure starts with a letter
|
// Ensure starts with a letter
|
||||||
if valid_name.is_empty() || !valid_name.chars().next().unwrap().is_ascii_lowercase() {
|
if valid_name.is_empty() || !valid_name.chars().next().unwrap().is_ascii_lowercase() {
|
||||||
valid_name = format!("m2c_mail_{}", valid_name);
|
valid_name = format!("m2c_mail_{valid_name}");
|
||||||
} else {
|
} else {
|
||||||
valid_name = format!("m2c_{}", valid_name);
|
valid_name = format!("m2c_{valid_name}");
|
||||||
}
|
}
|
||||||
|
|
||||||
valid_name
|
valid_name
|
||||||
|
|
@ -223,8 +232,14 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_generate_database_name() {
|
fn test_generate_database_name() {
|
||||||
assert_eq!(generate_database_name("Personal Gmail", ""), "m2c_personal_gmail");
|
assert_eq!(
|
||||||
assert_eq!(generate_database_name("", "user@example.com"), "m2c_user_example_com");
|
generate_database_name("Personal Gmail", ""),
|
||||||
|
"m2c_personal_gmail"
|
||||||
|
);
|
||||||
|
assert_eq!(
|
||||||
|
generate_database_name("", "user@example.com"),
|
||||||
|
"m2c_user_example_com"
|
||||||
|
);
|
||||||
assert_eq!(generate_database_name("123work", ""), "m2c_mail_123work");
|
assert_eq!(generate_database_name("123work", ""), "m2c_mail_123work");
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -250,12 +265,7 @@ mod tests {
|
||||||
|
|
||||||
#[test]
|
#[test]
|
||||||
fn test_sync_metadata_creation() {
|
fn test_sync_metadata_creation() {
|
||||||
let metadata = SyncMetadata::new(
|
let metadata = SyncMetadata::new("INBOX".to_string(), Utc::now(), 456, 100);
|
||||||
"INBOX".to_string(),
|
|
||||||
Utc::now(),
|
|
||||||
456,
|
|
||||||
100,
|
|
||||||
);
|
|
||||||
|
|
||||||
assert_eq!(metadata.id, Some("sync_metadata_INBOX".to_string()));
|
assert_eq!(metadata.id, Some("sync_metadata_INBOX".to_string()));
|
||||||
assert_eq!(metadata.doc_type, "sync_metadata");
|
assert_eq!(metadata.doc_type, "sync_metadata");
|
||||||
|
|
|
||||||
240
rust/src/sync.rs
240
rust/src/sync.rs
|
|
@ -3,14 +3,14 @@
|
||||||
//! This module coordinates the synchronization process between IMAP servers and CouchDB,
|
//! This module coordinates the synchronization process between IMAP servers and CouchDB,
|
||||||
//! implementing incremental sync with metadata tracking.
|
//! implementing incremental sync with metadata tracking.
|
||||||
|
|
||||||
use crate::config::{Config, MailSource, CommandLineArgs};
|
use crate::config::{CommandLineArgs, Config, MailSource};
|
||||||
use crate::couch::CouchClient;
|
use crate::couch::CouchClient;
|
||||||
use crate::filters::{get_filter_summary, validate_folder_patterns};
|
use crate::filters::{get_filter_summary, validate_folder_patterns};
|
||||||
use crate::imap::{ImapClient, should_process_message};
|
use crate::imap::{should_process_message, ImapClient};
|
||||||
use crate::schemas::{SyncMetadata, generate_database_name};
|
use crate::schemas::{generate_database_name, SyncMetadata};
|
||||||
use anyhow::{anyhow, Result};
|
use anyhow::{anyhow, Result};
|
||||||
use chrono::{DateTime, Utc};
|
use chrono::{DateTime, Utc};
|
||||||
use log::{info, warn, error, debug};
|
use log::{debug, error, info, warn};
|
||||||
|
|
||||||
/// Main synchronization coordinator
|
/// Main synchronization coordinator
|
||||||
pub struct SyncCoordinator {
|
pub struct SyncCoordinator {
|
||||||
|
|
@ -57,7 +57,9 @@ impl SyncCoordinator {
|
||||||
/// Test connections to all services
|
/// Test connections to all services
|
||||||
pub async fn test_connections(&self) -> Result<()> {
|
pub async fn test_connections(&self) -> Result<()> {
|
||||||
info!("Testing CouchDB connection...");
|
info!("Testing CouchDB connection...");
|
||||||
self.couch_client.test_connection().await
|
self.couch_client
|
||||||
|
.test_connection()
|
||||||
|
.await
|
||||||
.map_err(|e| anyhow!("CouchDB connection failed: {}", e))?;
|
.map_err(|e| anyhow!("CouchDB connection failed: {}", e))?;
|
||||||
info!("✅ CouchDB connection successful");
|
info!("✅ CouchDB connection successful");
|
||||||
|
|
||||||
|
|
@ -68,7 +70,8 @@ impl SyncCoordinator {
|
||||||
}
|
}
|
||||||
|
|
||||||
info!("Testing IMAP connection to {}...", source.name);
|
info!("Testing IMAP connection to {}...", source.name);
|
||||||
let imap_client = ImapClient::connect(source.clone()).await
|
let imap_client = ImapClient::connect(source.clone())
|
||||||
|
.await
|
||||||
.map_err(|e| anyhow!("IMAP connection to {} failed: {}", source.name, e))?;
|
.map_err(|e| anyhow!("IMAP connection to {} failed: {}", source.name, e))?;
|
||||||
|
|
||||||
imap_client.close().await?;
|
imap_client.close().await?;
|
||||||
|
|
@ -95,9 +98,7 @@ impl SyncCoordinator {
|
||||||
Ok(result) => {
|
Ok(result) => {
|
||||||
info!(
|
info!(
|
||||||
"✅ Completed sync for {}: {} messages across {} mailboxes",
|
"✅ Completed sync for {}: {} messages across {} mailboxes",
|
||||||
result.source_name,
|
result.source_name, result.total_messages, result.mailboxes_processed
|
||||||
result.total_messages,
|
|
||||||
result.mailboxes_processed
|
|
||||||
);
|
);
|
||||||
results.push(result);
|
results.push(result);
|
||||||
}
|
}
|
||||||
|
|
@ -117,38 +118,49 @@ impl SyncCoordinator {
|
||||||
|
|
||||||
// Generate database name
|
// Generate database name
|
||||||
let db_name = generate_database_name(&source.name, &source.user);
|
let db_name = generate_database_name(&source.name, &source.user);
|
||||||
info!("Using database: {}", db_name);
|
info!("Using database: {db_name}");
|
||||||
|
|
||||||
// Create database if it doesn't exist (skip in dry-run mode)
|
// Create database if it doesn't exist (skip in dry-run mode)
|
||||||
if !self.args.dry_run {
|
if !self.args.dry_run {
|
||||||
self.couch_client.create_database(&db_name).await?;
|
self.couch_client.create_database(&db_name).await?;
|
||||||
} else {
|
} else {
|
||||||
info!("🔍 DRY-RUN: Would create database {}", db_name);
|
info!("🔍 DRY-RUN: Would create database {db_name}");
|
||||||
}
|
}
|
||||||
|
|
||||||
// Connect to IMAP server
|
// Connect to IMAP server
|
||||||
let mut imap_client = ImapClient::connect(source.clone()).await?;
|
let mut imap_client = ImapClient::connect(source.clone()).await?;
|
||||||
|
|
||||||
// Use IMAP LIST with patterns for server-side filtering
|
// Use IMAP LIST with patterns for server-side filtering
|
||||||
let filtered_mailboxes = imap_client.list_filtered_mailboxes(&source.folder_filter).await?;
|
let filtered_mailboxes = imap_client
|
||||||
info!("Found {} matching mailboxes after server-side filtering", filtered_mailboxes.len());
|
.list_filtered_mailboxes(&source.folder_filter)
|
||||||
|
.await?;
|
||||||
|
info!(
|
||||||
|
"Found {} matching mailboxes after server-side filtering",
|
||||||
|
filtered_mailboxes.len()
|
||||||
|
);
|
||||||
|
|
||||||
// For validation and summary, we still need the full list
|
// For validation and summary, we still need the full list
|
||||||
let all_mailboxes = if !source.folder_filter.include.is_empty() || !source.folder_filter.exclude.is_empty() {
|
let all_mailboxes = if !source.folder_filter.include.is_empty()
|
||||||
|
|| !source.folder_filter.exclude.is_empty()
|
||||||
|
{
|
||||||
// Only fetch all mailboxes if we have filters (for logging/validation)
|
// Only fetch all mailboxes if we have filters (for logging/validation)
|
||||||
imap_client.list_mailboxes().await.unwrap_or_else(|_| Vec::new())
|
imap_client
|
||||||
|
.list_mailboxes()
|
||||||
|
.await
|
||||||
|
.unwrap_or_else(|_| Vec::new())
|
||||||
} else {
|
} else {
|
||||||
filtered_mailboxes.clone()
|
filtered_mailboxes.clone()
|
||||||
};
|
};
|
||||||
|
|
||||||
if !all_mailboxes.is_empty() {
|
if !all_mailboxes.is_empty() {
|
||||||
let filter_summary = get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter);
|
let filter_summary =
|
||||||
info!("{}", filter_summary);
|
get_filter_summary(&all_mailboxes, &filtered_mailboxes, &source.folder_filter);
|
||||||
|
info!("{filter_summary}");
|
||||||
|
|
||||||
// Validate folder patterns and show warnings
|
// Validate folder patterns and show warnings
|
||||||
let warnings = validate_folder_patterns(&source.folder_filter, &all_mailboxes);
|
let warnings = validate_folder_patterns(&source.folder_filter, &all_mailboxes);
|
||||||
for warning in warnings {
|
for warning in warnings {
|
||||||
warn!("{}", warning);
|
warn!("{warning}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -157,9 +169,12 @@ impl SyncCoordinator {
|
||||||
let mut total_messages = 0;
|
let mut total_messages = 0;
|
||||||
|
|
||||||
for mailbox in &filtered_mailboxes {
|
for mailbox in &filtered_mailboxes {
|
||||||
info!("Syncing mailbox: {}", mailbox);
|
info!("Syncing mailbox: {mailbox}");
|
||||||
|
|
||||||
match self.sync_mailbox(&mut imap_client, &db_name, mailbox, source).await {
|
match self
|
||||||
|
.sync_mailbox(&mut imap_client, &db_name, mailbox, source)
|
||||||
|
.await
|
||||||
|
{
|
||||||
Ok(result) => {
|
Ok(result) => {
|
||||||
if result.messages_deleted > 0 {
|
if result.messages_deleted > 0 {
|
||||||
info!(
|
info!(
|
||||||
|
|
@ -183,7 +198,7 @@ impl SyncCoordinator {
|
||||||
mailbox_results.push(result);
|
mailbox_results.push(result);
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
error!(" ❌ Failed to sync mailbox {}: {}", mailbox, e);
|
error!(" ❌ Failed to sync mailbox {mailbox}: {e}");
|
||||||
// Continue with other mailboxes
|
// Continue with other mailboxes
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -214,35 +229,45 @@ impl SyncCoordinator {
|
||||||
|
|
||||||
// Select the mailbox
|
// Select the mailbox
|
||||||
let mailbox_info = imap_client.select_mailbox(mailbox).await?;
|
let mailbox_info = imap_client.select_mailbox(mailbox).await?;
|
||||||
debug!("Selected mailbox {}: {} messages", mailbox, mailbox_info.exists);
|
debug!(
|
||||||
|
"Selected mailbox {}: {} messages",
|
||||||
|
mailbox, mailbox_info.exists
|
||||||
|
);
|
||||||
|
|
||||||
// Get last sync metadata (skip in dry-run mode)
|
// Get last sync metadata (skip in dry-run mode)
|
||||||
let since_date = if !self.args.dry_run {
|
let since_date = if !self.args.dry_run {
|
||||||
match self.couch_client.get_sync_metadata(db_name, mailbox).await {
|
match self.couch_client.get_sync_metadata(db_name, mailbox).await {
|
||||||
Ok(metadata) => {
|
Ok(metadata) => {
|
||||||
info!(" Found sync metadata, last sync: {}", metadata.last_sync_time);
|
info!(
|
||||||
|
" Found sync metadata, last sync: {}",
|
||||||
|
metadata.last_sync_time
|
||||||
|
);
|
||||||
Some(metadata.last_sync_time)
|
Some(metadata.last_sync_time)
|
||||||
}
|
}
|
||||||
Err(_) => {
|
Err(_) => {
|
||||||
info!(" No sync metadata found, performing full sync");
|
info!(" No sync metadata found, performing full sync");
|
||||||
// Parse since date from message filter if provided
|
// Parse since date from message filter if provided
|
||||||
source.message_filter.since.as_ref()
|
source.message_filter.since.as_ref().and_then(|since_str| {
|
||||||
.and_then(|since_str| {
|
DateTime::parse_from_str(
|
||||||
DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z")
|
&format!("{since_str} 00:00:00 +0000"),
|
||||||
.map(|dt| dt.with_timezone(&Utc))
|
"%Y-%m-%d %H:%M:%S %z",
|
||||||
.ok()
|
)
|
||||||
})
|
.map(|dt| dt.with_timezone(&Utc))
|
||||||
|
.ok()
|
||||||
|
})
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
info!(" 🔍 DRY-RUN: Would check for sync metadata");
|
info!(" 🔍 DRY-RUN: Would check for sync metadata");
|
||||||
// In dry-run mode, use config since date if available
|
// In dry-run mode, use config since date if available
|
||||||
source.message_filter.since.as_ref()
|
source.message_filter.since.as_ref().and_then(|since_str| {
|
||||||
.and_then(|since_str| {
|
DateTime::parse_from_str(
|
||||||
DateTime::parse_from_str(&format!("{} 00:00:00 +0000", since_str), "%Y-%m-%d %H:%M:%S %z")
|
&format!("{since_str} 00:00:00 +0000"),
|
||||||
.map(|dt| dt.with_timezone(&Utc))
|
"%Y-%m-%d %H:%M:%S %z",
|
||||||
.ok()
|
)
|
||||||
})
|
.map(|dt| dt.with_timezone(&Utc))
|
||||||
|
.ok()
|
||||||
|
})
|
||||||
};
|
};
|
||||||
|
|
||||||
// Search for messages using server-side IMAP SEARCH with keyword filtering when possible
|
// Search for messages using server-side IMAP SEARCH with keyword filtering when possible
|
||||||
|
|
@ -260,30 +285,39 @@ impl SyncCoordinator {
|
||||||
};
|
};
|
||||||
|
|
||||||
info!(" Using IMAP SEARCH with keyword filters");
|
info!(" Using IMAP SEARCH with keyword filters");
|
||||||
imap_client.search_messages_advanced(
|
imap_client
|
||||||
since_date.as_ref(),
|
.search_messages_advanced(
|
||||||
None, // before_date
|
since_date.as_ref(),
|
||||||
subject_keywords,
|
None, // before_date
|
||||||
from_keywords,
|
subject_keywords,
|
||||||
).await?
|
from_keywords,
|
||||||
|
)
|
||||||
|
.await?
|
||||||
} else {
|
} else {
|
||||||
// Use simple date-based search
|
// Use simple date-based search
|
||||||
imap_client.search_messages(since_date.as_ref()).await?
|
imap_client.search_messages(since_date.as_ref()).await?
|
||||||
};
|
};
|
||||||
info!(" Found {} messages matching search criteria", message_uids.len());
|
info!(
|
||||||
|
" Found {} messages matching search criteria",
|
||||||
|
message_uids.len()
|
||||||
|
);
|
||||||
|
|
||||||
// Handle sync mode - check for deleted messages (skip in dry-run mode)
|
// Handle sync mode - check for deleted messages (skip in dry-run mode)
|
||||||
let mut messages_deleted = 0;
|
let mut messages_deleted = 0;
|
||||||
if source.mode == "sync" {
|
if source.mode == "sync" {
|
||||||
if !self.args.dry_run {
|
if !self.args.dry_run {
|
||||||
messages_deleted = self.handle_deleted_messages(db_name, mailbox, &message_uids).await
|
messages_deleted = self
|
||||||
|
.handle_deleted_messages(db_name, mailbox, &message_uids)
|
||||||
|
.await
|
||||||
.unwrap_or_else(|e| {
|
.unwrap_or_else(|e| {
|
||||||
warn!(" Failed to handle deleted messages: {}", e);
|
warn!(" Failed to handle deleted messages: {e}");
|
||||||
0
|
0
|
||||||
});
|
});
|
||||||
|
|
||||||
if messages_deleted > 0 {
|
if messages_deleted > 0 {
|
||||||
info!(" 🗑️ Deleted {} messages that no longer exist on server", messages_deleted);
|
info!(
|
||||||
|
" 🗑️ Deleted {messages_deleted} messages that no longer exist on server"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
info!(" 🔍 DRY-RUN: Would check for deleted messages in sync mode");
|
info!(" 🔍 DRY-RUN: Would check for deleted messages in sync mode");
|
||||||
|
|
@ -305,7 +339,7 @@ impl SyncCoordinator {
|
||||||
// Apply max message limit if specified
|
// Apply max message limit if specified
|
||||||
let uids_to_process = if let Some(max) = self.args.max_messages {
|
let uids_to_process = if let Some(max) = self.args.max_messages {
|
||||||
if message_uids.len() > max as usize {
|
if message_uids.len() > max as usize {
|
||||||
info!(" Limiting to {} messages due to --max-messages flag", max);
|
info!(" Limiting to {max} messages due to --max-messages flag");
|
||||||
&message_uids[..max as usize]
|
&message_uids[..max as usize]
|
||||||
} else {
|
} else {
|
||||||
&message_uids
|
&message_uids
|
||||||
|
|
@ -315,7 +349,9 @@ impl SyncCoordinator {
|
||||||
};
|
};
|
||||||
|
|
||||||
// Fetch and process messages
|
// Fetch and process messages
|
||||||
let messages = imap_client.fetch_messages(uids_to_process, self.args.max_messages, mailbox).await?;
|
let messages = imap_client
|
||||||
|
.fetch_messages(uids_to_process, self.args.max_messages, mailbox)
|
||||||
|
.await?;
|
||||||
|
|
||||||
let mut messages_stored = 0;
|
let mut messages_stored = 0;
|
||||||
let mut messages_skipped = 0;
|
let mut messages_skipped = 0;
|
||||||
|
|
@ -333,25 +369,33 @@ impl SyncCoordinator {
|
||||||
|
|
||||||
// Store the message document first (skip in dry-run mode)
|
// Store the message document first (skip in dry-run mode)
|
||||||
if !self.args.dry_run {
|
if !self.args.dry_run {
|
||||||
match self.couch_client.store_mail_document(db_name, mail_doc).await {
|
match self
|
||||||
|
.couch_client
|
||||||
|
.store_mail_document(db_name, mail_doc)
|
||||||
|
.await
|
||||||
|
{
|
||||||
Ok(doc_id) => {
|
Ok(doc_id) => {
|
||||||
messages_stored += 1;
|
messages_stored += 1;
|
||||||
|
|
||||||
// Store attachments if any exist
|
// Store attachments if any exist
|
||||||
if !attachments.is_empty() {
|
if !attachments.is_empty() {
|
||||||
for (filename, content_type, data) in attachments {
|
for (filename, content_type, data) in attachments {
|
||||||
match self.couch_client.store_attachment(
|
match self
|
||||||
db_name,
|
.couch_client
|
||||||
&doc_id,
|
.store_attachment(
|
||||||
&filename,
|
db_name,
|
||||||
&content_type,
|
&doc_id,
|
||||||
&data,
|
&filename,
|
||||||
).await {
|
&content_type,
|
||||||
|
&data,
|
||||||
|
)
|
||||||
|
.await
|
||||||
|
{
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
debug!(" Stored attachment: {}", filename);
|
debug!(" Stored attachment: {filename}");
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
warn!(" Failed to store attachment {}: {}", filename, e);
|
warn!(" Failed to store attachment {filename}: {e}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -363,18 +407,23 @@ impl SyncCoordinator {
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
warn!(" Failed to store message {}: {}", uid_str, e);
|
warn!(" Failed to store message {uid_str}: {e}");
|
||||||
messages_skipped += 1;
|
messages_skipped += 1;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
// In dry-run mode, simulate successful storage
|
// In dry-run mode, simulate successful storage
|
||||||
messages_stored += 1;
|
messages_stored += 1;
|
||||||
debug!(" 🔍 DRY-RUN: Would store message {} (Subject: {})",
|
debug!(
|
||||||
uid_str, mail_doc.subject);
|
" 🔍 DRY-RUN: Would store message {} (Subject: {})",
|
||||||
|
uid_str, mail_doc.subject
|
||||||
|
);
|
||||||
|
|
||||||
if !attachments.is_empty() {
|
if !attachments.is_empty() {
|
||||||
debug!(" 🔍 DRY-RUN: Would store {} attachments", attachments.len());
|
debug!(
|
||||||
|
" 🔍 DRY-RUN: Would store {} attachments",
|
||||||
|
attachments.len()
|
||||||
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Parse UID from source_uid
|
// Parse UID from source_uid
|
||||||
|
|
@ -387,18 +436,20 @@ impl SyncCoordinator {
|
||||||
// Update sync metadata (skip in dry-run mode)
|
// Update sync metadata (skip in dry-run mode)
|
||||||
if let Some(uid) = last_uid {
|
if let Some(uid) = last_uid {
|
||||||
if !self.args.dry_run {
|
if !self.args.dry_run {
|
||||||
let sync_metadata = SyncMetadata::new(
|
let sync_metadata =
|
||||||
mailbox.to_string(),
|
SyncMetadata::new(mailbox.to_string(), start_time, uid, messages_stored);
|
||||||
start_time,
|
|
||||||
uid,
|
|
||||||
messages_stored,
|
|
||||||
);
|
|
||||||
|
|
||||||
if let Err(e) = self.couch_client.store_sync_metadata(db_name, &sync_metadata).await {
|
if let Err(e) = self
|
||||||
warn!(" Failed to store sync metadata: {}", e);
|
.couch_client
|
||||||
|
.store_sync_metadata(db_name, &sync_metadata)
|
||||||
|
.await
|
||||||
|
{
|
||||||
|
warn!(" Failed to store sync metadata: {e}");
|
||||||
}
|
}
|
||||||
} else {
|
} else {
|
||||||
info!(" 🔍 DRY-RUN: Would update sync metadata (last UID: {}, {} messages)", uid, messages_stored);
|
info!(
|
||||||
|
" 🔍 DRY-RUN: Would update sync metadata (last UID: {uid}, {messages_stored} messages)"
|
||||||
|
);
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
@ -430,21 +481,22 @@ impl SyncCoordinator {
|
||||||
}
|
}
|
||||||
|
|
||||||
// Find UIDs that exist in CouchDB but not on the server
|
// Find UIDs that exist in CouchDB but not on the server
|
||||||
let server_uid_set: std::collections::HashSet<u32> = current_server_uids.iter().cloned().collect();
|
let server_uid_set: std::collections::HashSet<u32> =
|
||||||
|
current_server_uids.iter().cloned().collect();
|
||||||
let mut deleted_count = 0;
|
let mut deleted_count = 0;
|
||||||
|
|
||||||
for stored_uid in stored_uids {
|
for stored_uid in stored_uids {
|
||||||
if !server_uid_set.contains(&stored_uid) {
|
if !server_uid_set.contains(&stored_uid) {
|
||||||
// This message was deleted from the server, remove it from CouchDB
|
// This message was deleted from the server, remove it from CouchDB
|
||||||
let doc_id = format!("{}_{}", mailbox, stored_uid);
|
let doc_id = format!("{mailbox}_{stored_uid}");
|
||||||
|
|
||||||
match self.couch_client.delete_document(db_name, &doc_id).await {
|
match self.couch_client.delete_document(db_name, &doc_id).await {
|
||||||
Ok(_) => {
|
Ok(_) => {
|
||||||
debug!(" Deleted document: {}", doc_id);
|
debug!(" Deleted document: {doc_id}");
|
||||||
deleted_count += 1;
|
deleted_count += 1;
|
||||||
}
|
}
|
||||||
Err(e) => {
|
Err(e) => {
|
||||||
warn!(" Failed to delete document {}: {}", doc_id, e);
|
warn!(" Failed to delete document {doc_id}: {e}");
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
@ -499,18 +551,16 @@ impl SyncCoordinator {
|
||||||
info!("{}", "=".repeat(50));
|
info!("{}", "=".repeat(50));
|
||||||
if self.args.dry_run {
|
if self.args.dry_run {
|
||||||
info!(
|
info!(
|
||||||
"📊 DRY-RUN Total: {} sources, {} mailboxes, {} messages found",
|
"📊 DRY-RUN Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages found"
|
||||||
total_sources, total_mailboxes, total_messages
|
|
||||||
);
|
);
|
||||||
} else {
|
} else {
|
||||||
info!(
|
info!(
|
||||||
"📊 Total: {} sources, {} mailboxes, {} messages",
|
"📊 Total: {total_sources} sources, {total_mailboxes} mailboxes, {total_messages} messages"
|
||||||
total_sources, total_mailboxes, total_messages
|
|
||||||
);
|
);
|
||||||
}
|
}
|
||||||
|
|
||||||
if let Some(max) = self.args.max_messages {
|
if let Some(max) = self.args.max_messages {
|
||||||
info!("⚠️ Message limit was applied: {} per mailbox", max);
|
info!("⚠️ Message limit was applied: {max} per mailbox");
|
||||||
}
|
}
|
||||||
|
|
||||||
if self.args.dry_run {
|
if self.args.dry_run {
|
||||||
|
|
@ -531,23 +581,21 @@ mod tests {
|
||||||
user: "admin".to_string(),
|
user: "admin".to_string(),
|
||||||
password: "password".to_string(),
|
password: "password".to_string(),
|
||||||
},
|
},
|
||||||
mail_sources: vec![
|
mail_sources: vec![MailSource {
|
||||||
MailSource {
|
name: "Test Account".to_string(),
|
||||||
name: "Test Account".to_string(),
|
enabled: true,
|
||||||
enabled: true,
|
protocol: "imap".to_string(),
|
||||||
protocol: "imap".to_string(),
|
host: "localhost".to_string(),
|
||||||
host: "localhost".to_string(),
|
port: 3143,
|
||||||
port: 3143,
|
user: "testuser".to_string(),
|
||||||
user: "testuser".to_string(),
|
password: "testpass".to_string(),
|
||||||
password: "testpass".to_string(),
|
mode: "archive".to_string(),
|
||||||
mode: "archive".to_string(),
|
folder_filter: FolderFilter {
|
||||||
folder_filter: FolderFilter {
|
include: vec!["*".to_string()],
|
||||||
include: vec!["*".to_string()],
|
exclude: vec!["Trash".to_string()],
|
||||||
exclude: vec!["Trash".to_string()],
|
},
|
||||||
},
|
message_filter: MessageFilter::default(),
|
||||||
message_filter: MessageFilter::default(),
|
}],
|
||||||
}
|
|
||||||
],
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -73,13 +73,13 @@ check_results() {
|
||||||
echo -e "${BLUE}🔍 Checking results...${NC}"
|
echo -e "${BLUE}🔍 Checking results...${NC}"
|
||||||
|
|
||||||
echo -e "${BLUE} Listing all databases:${NC}"
|
echo -e "${BLUE} Listing all databases:${NC}"
|
||||||
curl -s http://localhost:5984/_all_dbs | python3 -m json.tool
|
curl -s -u admin:password http://localhost:5984/_all_dbs | python3 -m json.tool
|
||||||
|
|
||||||
echo -e "\n${BLUE} Go implementation databases:${NC}"
|
echo -e "\n${BLUE} Go implementation databases:${NC}"
|
||||||
for db in go_wildcard_all_folders_test go_work_pattern_test go_specific_folders_only; do
|
for db in go_wildcard_all_folders_test go_work_pattern_test go_specific_folders_only; do
|
||||||
db_name="m2c_${db}"
|
db_name="m2c_${db}"
|
||||||
if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||||
doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||||
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
||||||
else
|
else
|
||||||
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
||||||
|
|
@ -89,8 +89,8 @@ check_results() {
|
||||||
echo -e "\n${BLUE} Rust implementation databases:${NC}"
|
echo -e "\n${BLUE} Rust implementation databases:${NC}"
|
||||||
for db in rust_wildcard_all_folders_test rust_work_pattern_test rust_specific_folders_only; do
|
for db in rust_wildcard_all_folders_test rust_work_pattern_test rust_specific_folders_only; do
|
||||||
db_name="m2c_${db}"
|
db_name="m2c_${db}"
|
||||||
if curl -s "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
if curl -s -u admin:password "http://localhost:5984/${db_name}" >/dev/null 2>&1; then
|
||||||
doc_count=$(curl -s "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
doc_count=$(curl -s -u admin:password "http://localhost:5984/${db_name}" | python3 -c "import sys, json; print(json.load(sys.stdin).get('doc_count', 0))")
|
||||||
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
echo -e "${GREEN} ✅ ${db_name}: ${doc_count} documents${NC}"
|
||||||
else
|
else
|
||||||
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
echo -e "${RED} ❌ ${db_name}: not found${NC}"
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue