Compare commits
3 commits
651d95e98b
...
5a125ba410
| Author | SHA1 | Date | |
|---|---|---|---|
| 5a125ba410 | |||
| 18e1350006 | |||
| 031dd86b0d |
10 changed files with 93 additions and 33 deletions
29
ANALYSIS.md
Normal file
29
ANALYSIS.md
Normal file
|
|
@ -0,0 +1,29 @@
|
|||
### Final Project Analysis
|
||||
|
||||
**What it does:**
|
||||
`mail2couch` is a Go-based command-line tool that archives emails from IMAP servers into a CouchDB database. It performs efficient incremental syncs, allows for flexible filtering of folders and messages, and stores attachments natively within CouchDB. Each mail account is stored in a separate database for clear organization.
|
||||
|
||||
**How it does it:**
|
||||
The application is written in Go and uses a `config.json` file to manage CouchDB and IMAP credentials. It leverages the `go-imap` library for IMAP communication and `kivik` for interacting with CouchDB. It maintains a `sync_metadata` document in each CouchDB database to track the last sync time, enabling it to only fetch new messages on subsequent runs. A `sync` mode is available to keep the archive as a 1-to-1 mirror of the server, but the default `archive` mode preserves all fetched mail. The project also includes a comprehensive test suite using Podman to validate its core features.
|
||||
|
||||
**Problems, Missing Features, and Suggested Improvements:**
|
||||
|
||||
* **Primary Weakness: Security:** The application requires storing IMAP and CouchDB passwords in plain text within the `config.json` file. This is a significant security risk.
|
||||
* **Suggestion:** Prioritize adding support for reading secrets from environment variables (e.g., `M2C_COUCH_PASSWORD`) or integrating with a secrets management tool. For services like Gmail, implementing OAuth2 would be a more secure and modern authentication method than app passwords.
|
||||
|
||||
* **Incomplete Rust Implementation:** The `rust/` directory contains a non-functional placeholder for a Rust version of the tool. This could be confusing for contributors.
|
||||
* **Suggestion:** The `README.md` should explicitly state that the Rust implementation is aspirational and not functional. Alternatively, if there are no plans to develop it, it could be removed to avoid confusion.
|
||||
|
||||
* **Missing Core Feature: Web Interface:** The `README.md` heavily promotes a future web interface for viewing the archived emails, which is a key feature for making the archive useful. This feature is not yet implemented.
|
||||
* **Suggestion:** This should be the highest priority for new feature development. The existing CouchDB schema is well-suited for this, and implementing CouchDB design documents with list and show functions would be the next logical step.
|
||||
|
||||
* **Performance for Large-Scale Use:** The application processes accounts and mailboxes sequentially.
|
||||
* **Suggestion:** For users with many accounts or mailboxes, performance could be significantly improved by introducing concurrency. Using Go's concurrency features (goroutines and channels) to process multiple mailboxes or even multiple accounts in parallel would be a valuable enhancement.
|
||||
|
||||
* **Inefficient Keyword Filtering:** Message filtering by keywords (subject, sender, etc.) is done client-side *after* downloading the messages.
|
||||
* **Suggestion:** Modify the IMAP fetching logic to use server-side `IMAP SEARCH` with keyword criteria. This would reduce bandwidth and processing time, especially for users who only need to archive a small subset of messages from a large mailbox.
|
||||
|
||||
|
||||
|
||||
* **Usability Enhancement: Dry-Run Mode:** Users have no way to test their configuration (especially folder and message filters) without performing a live sync.
|
||||
* **Suggestion:** Implement a `--dry-run` flag that would log which mailboxes and messages *would* be processed and stored, without actually writing any data to CouchDB.
|
||||
10
CLAUDE.md
10
CLAUDE.md
|
|
@ -20,13 +20,13 @@ cd go && go build -o mail2couch .
|
|||
cd go && ./mail2couch
|
||||
|
||||
# Run with specific config file
|
||||
cd go && ./mail2couch -config /path/to/config.json
|
||||
cd go && ./mail2couch --config /path/to/config.json
|
||||
|
||||
# Run with message limit (useful for large mailboxes)
|
||||
cd go && ./mail2couch -max-messages 100
|
||||
cd go && ./mail2couch --max-messages 100
|
||||
|
||||
# Run with both config and message limit
|
||||
cd go && ./mail2couch -config /path/to/config.json -max-messages 50
|
||||
cd go && ./mail2couch --config /path/to/config.json --max-messages 50
|
||||
|
||||
# Run linting/static analysis
|
||||
cd go && go vet ./...
|
||||
|
|
@ -80,7 +80,7 @@ The application uses `config.json` for configuration with the following structur
|
|||
### Configuration File Discovery
|
||||
|
||||
The application automatically searches for configuration files in the following order:
|
||||
1. Path specified by `-config` command line flag
|
||||
1. Path specified by `--config`/`-c` command line flag
|
||||
2. `./config.json` (current working directory)
|
||||
3. `./config/config.json` (config subdirectory)
|
||||
4. `~/.config/mail2couch/config.json` (user XDG config directory)
|
||||
|
|
@ -104,7 +104,7 @@ This design ensures the same `config.json` format will work for both Go and Rust
|
|||
- ✅ Sync vs Archive mode implementation
|
||||
- ✅ CouchDB attachment storage for email attachments
|
||||
- ✅ Full message body and attachment handling with MIME multipart support
|
||||
- ✅ Command line argument support (--max-messages flag)
|
||||
- ✅ Command line argument support (GNU-style --max-messages/-m and --config/-c flags)
|
||||
- ✅ Per-account CouchDB databases for better organization
|
||||
- ✅ Incremental sync functionality with IMAP SEARCH and sync metadata tracking
|
||||
- ❌ Rust implementation
|
||||
|
|
|
|||
20
LICENSE
20
LICENSE
|
|
@ -1,9 +1,21 @@
|
|||
MIT License
|
||||
|
||||
Copyright (c) 2025 olemd
|
||||
Copyright (c) 2025 Ole-Morten Duesund <olemd@glemt.net>
|
||||
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
|
||||
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||
of this software and associated documentation files (the "Software"), to deal
|
||||
in the Software without restriction, including without limitation the rights
|
||||
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the Software is
|
||||
furnished to do so, subject to the following conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
|
||||
The above copyright notice and this permission notice shall be included in all
|
||||
copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||
SOFTWARE.
|
||||
15
README.md
15
README.md
|
|
@ -39,7 +39,7 @@ A powerful email backup utility that synchronizes mail from IMAP accounts to Cou
|
|||
|
||||
### Operational Features
|
||||
- **Automatic Config Discovery**: Finds configuration files in standard locations
|
||||
- **Command Line Control**: Override settings with `--max-messages` and `--config` flags
|
||||
- **Command Line Control**: GNU-style options with `--max-messages`/`-m` and `--config`/`-c` flags
|
||||
- **Comprehensive Logging**: Detailed output for monitoring and troubleshooting
|
||||
- **Error Resilience**: Graceful handling of network issues and server problems
|
||||
|
||||
|
|
@ -105,7 +105,7 @@ The application will:
|
|||
### Configuration File Discovery
|
||||
|
||||
mail2couch automatically searches for configuration files in this order:
|
||||
1. Path specified by `--config` flag
|
||||
1. Path specified by `--config`/`-c` flag
|
||||
2. `./config.json` (current directory)
|
||||
3. `./config/config.json` (config subdirectory)
|
||||
4. `~/.config/mail2couch/config.json` (user config directory)
|
||||
|
|
@ -117,8 +117,9 @@ mail2couch automatically searches for configuration files in this order:
|
|||
./mail2couch [options]
|
||||
|
||||
Options:
|
||||
--config PATH Specify configuration file path
|
||||
--max-messages N Limit messages processed per mailbox per run (0 = unlimited)
|
||||
-c, --config FILE Path to configuration file
|
||||
-m, --max-messages N Limit messages processed per mailbox per run (0 = unlimited)
|
||||
-h, --help Show help message
|
||||
```
|
||||
|
||||
### Folder Pattern Examples
|
||||
|
|
@ -382,7 +383,7 @@ Complex setup with multiple accounts, filtering, and different sync modes:
|
|||
- Schedule regular backups of CouchDB databases
|
||||
|
||||
### Performance Tuning
|
||||
- Use `--max-messages` to limit processing load
|
||||
- Use `--max-messages`/`-m` to limit processing load
|
||||
- Run during off-peak hours for large initial syncs
|
||||
- Monitor IMAP server rate limits and connection limits
|
||||
- Consider running multiple instances for different accounts
|
||||
|
|
@ -408,7 +409,7 @@ Complex setup with multiple accounts, filtering, and different sync modes:
|
|||
|
||||
**Performance Problems**:
|
||||
- Use date filtering (`since`) for large mailboxes
|
||||
- Implement `--max-messages` limits for initial syncs
|
||||
- Implement `--max-messages`/`-m` limits for initial syncs
|
||||
- Monitor server-side rate limiting
|
||||
|
||||
For detailed troubleshooting, see the [test environment documentation](test/README.md).
|
||||
|
|
@ -443,4 +444,4 @@ This project welcomes contributions! Please see [CLAUDE.md](CLAUDE.md) for devel
|
|||
|
||||
## License
|
||||
|
||||
[License information to be added]
|
||||
This project is licensed under the MIT License. See the [LICENSE](LICENSE) file for details.
|
||||
|
|
|
|||
|
|
@ -2,10 +2,11 @@ package config
|
|||
|
||||
import (
|
||||
"encoding/json"
|
||||
"flag"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/spf13/pflag"
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
|
|
@ -87,23 +88,37 @@ type CommandLineArgs struct {
|
|||
MaxMessages int
|
||||
}
|
||||
|
||||
// ParseCommandLine parses command line arguments
|
||||
// ParseCommandLine parses command line arguments using GNU-style options
|
||||
func ParseCommandLine() *CommandLineArgs {
|
||||
configFlag := flag.String("config", "", "Path to configuration file")
|
||||
maxMessagesFlag := flag.Int("max-messages", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)")
|
||||
flag.Parse()
|
||||
var args CommandLineArgs
|
||||
|
||||
return &CommandLineArgs{
|
||||
ConfigPath: *configFlag,
|
||||
MaxMessages: *maxMessagesFlag,
|
||||
// Define long options with -- and short options with -
|
||||
pflag.StringVarP(&args.ConfigPath, "config", "c", "", "Path to configuration file")
|
||||
pflag.IntVarP(&args.MaxMessages, "max-messages", "m", 0, "Maximum number of messages to process per mailbox per run (0 = no limit)")
|
||||
|
||||
// Add help option
|
||||
pflag.BoolP("help", "h", false, "Show help message")
|
||||
|
||||
pflag.Parse()
|
||||
|
||||
// Handle help flag
|
||||
if help, _ := pflag.CommandLine.GetBool("help"); help {
|
||||
fmt.Fprintf(os.Stderr, "mail2couch - Email backup utility for CouchDB\n\n")
|
||||
fmt.Fprintf(os.Stderr, "Usage: %s [OPTIONS]\n\n", os.Args[0])
|
||||
fmt.Fprintf(os.Stderr, "Options:\n")
|
||||
pflag.PrintDefaults()
|
||||
os.Exit(0)
|
||||
}
|
||||
|
||||
return &args
|
||||
}
|
||||
|
||||
// FindConfigFile searches for config.json in the following order:
|
||||
// 1. Path specified by -config flag
|
||||
// 1. Path specified by --config/-c flag
|
||||
// 2. ./config.json (current directory)
|
||||
// 3. ~/.config/mail2couch/config.json (user config directory)
|
||||
// 4. ~/.mail2couch.json (user home directory)
|
||||
// 3. ./config/config.json (config subdirectory)
|
||||
// 4. ~/.config/mail2couch/config.json (user config directory)
|
||||
// 5. ~/.mail2couch.json (user home directory)
|
||||
func FindConfigFile(args *CommandLineArgs) (string, error) {
|
||||
if args.ConfigPath != "" {
|
||||
if _, err := os.Stat(args.ConfigPath); err == nil {
|
||||
|
|
|
|||
|
|
@ -6,6 +6,7 @@ require (
|
|||
github.com/emersion/go-imap/v2 v2.0.0-beta.5
|
||||
github.com/emersion/go-message v0.18.1
|
||||
github.com/go-kivik/kivik/v4 v4.4.0
|
||||
github.com/spf13/pflag v1.0.7
|
||||
)
|
||||
|
||||
require (
|
||||
|
|
|
|||
|
|
@ -24,6 +24,8 @@ github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
|||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2 h1:Jamvg5psRIccs7FGNTlIRMkT8wgtp5eCXdBlqhYGL6U=
|
||||
github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/spf13/pflag v1.0.7 h1:vN6T9TfwStFPFM5XzjsvmzZkLuaLX+HS+0SeFLRgU6M=
|
||||
github.com/spf13/pflag v1.0.7/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/yuin/goldmark v1.4.13/go.mod h1:6yULJ656Px+3vBD8DxQVa3kxgyrAnzto9xy5taEt/CY=
|
||||
gitlab.com/flimzy/testy v0.14.0 h1:2nZV4Wa1OSJb3rOKHh0GJqvvhtE03zT+sKnPCI0owfQ=
|
||||
gitlab.com/flimzy/testy v0.14.0/go.mod h1:m3aGuwdXc+N3QgnH+2Ar2zf1yg0UxNdIaXKvC5SlfMk=
|
||||
|
|
|
|||
|
|
@ -83,7 +83,7 @@ cd ../test
|
|||
|
||||
# Run mail2couch with test configuration
|
||||
print_status "Running mail2couch with test configuration..."
|
||||
../go/mail2couch -config config-test.json -max-messages 3
|
||||
../go/mail2couch --config config-test.json --max-messages 3
|
||||
|
||||
# Verify results
|
||||
print_status "Verifying test results..."
|
||||
|
|
@ -112,7 +112,7 @@ done
|
|||
|
||||
# Test sync mode by running again (should show incremental behavior)
|
||||
print_status "Running mail2couch again to test incremental sync..."
|
||||
../go/mail2couch -config config-test.json -max-messages 3
|
||||
../go/mail2couch --config config-test.json --max-messages 3
|
||||
|
||||
print_status "🎉 Basic integration tests completed successfully!"
|
||||
|
||||
|
|
|
|||
|
|
@ -67,7 +67,7 @@ build_app() {
|
|||
run_first_sync() {
|
||||
echo -e "\n${BLUE}Running first sync...${NC}"
|
||||
cd go
|
||||
./mail2couch -config ../test/config-test.json -max-messages 5
|
||||
./mail2couch --config ../test/config-test.json --max-messages 5
|
||||
cd ..
|
||||
}
|
||||
|
||||
|
|
@ -138,7 +138,7 @@ EOF
|
|||
run_incremental_sync() {
|
||||
echo -e "\n${BLUE}Running incremental sync...${NC}"
|
||||
cd go
|
||||
./mail2couch -config ../test/config-test.json -max-messages 10
|
||||
./mail2couch --config ../test/config-test.json --max-messages 10
|
||||
cd ..
|
||||
}
|
||||
|
||||
|
|
|
|||
|
|
@ -31,7 +31,7 @@ run_test() {
|
|||
|
||||
# Run mail2couch and capture output
|
||||
cd go
|
||||
if ./mail2couch -config "../test/$config_file" -max-messages "$max_messages" 2>&1; then
|
||||
if ./mail2couch --config "../test/$config_file" --max-messages "$max_messages" 2>&1; then
|
||||
echo -e "${GREEN}✅ Test completed successfully${NC}"
|
||||
else
|
||||
echo -e "${RED}❌ Test failed${NC}"
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue