feat: Add SQLite database integration for aircraft history and callsign enhancement
- Implement comprehensive database package with versioned migrations - Add skyview-data CLI tool for managing aviation reference data - Integrate database with merger for real-time aircraft history persistence - Support OurAirports and OpenFlights data sources (runtime loading) - Add systemd timer for automated database updates - Fix transaction-based bulk loading for 2400% performance improvement - Add callsign enhancement system with airline/airport lookups - Update Debian packaging with database directory and permissions Database features: - Aircraft position history with configurable retention - External aviation data loading (airlines, airports) - Callsign parsing and enhancement - API client for external lookups (OpenSky, etc.) - Privacy mode for complete offline operation CLI commands: - skyview-data status: Show database statistics - skyview-data update: Load aviation reference data - skyview-data list: Show available data sources - skyview-data clear: Remove specific data sources 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
This commit is contained in:
parent
cd51d3ecc0
commit
37c4fa2b57
25 changed files with 4771 additions and 12 deletions
442
docs/DATABASE.md
Normal file
442
docs/DATABASE.md
Normal file
|
|
@ -0,0 +1,442 @@
|
|||
# SkyView Database Architecture
|
||||
|
||||
This document describes SkyView's SQLite database architecture, migration system, and integration approach for persistent data storage.
|
||||
|
||||
## Overview
|
||||
|
||||
SkyView uses a single SQLite database to store:
|
||||
- **Historic aircraft data**: Position history, message counts, signal strength
|
||||
- **Callsign lookup data**: Cached airline/airport information from external APIs
|
||||
- **Embedded aviation data**: OpenFlights airline and airport databases
|
||||
|
||||
## Database Design Principles
|
||||
|
||||
### Embedded Architecture
|
||||
- Single SQLite file for all persistent data
|
||||
- No external database dependencies
|
||||
- Self-contained deployment with embedded schemas
|
||||
- Backward compatibility through versioned migrations
|
||||
|
||||
### Performance Optimization
|
||||
- Strategic indexing for time-series aircraft data
|
||||
- Efficient lookups for callsign enhancement
|
||||
- Configurable data retention policies
|
||||
- Query optimization for real-time operations
|
||||
|
||||
### Data Safety
|
||||
- Atomic migration transactions
|
||||
- Pre-migration backups for destructive changes
|
||||
- Data loss warnings for schema changes
|
||||
- Rollback capabilities where possible
|
||||
|
||||
## Database Schema
|
||||
|
||||
### Core Tables
|
||||
|
||||
#### `schema_info`
|
||||
Tracks database version and applied migrations:
|
||||
```sql
|
||||
CREATE TABLE schema_info (
|
||||
version INTEGER PRIMARY KEY,
|
||||
applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
description TEXT,
|
||||
checksum TEXT
|
||||
);
|
||||
```
|
||||
|
||||
#### `aircraft_history`
|
||||
Stores time-series aircraft position and message data:
|
||||
```sql
|
||||
CREATE TABLE aircraft_history (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
icao_hex TEXT NOT NULL,
|
||||
timestamp TIMESTAMP NOT NULL,
|
||||
latitude REAL,
|
||||
longitude REAL,
|
||||
altitude INTEGER,
|
||||
speed INTEGER,
|
||||
track INTEGER,
|
||||
vertical_rate INTEGER,
|
||||
squawk TEXT,
|
||||
callsign TEXT,
|
||||
source_id TEXT,
|
||||
signal_strength REAL,
|
||||
message_count INTEGER DEFAULT 1
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes:**
|
||||
- `idx_aircraft_history_icao_time`: Fast queries by aircraft and time range
|
||||
- `idx_aircraft_history_timestamp`: Time-based cleanup and queries
|
||||
- `idx_aircraft_history_callsign`: Callsign-based searches
|
||||
|
||||
#### `airlines`
|
||||
OpenFlights embedded airline database:
|
||||
```sql
|
||||
CREATE TABLE airlines (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
alias TEXT,
|
||||
iata TEXT,
|
||||
icao TEXT,
|
||||
callsign TEXT,
|
||||
country TEXT,
|
||||
active BOOLEAN DEFAULT 1
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes:**
|
||||
- `idx_airlines_icao`: ICAO code lookup (primary for callsign enhancement)
|
||||
- `idx_airlines_iata`: IATA code lookup
|
||||
|
||||
#### `airports`
|
||||
OpenFlights embedded airport database:
|
||||
```sql
|
||||
CREATE TABLE airports (
|
||||
id INTEGER PRIMARY KEY,
|
||||
name TEXT NOT NULL,
|
||||
city TEXT,
|
||||
country TEXT,
|
||||
iata TEXT,
|
||||
icao TEXT,
|
||||
latitude REAL,
|
||||
longitude REAL,
|
||||
altitude INTEGER,
|
||||
timezone_offset REAL,
|
||||
dst_type TEXT,
|
||||
timezone TEXT
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes:**
|
||||
- `idx_airports_icao`: ICAO code lookup
|
||||
- `idx_airports_iata`: IATA code lookup
|
||||
|
||||
#### `callsign_cache`
|
||||
Caches external API lookups for callsign enhancement:
|
||||
```sql
|
||||
CREATE TABLE callsign_cache (
|
||||
callsign TEXT PRIMARY KEY,
|
||||
airline_icao TEXT,
|
||||
airline_name TEXT,
|
||||
flight_number TEXT,
|
||||
origin_iata TEXT,
|
||||
destination_iata TEXT,
|
||||
aircraft_type TEXT,
|
||||
cached_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
|
||||
expires_at TIMESTAMP,
|
||||
source TEXT DEFAULT 'local'
|
||||
);
|
||||
```
|
||||
|
||||
**Indexes:**
|
||||
- `idx_callsign_cache_expires`: Efficient cache cleanup
|
||||
|
||||
## Database Location Strategy
|
||||
|
||||
### Path Resolution Order
|
||||
1. **Explicit configuration**: `database.path` in config file
|
||||
2. **System service**: `/var/lib/skyview/skyview.db`
|
||||
3. **User mode**: `~/.local/share/skyview/skyview.db`
|
||||
4. **Fallback**: `./skyview.db` in current directory
|
||||
|
||||
### Directory Permissions
|
||||
- System: `root:root` with `755` permissions for `/var/lib/skyview/`
|
||||
- User: User-owned directories with standard permissions
|
||||
- Service: `skyview:skyview` user/group for system service
|
||||
|
||||
## Migration System
|
||||
|
||||
### Migration Structure
|
||||
```go
|
||||
type Migration struct {
|
||||
Version int // Sequential version number
|
||||
Description string // Human-readable description
|
||||
Up string // SQL for applying migration
|
||||
Down string // SQL for rollback (optional)
|
||||
DataLoss bool // Warning flag for destructive changes
|
||||
}
|
||||
```
|
||||
|
||||
### Migration Process
|
||||
1. **Version Check**: Compare current schema version with available migrations
|
||||
2. **Backup**: Create automatic backup before destructive changes
|
||||
3. **Transaction**: Wrap each migration in atomic transaction
|
||||
4. **Validation**: Verify schema integrity after migration
|
||||
5. **Logging**: Record successful migrations in `schema_info`
|
||||
|
||||
### Data Loss Protection
|
||||
- Migrations marked with `DataLoss: true` require explicit user consent
|
||||
- Automatic backups created before destructive operations
|
||||
- Warning messages displayed during upgrade process
|
||||
- Rollback SQL provided where possible
|
||||
|
||||
### Example Migration Sequence
|
||||
```go
|
||||
var migrations = []Migration{
|
||||
{
|
||||
Version: 1,
|
||||
Description: "Initial schema with aircraft history",
|
||||
Up: createInitialSchema,
|
||||
DataLoss: false,
|
||||
},
|
||||
{
|
||||
Version: 2,
|
||||
Description: "Add OpenFlights airline and airport data",
|
||||
Up: addAviationTables,
|
||||
DataLoss: false,
|
||||
},
|
||||
{
|
||||
Version: 3,
|
||||
Description: "Add callsign lookup cache",
|
||||
Up: addCallsignCache,
|
||||
DataLoss: false,
|
||||
},
|
||||
}
|
||||
```
|
||||
|
||||
## Configuration Integration
|
||||
|
||||
### Database Configuration
|
||||
```json
|
||||
{
|
||||
"database": {
|
||||
"path": "/var/lib/skyview/skyview.db",
|
||||
"max_history_days": 7,
|
||||
"backup_on_upgrade": true
|
||||
},
|
||||
"callsign": {
|
||||
"enabled": true,
|
||||
"cache_hours": 24,
|
||||
"external_apis": true,
|
||||
"privacy_mode": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Configuration Fields
|
||||
|
||||
#### `database`
|
||||
- **`path`**: Database file location (empty = auto-resolve)
|
||||
- **`max_history_days`**: Retention policy for aircraft history (0 = unlimited)
|
||||
- **`backup_on_upgrade`**: Create backup before schema migrations
|
||||
|
||||
#### `callsign`
|
||||
- **`enabled`**: Enable callsign enhancement features
|
||||
- **`cache_hours`**: TTL for cached external API results
|
||||
- **`privacy_mode`**: Disable all external data requests
|
||||
- **`sources`**: Independent control for each data source
|
||||
|
||||
### Enhanced Configuration Example
|
||||
```json
|
||||
{
|
||||
"callsign": {
|
||||
"enabled": true,
|
||||
"cache_hours": 24,
|
||||
"privacy_mode": false,
|
||||
"sources": {
|
||||
"openflights_embedded": {
|
||||
"enabled": true,
|
||||
"priority": 1,
|
||||
"license": "AGPL-3.0"
|
||||
},
|
||||
"faa_registry": {
|
||||
"enabled": false,
|
||||
"priority": 2,
|
||||
"update_frequency": "weekly",
|
||||
"license": "public_domain"
|
||||
},
|
||||
"opensky_api": {
|
||||
"enabled": false,
|
||||
"priority": 3,
|
||||
"timeout_seconds": 5,
|
||||
"max_retries": 2,
|
||||
"requires_consent": true,
|
||||
"license_warning": "Commercial use requires OpenSky Network consent",
|
||||
"user_accepts_terms": false
|
||||
},
|
||||
"custom_database": {
|
||||
"enabled": false,
|
||||
"priority": 4,
|
||||
"path": "",
|
||||
"license": "user_verified"
|
||||
}
|
||||
},
|
||||
"fallback_chain": ["openflights_embedded", "faa_registry", "opensky_api", "custom_database"]
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Individual Source Configuration Options
|
||||
- **`enabled`**: Enable/disable this specific source
|
||||
- **`priority`**: Processing order (lower numbers = higher priority)
|
||||
- **`license`**: License type for compliance tracking
|
||||
- **`requires_consent`**: Whether source requires explicit user consent
|
||||
- **`user_accepts_terms`**: User acknowledgment of licensing terms
|
||||
- **`timeout_seconds`**: Per-source timeout configuration
|
||||
- **`max_retries`**: Per-source retry limits
|
||||
- **`update_frequency`**: For downloadable sources (daily/weekly/monthly)
|
||||
|
||||
## Debian Package Integration
|
||||
|
||||
### Package Structure
|
||||
```
|
||||
/var/lib/skyview/ # Database directory
|
||||
/etc/skyview/config.json # Default configuration
|
||||
/usr/bin/skyview # Main application
|
||||
/usr/share/skyview/ # Embedded resources
|
||||
```
|
||||
|
||||
### Installation Process
|
||||
1. **`postinst`**: Create directories, user accounts, permissions
|
||||
2. **First Run**: Database initialization and migration on startup
|
||||
3. **Upgrades**: Automatic schema migration with backup
|
||||
4. **Service**: Systemd integration with proper database access
|
||||
|
||||
### Service User
|
||||
- User: `skyview`
|
||||
- Home: `/var/lib/skyview`
|
||||
- Shell: `/bin/false` (service account)
|
||||
- Database: Read/write access to `/var/lib/skyview/`
|
||||
|
||||
## Data Retention and Cleanup
|
||||
|
||||
### Automatic Cleanup
|
||||
- **Aircraft History**: Configurable retention period (`max_history_days`)
|
||||
- **Cache Expiration**: TTL-based cleanup of external API cache
|
||||
- **Optimization**: Periodic VACUUM operations for storage efficiency
|
||||
|
||||
### Manual Maintenance
|
||||
```sql
|
||||
-- Clean old aircraft history (example: 7 days)
|
||||
DELETE FROM aircraft_history
|
||||
WHERE timestamp < datetime('now', '-7 days');
|
||||
|
||||
-- Clean expired cache entries
|
||||
DELETE FROM callsign_cache
|
||||
WHERE expires_at < datetime('now');
|
||||
|
||||
-- Optimize database storage
|
||||
VACUUM;
|
||||
```
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Query Optimization
|
||||
- Time-range queries use `idx_aircraft_history_icao_time`
|
||||
- Callsign lookups prioritize local cache over external APIs
|
||||
- Bulk operations use transactions for consistency
|
||||
|
||||
### Storage Efficiency
|
||||
- Configurable history limits prevent unbounded growth
|
||||
- Periodic VACUUM operations reclaim deleted space
|
||||
- Compressed timestamps and efficient data types
|
||||
|
||||
### Memory Usage
|
||||
- WAL mode for concurrent read/write access
|
||||
- Connection pooling for multiple goroutines
|
||||
- Prepared statements for repeated queries
|
||||
|
||||
## Privacy and Security
|
||||
|
||||
### Privacy Mode
|
||||
SkyView includes comprehensive privacy controls through the `privacy_mode` configuration option:
|
||||
|
||||
```json
|
||||
{
|
||||
"callsign": {
|
||||
"enabled": true,
|
||||
"privacy_mode": true,
|
||||
"external_apis": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Privacy Mode Features
|
||||
- **No External Calls**: Completely disables all external API requests
|
||||
- **Local-Only Lookups**: Uses only embedded OpenFlights database for callsign enhancement
|
||||
- **No Data Transmission**: Aircraft data never leaves the local system
|
||||
- **Compliance**: Suitable for sensitive environments requiring air-gapped operation
|
||||
|
||||
#### Privacy Mode Behavior
|
||||
| Feature | Privacy Mode ON | Privacy Mode OFF |
|
||||
|---------|----------------|------------------|
|
||||
| External API calls | ❌ Disabled | ✅ Configurable |
|
||||
| OpenFlights lookup | ✅ Enabled | ✅ Enabled |
|
||||
| Callsign caching | ✅ Local only | ✅ Full caching |
|
||||
| Data transmission | ❌ None | ⚠️ API calls only |
|
||||
|
||||
#### Use Cases for Privacy Mode
|
||||
- **Military installations**: No external data transmission allowed
|
||||
- **Air-gapped networks**: No internet connectivity available
|
||||
- **Corporate policies**: External API usage prohibited
|
||||
- **Personal privacy**: User preference for local-only operation
|
||||
|
||||
### Security Considerations
|
||||
|
||||
#### File Permissions
|
||||
- Database files readable only by skyview user/group
|
||||
- Configuration files protected from unauthorized access
|
||||
- Backup files inherit secure permissions
|
||||
|
||||
#### Data Protection
|
||||
- Local SQLite database with file-system level security
|
||||
- No cloud storage or external database dependencies
|
||||
- All aviation data processed and stored locally
|
||||
|
||||
#### Network Security
|
||||
- External API calls (when enabled) use HTTPS only
|
||||
- No persistent connections to external services
|
||||
- Optional certificate validation for API endpoints
|
||||
|
||||
### Data Integrity
|
||||
- Foreign key constraints where applicable
|
||||
- Transaction isolation for concurrent operations
|
||||
- Checksums for migration verification
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Common Issues
|
||||
|
||||
#### Database Locked
|
||||
```
|
||||
Error: database is locked
|
||||
```
|
||||
**Solution**: Stop SkyView service, check for stale lock files, restart
|
||||
|
||||
#### Migration Failures
|
||||
```
|
||||
Error: migration 3 failed: table already exists
|
||||
```
|
||||
**Solution**: Check schema version, restore from backup, retry migration
|
||||
|
||||
#### Permission Denied
|
||||
```
|
||||
Error: unable to open database file
|
||||
```
|
||||
**Solution**: Verify file permissions, check directory ownership, ensure disk space
|
||||
|
||||
### Diagnostic Commands
|
||||
```bash
|
||||
# Check database integrity
|
||||
sqlite3 /var/lib/skyview/skyview.db "PRAGMA integrity_check;"
|
||||
|
||||
# View schema version
|
||||
sqlite3 /var/lib/skyview/skyview.db "SELECT * FROM schema_info;"
|
||||
|
||||
# Database statistics
|
||||
sqlite3 /var/lib/skyview/skyview.db ".dbinfo"
|
||||
```
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Planned Features
|
||||
- **Compression**: Time-series compression for long-term storage
|
||||
- **Partitioning**: Date-based partitioning for large datasets
|
||||
- **Replication**: Read replica support for high-availability setups
|
||||
- **Analytics**: Built-in reporting and statistics tables
|
||||
|
||||
### Migration Path
|
||||
- All enhancements will use versioned migrations
|
||||
- Backward compatibility maintained for existing installations
|
||||
- Data preservation prioritized over schema optimization
|
||||
Loading…
Add table
Add a link
Reference in a new issue