skyview/docs/DATABASE.md

# SkyView Database Architecture

This document describes SkyView's SQLite database architecture, migration system, and integration approach for persistent data storage.

## Overview

SkyView uses a single SQLite database to store:
- **Historic aircraft data**: Position history, message counts, signal strength
- **Callsign lookup data**: Cached airline/airport information from external APIs
- **Embedded aviation data**: OpenFlights airline and airport databases

## Database Design Principles

### Embedded Architecture
- Single SQLite file for all persistent data
- No external database dependencies
- Self-contained deployment with embedded schemas
- Backward compatibility through versioned migrations

### Performance Optimization
- Strategic indexing for time-series aircraft data
- Efficient lookups for callsign enhancement
- Configurable data retention policies
- Query optimization for real-time operations

### Data Safety
- Atomic migration transactions
- Pre-migration backups for destructive changes
- Data loss warnings for schema changes
- Rollback capabilities where possible

## Database Schema

### Core Tables

#### `schema_info`
Tracks database version and applied migrations:
```sql
CREATE TABLE schema_info (
    version INTEGER PRIMARY KEY,
    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    description TEXT,
    checksum TEXT
);
```

#### `aircraft_history`
Stores time-series aircraft position and message data:
```sql
CREATE TABLE aircraft_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    icao_hex TEXT NOT NULL,
    timestamp TIMESTAMP NOT NULL,
    latitude REAL,
    longitude REAL,
    altitude INTEGER,
    speed INTEGER,
    track INTEGER,
    vertical_rate INTEGER,
    squawk TEXT,
    callsign TEXT,
    source_id TEXT,
    signal_strength REAL,
    message_count INTEGER DEFAULT 1
);
```

**Indexes:**
- `idx_aircraft_history_icao_time`: Fast queries by aircraft and time range
- `idx_aircraft_history_timestamp`: Time-based cleanup and queries
- `idx_aircraft_history_callsign`: Callsign-based searches

#### `airlines`
OpenFlights embedded airline database:
```sql
CREATE TABLE airlines (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    alias TEXT,
    iata TEXT,
    icao TEXT,
    callsign TEXT,
    country TEXT,
    active BOOLEAN DEFAULT 1
);
```

**Indexes:**
- `idx_airlines_icao`: ICAO code lookup (primary for callsign enhancement)
- `idx_airlines_iata`: IATA code lookup

#### `airports`
OpenFlights embedded airport database:
```sql
CREATE TABLE airports (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    city TEXT,
    country TEXT,
    iata TEXT,
    icao TEXT,
    latitude REAL,
    longitude REAL,
    altitude INTEGER,
    timezone_offset REAL,
    dst_type TEXT,
    timezone TEXT
);
```

**Indexes:**
- `idx_airports_icao`: ICAO code lookup
- `idx_airports_iata`: IATA code lookup

#### `callsign_cache`
Caches external API lookups for callsign enhancement:
```sql
CREATE TABLE callsign_cache (
    callsign TEXT PRIMARY KEY,
    airline_icao TEXT,
    airline_name TEXT,
    flight_number TEXT,
    origin_iata TEXT,
    destination_iata TEXT,
    aircraft_type TEXT,
    cached_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP,
    source TEXT DEFAULT 'local'
);
```

**Indexes:**
- `idx_callsign_cache_expires`: Efficient cache cleanup

## Database Location Strategy

### Path Resolution Order
1. **Explicit configuration**: `database.path` in config file
2. **System service**: `/var/lib/skyview/skyview.db`
3. **User mode**: `~/.local/share/skyview/skyview.db`
4. **Fallback**: `./skyview.db` in current directory

### Directory Permissions
- System: `root:root` with `755` permissions for `/var/lib/skyview/`
- User: User-owned directories with standard permissions
- Service: `skyview:skyview` user/group for system service

## Migration System

### Migration Structure
```go
type Migration struct {
    Version     int    // Sequential version number
    Description string // Human-readable description
    Up          string // SQL for applying migration
    Down        string // SQL for rollback (optional)
    DataLoss    bool   // Warning flag for destructive changes
}
```

### Migration Process
1. **Version Check**: Compare current schema version with available migrations
2. **Backup**: Create automatic backup before destructive changes
3. **Transaction**: Wrap each migration in atomic transaction
4. **Validation**: Verify schema integrity after migration
5. **Logging**: Record successful migrations in `schema_info`

### Data Loss Protection
- Migrations marked with `DataLoss: true` require explicit user consent
- Automatic backups created before destructive operations
- Warning messages displayed during upgrade process
- Rollback SQL provided where possible

### Example Migration Sequence
```go
var migrations = []Migration{
    {
        Version:     1,
        Description: "Initial schema with aircraft history",
        Up:          createInitialSchema,
        DataLoss:    false,
    },
    {
        Version:     2,
        Description: "Add OpenFlights airline and airport data",
        Up:          addAviationTables,
        DataLoss:    false,
    },
    {
        Version:     3,
        Description: "Add callsign lookup cache",
        Up:          addCallsignCache,
        DataLoss:    false,
    },
}
```

## Configuration Integration

### Database Configuration
```json
{
  "database": {
    "path": "/var/lib/skyview/skyview.db",
    "max_history_days": 7,
    "backup_on_upgrade": true
  },
  "callsign": {
    "enabled": true,
    "cache_hours": 24,
    "external_apis": true,
    "privacy_mode": false
  }
}
```

### Configuration Fields

#### `database`
- **`path`**: Database file location (empty = auto-resolve)
- **`max_history_days`**: Retention policy for aircraft history (0 = unlimited)
- **`backup_on_upgrade`**: Create backup before schema migrations

#### `callsign`
- **`enabled`**: Enable callsign enhancement features
- **`cache_hours`**: TTL for cached external API results
- **`privacy_mode`**: Disable all external data requests
- **`sources`**: Independent control for each data source

### Enhanced Configuration Example
```json
{
  "callsign": {
    "enabled": true,
    "cache_hours": 24,
    "privacy_mode": false,
    "sources": {
      "openflights_embedded": {
        "enabled": true,
        "priority": 1,
        "license": "AGPL-3.0"
      },
      "faa_registry": {
        "enabled": false,
        "priority": 2,
        "update_frequency": "weekly",
        "license": "public_domain"
      },
      "opensky_api": {
        "enabled": false,
        "priority": 3,
        "timeout_seconds": 5,
        "max_retries": 2,
        "requires_consent": true,
        "license_warning": "Commercial use requires OpenSky Network consent",
        "user_accepts_terms": false
      },
      "custom_database": {
        "enabled": false,
        "priority": 4,
        "path": "",
        "license": "user_verified"
      }
    },
    "fallback_chain": ["openflights_embedded", "faa_registry", "opensky_api", "custom_database"]
  }
}
```

#### Individual Source Configuration Options
- **`enabled`**: Enable/disable this specific source
- **`priority`**: Processing order (lower numbers = higher priority)
- **`license`**: License type for compliance tracking
- **`requires_consent`**: Whether source requires explicit user consent
- **`user_accepts_terms`**: User acknowledgment of licensing terms
- **`timeout_seconds`**: Per-source timeout configuration
- **`max_retries`**: Per-source retry limits
- **`update_frequency`**: For downloadable sources (daily/weekly/monthly)

## Debian Package Integration

### Package Structure
```
/var/lib/skyview/           # Database directory
/etc/skyview/config.json    # Default configuration
/usr/bin/skyview           # Main application
/usr/share/skyview/        # Embedded resources
```

### Installation Process
1. **`postinst`**: Create directories, user accounts, permissions
2. **First Run**: Database initialization and migration on startup
3. **Upgrades**: Automatic schema migration with backup
4. **Service**: Systemd integration with proper database access

### Service User
- User: `skyview`
- Home: `/var/lib/skyview`
- Shell: `/bin/false` (service account)
- Database: Read/write access to `/var/lib/skyview/`

## Data Retention and Cleanup

### Automatic Cleanup
- **Aircraft History**: Configurable retention period (`max_history_days`)
- **Cache Expiration**: TTL-based cleanup of external API cache
- **Optimization**: Periodic VACUUM operations for storage efficiency

### Manual Maintenance
```sql
-- Clean old aircraft history (example: 7 days)
DELETE FROM aircraft_history 
WHERE timestamp < datetime('now', '-7 days');

-- Clean expired cache entries
DELETE FROM callsign_cache 
WHERE expires_at < datetime('now');

-- Optimize database storage
VACUUM;
```

## Performance Considerations

### Query Optimization
- Time-range queries use `idx_aircraft_history_icao_time`
- Callsign lookups prioritize local cache over external APIs
- Bulk operations use transactions for consistency

### Storage Efficiency
- Configurable history limits prevent unbounded growth
- Periodic VACUUM operations reclaim deleted space
- Compressed timestamps and efficient data types

### Memory Usage
- WAL mode for concurrent read/write access
- Connection pooling for multiple goroutines
- Prepared statements for repeated queries

## Privacy and Security

### Privacy Mode
SkyView includes comprehensive privacy controls through the `privacy_mode` configuration option:

```json
{
  "callsign": {
    "enabled": true,
    "privacy_mode": true,
    "external_apis": false
  }
}
```

#### Privacy Mode Features
- **No External Calls**: Completely disables all external API requests
- **Local-Only Lookups**: Uses only embedded OpenFlights database for callsign enhancement
- **No Data Transmission**: Aircraft data never leaves the local system
- **Compliance**: Suitable for sensitive environments requiring air-gapped operation

#### Privacy Mode Behavior
| Feature | Privacy Mode ON | Privacy Mode OFF |
|---------|----------------|------------------|
| External API calls | ❌ Disabled | ✅ Configurable |
| OpenFlights lookup | ✅ Enabled | ✅ Enabled |
| Callsign caching | ✅ Local only | ✅ Full caching |
| Data transmission | ❌ None | ⚠️ API calls only |

#### Use Cases for Privacy Mode
- **Military installations**: No external data transmission allowed
- **Air-gapped networks**: No internet connectivity available
- **Corporate policies**: External API usage prohibited
- **Personal privacy**: User preference for local-only operation

### Security Considerations

#### File Permissions
- Database files readable only by skyview user/group
- Configuration files protected from unauthorized access
- Backup files inherit secure permissions

#### Data Protection
- Local SQLite database with file-system level security
- No cloud storage or external database dependencies
- All aviation data processed and stored locally

#### Network Security
- External API calls (when enabled) use HTTPS only
- No persistent connections to external services
- Optional certificate validation for API endpoints

### Data Integrity
- Foreign key constraints where applicable
- Transaction isolation for concurrent operations
- Checksums for migration verification

## Troubleshooting

### Common Issues

#### Database Locked
```
Error: database is locked
```
**Solution**: Stop SkyView service, check for stale lock files, restart

#### Migration Failures
```
Error: migration 3 failed: table already exists
```
**Solution**: Check schema version, restore from backup, retry migration

#### Permission Denied
```
Error: unable to open database file
```
**Solution**: Verify file permissions, check directory ownership, ensure disk space

### Diagnostic Commands
```bash
# Check database integrity
sqlite3 /var/lib/skyview/skyview.db "PRAGMA integrity_check;"

# View schema version
sqlite3 /var/lib/skyview/skyview.db "SELECT * FROM schema_info;"

# Database statistics
sqlite3 /var/lib/skyview/skyview.db ".dbinfo"
```

## Future Enhancements

### Planned Features
- **Compression**: Time-series compression for long-term storage
- **Partitioning**: Date-based partitioning for large datasets
- **Replication**: Read replica support for high-availability setups
- **Analytics**: Built-in reporting and statistics tables

### Migration Path
- All enhancements will use versioned migrations
- Backward compatibility maintained for existing installations
- Data preservation prioritized over schema optimization