skyview/docs/DATABASE.md

SkyView Database Architecture

This document describes SkyView's SQLite database architecture, migration system, and integration approach for persistent data storage.

Overview

SkyView uses a single SQLite database to store:

• Historic aircraft data: Position history, message counts, signal strength
• Callsign lookup data: Cached airline/airport information from external APIs
• Embedded aviation data: OpenFlights airline and airport databases

Database Design Principles

Embedded Architecture

• Single SQLite file for all persistent data
• No external database dependencies
• Self-contained deployment with embedded schemas
• Backward compatibility through versioned migrations

Performance Optimization

• Strategic indexing for time-series aircraft data
• Efficient lookups for callsign enhancement
• Configurable data retention policies
• Query optimization for real-time operations

Data Safety

• Atomic migration transactions
• Pre-migration backups for destructive changes
• Data loss warnings for schema changes
• Rollback capabilities where possible

Database Schema

Core Tables

schema_info

Tracks database version and applied migrations:

CREATE TABLE schema_info (
    version INTEGER PRIMARY KEY,
    applied_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    description TEXT,
    checksum TEXT
);

aircraft_history

Stores time-series aircraft position and message data:

CREATE TABLE aircraft_history (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    icao_hex TEXT NOT NULL,
    timestamp TIMESTAMP NOT NULL,
    latitude REAL,
    longitude REAL,
    altitude INTEGER,
    speed INTEGER,
    track INTEGER,
    vertical_rate INTEGER,
    squawk TEXT,
    callsign TEXT,
    source_id TEXT,
    signal_strength REAL,
    message_count INTEGER DEFAULT 1
);

Indexes:

• idx_aircraft_history_icao_time: Fast queries by aircraft and time range
• idx_aircraft_history_timestamp: Time-based cleanup and queries
• idx_aircraft_history_callsign: Callsign-based searches

airlines

OpenFlights embedded airline database:

CREATE TABLE airlines (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    alias TEXT,
    iata TEXT,
    icao TEXT,
    callsign TEXT,
    country TEXT,
    active BOOLEAN DEFAULT 1
);

Indexes:

• idx_airlines_icao: ICAO code lookup (primary for callsign enhancement)
• idx_airlines_iata: IATA code lookup

airports

OpenFlights embedded airport database:

CREATE TABLE airports (
    id INTEGER PRIMARY KEY,
    name TEXT NOT NULL,
    city TEXT,
    country TEXT,
    iata TEXT,
    icao TEXT,
    latitude REAL,
    longitude REAL,
    altitude INTEGER,
    timezone_offset REAL,
    dst_type TEXT,
    timezone TEXT
);

Indexes:

• idx_airports_icao: ICAO code lookup
• idx_airports_iata: IATA code lookup

callsign_cache

Caches external API lookups for callsign enhancement:

CREATE TABLE callsign_cache (
    callsign TEXT PRIMARY KEY,
    airline_icao TEXT,
    airline_name TEXT,
    flight_number TEXT,
    origin_iata TEXT,
    destination_iata TEXT,
    aircraft_type TEXT,
    cached_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    expires_at TIMESTAMP,
    source TEXT DEFAULT 'local'
);

Indexes:

• idx_callsign_cache_expires: Efficient cache cleanup

Database Location Strategy

Path Resolution Order

1. Explicit configuration: database.path in config file
2. System service: /var/lib/skyview/skyview.db
3. User mode: ~/.local/share/skyview/skyview.db
4. Fallback: ./skyview.db in current directory

Directory Permissions

• System: root:root with 755 permissions for /var/lib/skyview/
• User: User-owned directories with standard permissions
• Service: skyview:skyview user/group for system service

Migration System

Migration Structure

type Migration struct {
    Version     int    // Sequential version number
    Description string // Human-readable description
    Up          string // SQL for applying migration
    Down        string // SQL for rollback (optional)
    DataLoss    bool   // Warning flag for destructive changes
}

Migration Process

1. Version Check: Compare current schema version with available migrations
2. Backup: Create automatic backup before destructive changes
3. Transaction: Wrap each migration in atomic transaction
4. Validation: Verify schema integrity after migration
5. Logging: Record successful migrations in schema_info

Data Loss Protection

• Migrations marked with DataLoss: true require explicit user consent
• Automatic backups created before destructive operations
• Warning messages displayed during upgrade process
• Rollback SQL provided where possible

Example Migration Sequence

var migrations = []Migration{
    {
        Version:     1,
        Description: "Initial schema with aircraft history",
        Up:          createInitialSchema,
        DataLoss:    false,
    },
    {
        Version:     2,
        Description: "Add OpenFlights airline and airport data",
        Up:          addAviationTables,
        DataLoss:    false,
    },
    {
        Version:     3,
        Description: "Add callsign lookup cache",
        Up:          addCallsignCache,
        DataLoss:    false,
    },
}

Configuration Integration

Database Configuration

{
  "database": {
    "path": "/var/lib/skyview/skyview.db",
    "max_history_days": 7,
    "backup_on_upgrade": true
  },
  "callsign": {
    "enabled": true,
    "cache_hours": 24,
    "external_apis": true,
    "privacy_mode": false
  }
}

Configuration Fields

database

• path: Database file location (empty = auto-resolve)
• max_history_days: Retention policy for aircraft history (0 = unlimited)
• backup_on_upgrade: Create backup before schema migrations

callsign

• enabled: Enable callsign enhancement features
• cache_hours: TTL for cached external API results
• privacy_mode: Disable all external data requests
• sources: Independent control for each data source

Enhanced Configuration Example

{
  "callsign": {
    "enabled": true,
    "cache_hours": 24,
    "privacy_mode": false,
    "sources": {
      "openflights_embedded": {
        "enabled": true,
        "priority": 1,
        "license": "AGPL-3.0"
      },
      "faa_registry": {
        "enabled": false,
        "priority": 2,
        "update_frequency": "weekly",
        "license": "public_domain"
      },
      "opensky_api": {
        "enabled": false,
        "priority": 3,
        "timeout_seconds": 5,
        "max_retries": 2,
        "requires_consent": true,
        "license_warning": "Commercial use requires OpenSky Network consent",
        "user_accepts_terms": false
      },
      "custom_database": {
        "enabled": false,
        "priority": 4,
        "path": "",
        "license": "user_verified"
      }
    },
    "fallback_chain": ["openflights_embedded", "faa_registry", "opensky_api", "custom_database"]
  }
}

Individual Source Configuration Options

• enabled: Enable/disable this specific source
• priority: Processing order (lower numbers = higher priority)
• license: License type for compliance tracking
• requires_consent: Whether source requires explicit user consent
• user_accepts_terms: User acknowledgment of licensing terms
• timeout_seconds: Per-source timeout configuration
• max_retries: Per-source retry limits
• update_frequency: For downloadable sources (daily/weekly/monthly)

Debian Package Integration

Package Structure

/var/lib/skyview/           # Database directory
/etc/skyview/config.json    # Default configuration
/usr/bin/skyview           # Main application
/usr/share/skyview/        # Embedded resources

Installation Process

1. postinst: Create directories, user accounts, permissions
2. First Run: Database initialization and migration on startup
3. Upgrades: Automatic schema migration with backup
4. Service: Systemd integration with proper database access

Service User

• User: skyview
• Home: /var/lib/skyview
• Shell: /bin/false (service account)
• Database: Read/write access to /var/lib/skyview/

Data Retention and Cleanup

Automatic Cleanup

• Aircraft History: Configurable retention period (max_history_days)
• Cache Expiration: TTL-based cleanup of external API cache
• Optimization: Periodic VACUUM operations for storage efficiency

Manual Maintenance

-- Clean old aircraft history (example: 7 days)
DELETE FROM aircraft_history 
WHERE timestamp < datetime('now', '-7 days');

-- Clean expired cache entries
DELETE FROM callsign_cache 
WHERE expires_at < datetime('now');

-- Optimize database storage
VACUUM;

Performance Considerations

Query Optimization

• Time-range queries use idx_aircraft_history_icao_time
• Callsign lookups prioritize local cache over external APIs
• Bulk operations use transactions for consistency

Storage Efficiency

• Configurable history limits prevent unbounded growth
• Periodic VACUUM operations reclaim deleted space
• Compressed timestamps and efficient data types

Memory Usage

• WAL mode for concurrent read/write access
• Connection pooling for multiple goroutines
• Prepared statements for repeated queries

Privacy and Security

Privacy Mode

SkyView includes comprehensive privacy controls through the privacy_mode configuration option:

{
  "callsign": {
    "enabled": true,
    "privacy_mode": true,
    "external_apis": false
  }
}

Privacy Mode Features

• No External Calls: Completely disables all external API requests
• Local-Only Lookups: Uses only embedded OpenFlights database for callsign enhancement
• No Data Transmission: Aircraft data never leaves the local system
• Compliance: Suitable for sensitive environments requiring air-gapped operation

Privacy Mode Behavior

Feature	Privacy Mode ON	Privacy Mode OFF
External API calls	❌ Disabled	✅ Configurable
OpenFlights lookup	✅ Enabled	✅ Enabled
Callsign caching	✅ Local only	✅ Full caching
Data transmission	❌ None	⚠️ API calls only

Use Cases for Privacy Mode

• Military installations: No external data transmission allowed
• Air-gapped networks: No internet connectivity available
• Corporate policies: External API usage prohibited
• Personal privacy: User preference for local-only operation

Security Considerations

File Permissions

• Database files readable only by skyview user/group
• Configuration files protected from unauthorized access
• Backup files inherit secure permissions

Data Protection

• Local SQLite database with file-system level security
• No cloud storage or external database dependencies
• All aviation data processed and stored locally

Network Security

• External API calls (when enabled) use HTTPS only
• No persistent connections to external services
• Optional certificate validation for API endpoints

Data Integrity

• Foreign key constraints where applicable
• Transaction isolation for concurrent operations
• Checksums for migration verification

Troubleshooting

Common Issues

Database Locked

Error: database is locked

Solution: Stop SkyView service, check for stale lock files, restart

Migration Failures

Error: migration 3 failed: table already exists

Solution: Check schema version, restore from backup, retry migration

Permission Denied

Error: unable to open database file

Solution: Verify file permissions, check directory ownership, ensure disk space

Diagnostic Commands

# Check database integrity
sqlite3 /var/lib/skyview/skyview.db "PRAGMA integrity_check;"

# View schema version
sqlite3 /var/lib/skyview/skyview.db "SELECT * FROM schema_info;"

# Database statistics
sqlite3 /var/lib/skyview/skyview.db ".dbinfo"

Future Enhancements

Planned Features

• Compression: Time-series compression for long-term storage
• Partitioning: Date-based partitioning for large datasets
• Replication: Read replica support for high-availability setups
• Analytics: Built-in reporting and statistics tables

Migration Path

• All enhancements will use versioned migrations
• Backward compatibility maintained for existing installations
• Data preservation prioritized over schema optimization