Ole-Morten Duesund
37c4fa2b57
- Implement comprehensive database package with versioned migrations - Add skyview-data CLI tool for managing aviation reference data - Integrate database with merger for real-time aircraft history persistence - Support OurAirports and OpenFlights data sources (runtime loading) - Add systemd timer for automated database updates - Fix transaction-based bulk loading for 2400% performance improvement - Add callsign enhancement system with airline/airport lookups - Update Debian packaging with database directory and permissions Database features: - Aircraft position history with configurable retention - External aviation data loading (airlines, airports) - Callsign parsing and enhancement - API client for external lookups (OpenSky, etc.) - Privacy mode for complete offline operation CLI commands: - skyview-data status: Show database statistics - skyview-data update: Load aviation reference data - skyview-data list: Show available data sources - skyview-data clear: Remove specific data sources 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude <noreply@anthropic.com>
345 lines
No EOL
14 KiB
Markdown
345 lines
No EOL
14 KiB
Markdown
# SkyView Architecture Documentation
|
|
|
|
## Overview
|
|
|
|
SkyView is a high-performance, multi-source ADS-B aircraft tracking system built in Go with a modern JavaScript frontend. It connects to multiple dump1090 Beast format receivers, performs intelligent data fusion, and provides low-latency aircraft tracking through a responsive web interface.
|
|
|
|
## System Architecture
|
|
|
|
```
|
|
┌─────────────────┐ ┌──────────────┐ ┌─────────────────┐
|
|
│ dump1090 │ │ dump1090 │ │ dump1090 │
|
|
│ Receiver 1 │ │ Receiver 2 │ │ Receiver N │
|
|
│ Port 30005 │ │ Port 30005 │ │ Port 30005 │
|
|
└─────────┬───────┘ └──────┬───────┘ └─────────┬───────┘
|
|
│ │ │
|
|
│ Beast Binary │ Beast Binary │ Beast Binary
|
|
│ TCP Stream │ TCP Stream │ TCP Stream
|
|
│ │ │
|
|
└───────────────────┼──────────────────────┘
|
|
│
|
|
┌─────────▼──────────┐
|
|
│ SkyView Server │
|
|
│ │
|
|
│ ┌────────────────┐ │
|
|
│ │ Beast Client │ │ ── Multi-source TCP clients
|
|
│ │ Manager │ │
|
|
│ └────────────────┘ │
|
|
│ ┌────────────────┐ │
|
|
│ │ Mode S/ADS-B │ │ ── Message parsing & decoding
|
|
│ │ Decoder │ │
|
|
│ └────────────────┘ │
|
|
│ ┌────────────────┐ │
|
|
│ │ Data Merger │ │ ── Intelligent data fusion
|
|
│ │ & ICAO DB │ │
|
|
│ └────────────────┘ │
|
|
│ ┌────────────────┐ │
|
|
│ │ HTTP/WebSocket │ │ ── Low-latency web interface
|
|
│ │ Server │ │
|
|
│ └────────────────┘ │
|
|
└─────────┬──────────┘
|
|
│
|
|
┌─────────▼──────────┐
|
|
│ Web Interface │
|
|
│ │
|
|
│ • Interactive Maps │
|
|
│ • Low-latency Updates│
|
|
│ • Aircraft Details │
|
|
│ • Coverage Analysis│
|
|
│ • 3D Visualization │
|
|
└────────────────────┘
|
|
```
|
|
|
|
## Core Components
|
|
|
|
### 1. Multi-Format Clients (`internal/client/`)
|
|
|
|
**Purpose**: Manages TCP connections to dump1090/readsb receivers using multiple protocols
|
|
|
|
**Key Features**:
|
|
- Concurrent connection handling for multiple sources
|
|
- Automatic reconnection with exponential backoff
|
|
- Support for Beast binary and VRS JSON formats
|
|
- Per-source connection monitoring and statistics
|
|
- Mixed-format multi-source support
|
|
|
|
**Files**:
|
|
- `beast.go`: Beast binary client and multi-source manager
|
|
- `vrs.go`: VRS JSON format client
|
|
|
|
**Supported Formats**:
|
|
- **Beast Binary**: Traditional binary format from dump1090 (port 30005)
|
|
- **VRS JSON**: JSON format from readsb VRS output (port 33005)
|
|
|
|
### 2. Data Format Processors
|
|
|
|
#### Mode S/ADS-B Decoder (`internal/modes/`)
|
|
**Purpose**: Decodes raw Mode S and ADS-B messages into structured aircraft data
|
|
|
|
**Key Features**:
|
|
- CPR (Compact Position Reporting) decoding with zone ambiguity resolution
|
|
- ADS-B message type parsing (position, velocity, identification)
|
|
- Aircraft category and type classification
|
|
- Signal quality assessment
|
|
|
|
**Files**:
|
|
- `decoder.go`: Core decoding logic
|
|
|
|
#### VRS JSON Parser (`internal/vrs/`)
|
|
**Purpose**: Parses Virtual Radar Server JSON format aircraft data
|
|
|
|
**Key Features**:
|
|
- Newline-delimited JSON stream parsing
|
|
- Direct aircraft data extraction (no Mode S decoding required)
|
|
- Support for VRS-specific fields (registration, aircraft type, operator)
|
|
- Position source identification (ADS-B, MLAT, TIS-B, Satellite)
|
|
|
|
**Files**:
|
|
- `parser.go`: VRS JSON message parsing
|
|
|
|
### 3. Data Merger (`internal/merger/`)
|
|
|
|
**Purpose**: Fuses aircraft data from multiple sources using intelligent conflict resolution
|
|
|
|
**Key Features**:
|
|
- Signal strength-based source selection
|
|
- High-performance data fusion and conflict resolution
|
|
- Aircraft state management and lifecycle tracking
|
|
- Historical data collection (position, altitude, speed, signal trails)
|
|
- Automatic stale aircraft cleanup
|
|
- SQLite database integration for persistent storage
|
|
|
|
**Files**:
|
|
- `merger.go`: Multi-source data fusion engine
|
|
|
|
### 4. Database System (`internal/database/`)
|
|
|
|
**Purpose**: Provides persistent storage for historical aircraft data and callsign enhancement
|
|
|
|
**Key Features**:
|
|
- SQLite-based storage with versioned schema migrations
|
|
- Aircraft position history with configurable retention
|
|
- Embedded OpenFlights airline and airport databases
|
|
- Callsign lookup cache for external API results
|
|
- Privacy mode for air-gapped operation
|
|
- Automatic database maintenance and cleanup
|
|
|
|
**Files**:
|
|
- `database.go`: Core database operations and schema management
|
|
- `migrations.go`: Database schema versioning and migration system
|
|
- `callsign.go`: Callsign enhancement and cache management
|
|
|
|
**Storage Components**:
|
|
- **Aircraft History**: Time-series position data with source attribution
|
|
- **OpenFlights Data**: Embedded airline/airport reference data
|
|
- **Callsign Cache**: External API lookup results with TTL
|
|
- **Schema Versioning**: Migration tracking and rollback support
|
|
|
|
### 5. ICAO Country Database (`internal/icao/`)
|
|
|
|
**Purpose**: Provides comprehensive ICAO address to country mapping
|
|
|
|
**Key Features**:
|
|
- In-memory database with 100+ allocations covering 70+ countries and territories
|
|
- Based on official aerotransport.org ICAO allocation tables
|
|
- Fast binary search lookups on sorted allocation ranges
|
|
- Complete country names, ISO codes, and flag emojis
|
|
- Zero external dependencies - pure Go implementation
|
|
|
|
**Files**:
|
|
- `database.go`: In-memory ICAO allocation database with binary search
|
|
|
|
### 6. HTTP/WebSocket Server (`internal/server/`)
|
|
|
|
**Purpose**: Serves web interface and provides low-latency data streaming
|
|
|
|
**Key Features**:
|
|
- RESTful API for aircraft and system data
|
|
- WebSocket connections for low-latency updates
|
|
- Static asset serving with embedded resources
|
|
- Coverage analysis and signal heatmaps
|
|
|
|
**Files**:
|
|
- `server.go`: HTTP server and WebSocket handler
|
|
|
|
### 7. Web Frontend (`assets/static/`)
|
|
|
|
**Purpose**: Interactive web interface for aircraft tracking and visualization
|
|
|
|
**Key Technologies**:
|
|
- **Leaflet.js**: Interactive maps and aircraft markers
|
|
- **Three.js**: 3D radar visualization
|
|
- **Chart.js**: Live statistics and charts
|
|
- **WebSockets**: Live data streaming
|
|
- **Responsive CSS**: Mobile-optimized interface
|
|
|
|
**Files**:
|
|
- `index.html`: Main web interface
|
|
- `js/app.js`: Main application orchestrator
|
|
- `js/modules/`: Modular JavaScript components
|
|
- `aircraft-manager.js`: Aircraft marker and trail management
|
|
- `map-manager.js`: Map controls and overlays
|
|
- `ui-manager.js`: User interface state management
|
|
- `websocket.js`: Low-latency data connections
|
|
- `css/style.css`: Responsive styling and themes
|
|
- `icons/`: SVG aircraft type icons
|
|
|
|
## Data Flow
|
|
|
|
### 1. Data Ingestion
|
|
1. **Multi-format Clients** connect to dump1090/readsb receivers via TCP
|
|
2. **Format Parsers** process binary Beast or JSON VRS message streams
|
|
3. **Data Converters** convert messages to structured aircraft data (Mode S decoding for Beast, direct mapping for VRS)
|
|
4. **Data Merger** receives aircraft updates with source attribution
|
|
|
|
### 2. Data Fusion
|
|
1. **Signal Analysis**: Compare signal strength across sources
|
|
2. **Conflict Resolution**: Select best data based on signal quality and recency
|
|
3. **State Management**: Update aircraft position, velocity, and metadata
|
|
4. **History Tracking**: Maintain trails for visualization
|
|
|
|
### 3. Country Lookup
|
|
1. **ICAO Extraction**: Extract 24-bit ICAO address from aircraft data
|
|
2. **Database Query**: Lookup country information in in-memory database using binary search
|
|
3. **Data Enrichment**: Add country, country code, and flag to aircraft state
|
|
|
|
### 4. Data Distribution
|
|
1. **REST API**: Provide aircraft data via HTTP endpoints
|
|
2. **WebSocket Streaming**: Push low-latency updates to connected clients
|
|
3. **Frontend Processing**: Update maps, tables, and visualizations
|
|
4. **User Interface**: Display aircraft with country flags and details
|
|
|
|
## Configuration System
|
|
|
|
### Configuration Sources (Priority Order)
|
|
1. Command-line flags (highest priority)
|
|
2. Configuration file (JSON)
|
|
3. Default values (lowest priority)
|
|
|
|
### Configuration Structure
|
|
```json
|
|
{
|
|
"server": {
|
|
"host": "", // Bind address (empty = all interfaces)
|
|
"port": 8080 // HTTP server port
|
|
},
|
|
"sources": [
|
|
{
|
|
"id": "unique-id", // Source identifier
|
|
"name": "Display Name", // Human-readable name
|
|
"host": "hostname", // Receiver hostname/IP
|
|
"port": 30005, // Beast format port
|
|
"latitude": 51.4700, // Receiver location
|
|
"longitude": -0.4600,
|
|
"altitude": 50.0, // Meters above sea level
|
|
"enabled": true // Source enable/disable
|
|
}
|
|
],
|
|
"settings": {
|
|
"history_limit": 500, // Max trail points per aircraft
|
|
"stale_timeout": 60, // Seconds before aircraft removed
|
|
"update_rate": 1 // WebSocket update frequency
|
|
},
|
|
"origin": {
|
|
"latitude": 51.4700, // Map center point
|
|
"longitude": -0.4600,
|
|
"name": "Origin Name"
|
|
},
|
|
"database": {
|
|
"path": "", // Auto-resolved: /var/lib/skyview/skyview.db
|
|
"max_history_days": 7, // Data retention period
|
|
"backup_on_upgrade": true // Backup before migrations
|
|
},
|
|
"callsign": {
|
|
"enabled": true, // Enable callsign enhancement
|
|
"cache_hours": 24, // External API cache TTL
|
|
"external_apis": true, // Allow external API calls
|
|
"privacy_mode": false // Disable external data transmission
|
|
}
|
|
}
|
|
```
|
|
|
|
## Performance Characteristics
|
|
|
|
### Concurrency Model
|
|
- **Goroutine per Source**: Each Beast client runs in separate goroutine
|
|
- **Mutex-Protected Merger**: Thread-safe aircraft state management
|
|
- **WebSocket Broadcasting**: Concurrent client update distribution
|
|
- **Non-blocking I/O**: Asynchronous network operations
|
|
|
|
### Memory Management
|
|
- **Database Storage**: Persistent history reduces memory usage
|
|
- **Configurable Retention**: Database cleanup based on age and limits
|
|
- **Automatic Cleanup**: Stale aircraft removal to prevent memory leaks
|
|
- **Efficient Data Structures**: Maps for O(1) aircraft lookups
|
|
- **Embedded Assets**: Static files bundled in binary
|
|
|
|
### Scalability
|
|
- **Multi-source Support**: Tested with 10+ concurrent receivers
|
|
- **High Message Throughput**: Handles 1000+ messages/second per source
|
|
- **Low-latency Updates**: Sub-second latency for aircraft updates
|
|
- **Responsive Web UI**: Optimized for 100+ concurrent aircraft
|
|
|
|
## Security Considerations
|
|
|
|
### Network Security
|
|
- **No Authentication Required**: Designed for trusted network environments
|
|
- **Local Network Operation**: Intended for private receiver networks
|
|
- **WebSocket Origin Checking**: Basic CORS protection
|
|
|
|
### System Security
|
|
- **Unprivileged Execution**: Runs as non-root user in production
|
|
- **Filesystem Isolation**: Minimal file system access required
|
|
- **Network Isolation**: Only requires outbound TCP to receivers
|
|
- **Systemd Hardening**: Security features enabled in service file
|
|
|
|
### Data Privacy
|
|
- **Public ADS-B Data**: Only processes publicly broadcast aircraft data
|
|
- **No Personal Information**: Aircraft tracking only, no passenger data
|
|
- **Privacy Mode**: Complete offline operation with external API disable
|
|
- **Local Processing**: All data processed and stored locally
|
|
- **Historical Limits**: Configurable data retention periods
|
|
|
|
## External Resources
|
|
|
|
### Official Standards
|
|
- **ICAO Document 8585**: Designators for Aircraft Operating Agencies
|
|
- **RTCA DO-260B**: ADS-B Message Formats and Protocols
|
|
- **ITU-R M.1371-5**: Technical characteristics for universal ADS-B
|
|
|
|
### Technology Dependencies
|
|
- **Go Language**: https://golang.org/
|
|
- **Leaflet.js**: https://leafletjs.com/ - Interactive maps
|
|
- **Three.js**: https://threejs.org/ - 3D visualization
|
|
- **Chart.js**: https://www.chartjs.org/ - Statistics charts
|
|
- **WebSocket Protocol**: RFC 6455
|
|
|
|
### ADS-B Ecosystem
|
|
- **dump1090**: https://github.com/antirez/dump1090 - SDR ADS-B decoder
|
|
- **Beast Binary Format**: Mode S data interchange format
|
|
- **FlightAware**: ADS-B network and data provider
|
|
- **OpenSky Network**: Research-oriented ADS-B network
|
|
|
|
## Development Guidelines
|
|
|
|
### Code Organization
|
|
- **Package per Component**: Clear separation of concerns
|
|
- **Interface Abstractions**: Testable and mockable components
|
|
- **Error Handling**: Comprehensive error reporting and recovery
|
|
- **Documentation**: Extensive code comments and examples
|
|
|
|
### Testing Strategy
|
|
- **Unit Tests**: Component-level testing with mocks
|
|
- **Integration Tests**: End-to-end data flow validation
|
|
- **Performance Tests**: Load testing with simulated data
|
|
- **Manual Testing**: Real-world receiver validation
|
|
|
|
### Deployment Options
|
|
- **Standalone Binary**: Single executable with embedded assets
|
|
- **Debian Package**: Systemd service with configuration
|
|
- **Docker Container**: Containerized deployment option
|
|
- **Development Mode**: Hot-reload for frontend development
|
|
|
|
---
|
|
|
|
**SkyView Architecture** - Designed for reliability, performance, and extensibility in multi-source ADS-B tracking applications. |