olemd/skyview

Watch

Fork 0

Valid targets being filtered out compared to dump1090 #24

New issue

Closed

opened 2025-08-25 08:27:13 +02:00 by olemd · 3 comments

olemd commented

2025-08-25 08:27:13 +02:00

Owner

Problem Description

SkyView is filtering out several valid targets that dump1090 is able to show. This suggests we may have overly strict filtering or decoding problems that are rejecting legitimate ADS-B messages.

Expected Behavior

SkyView should be able to decode and display all valid aircraft that dump1090 can successfully process and display.

Potential Root Causes

Decoding problems in the Mode S/ADS-B message parser
Overly strict validation in the beast protocol handler
CPR position decoding being too conservative
Message type filtering excluding valid aircraft
Signal strength thresholds being too high
Incomplete implementation of certain ADS-B message types

Investigation Areas

beast/parser.go: Raw message parsing and validation
modes/decoder.go: Mode S/ADS-B message decoding logic
modes/cpr.go: CPR position decoding and validation
merger/merger.go: Data fusion logic that might filter out sources
Message type handling for DF17, DF18, and other formats

Steps to Reproduce

Run dump1090 and SkyView simultaneously on the same data source
Compare the aircraft list between both applications
Identify aircraft visible in dump1090 but missing in SkyView
Analyze the raw messages for the missing aircraft

Impact

Missing valid targets reduces the effectiveness of SkyView for aircraft tracking and may cause users to miss important aircraft in their coverage area.

Debugging Approach

Enable verbose logging to see rejected messages
Compare raw message handling between SkyView and dump1090
Log validation failures with specific reasons
Test with known good sample data from dump1090

## Problem Description SkyView is filtering out several valid targets that dump1090 is able to show. This suggests we may have overly strict filtering or decoding problems that are rejecting legitimate ADS-B messages. ## Expected Behavior SkyView should be able to decode and display all valid aircraft that dump1090 can successfully process and display. ## Potential Root Causes - Decoding problems in the Mode S/ADS-B message parser - Overly strict validation in the beast protocol handler - CPR position decoding being too conservative - Message type filtering excluding valid aircraft - Signal strength thresholds being too high - Incomplete implementation of certain ADS-B message types ## Investigation Areas - **beast/parser.go**: Raw message parsing and validation - **modes/decoder.go**: Mode S/ADS-B message decoding logic - **modes/cpr.go**: CPR position decoding and validation - **merger/merger.go**: Data fusion logic that might filter out sources - Message type handling for DF17, DF18, and other formats ## Steps to Reproduce 1. Run dump1090 and SkyView simultaneously on the same data source 2. Compare the aircraft list between both applications 3. Identify aircraft visible in dump1090 but missing in SkyView 4. Analyze the raw messages for the missing aircraft ## Impact Missing valid targets reduces the effectiveness of SkyView for aircraft tracking and may cause users to miss important aircraft in their coverage area. ## Debugging Approach - Enable verbose logging to see rejected messages - Compare raw message handling between SkyView and dump1090 - Log validation failures with specific reasons - Test with known good sample data from dump1090

olemd commented

2025-08-25 08:30:45 +02:00

Author

Owner

Additional Information - Position Decoding Errors

Some targets that we do manage to decode are appearing hundreds of kilometers away from their actual position. This suggests serious issues with our CPR (Compact Position Reporting) decoding implementation.

Symptoms

Aircraft appearing at incorrect locations (100+ km offset)
Position jumps that are physically impossible
Decoded positions that don't match known airport/route locations

Likely Causes

CPR Global/Local Decoding Issues:
- Incorrect reference position for local CPR decoding
- Wrong latitude/longitude zone calculation
- Missing or incorrect odd/even frame pairing
Reference Position Problems:
- Using wrong reference position for relative decoding
- Not properly handling position ambiguity near zone boundaries
- Incorrect handling of surface vs airborne position formats
Implementation Bugs:
- Integer overflow in position calculations
- Incorrect scaling factors for lat/lon conversion
- Wrong constants in CPR algorithm implementation

Critical Areas to Review

modes/cpr.go: Complete review of CPR decoding algorithm
Position reference selection logic
Odd/even frame synchronization
Zone boundary handling
Surface position vs airborne position differentiation

This position error issue is likely related to the filtering problem - we may be rejecting correctly decoded positions as 'impossible' while accepting incorrectly decoded ones.

## Additional Information - Position Decoding Errors Some targets that we do manage to decode are appearing hundreds of kilometers away from their actual position. This suggests serious issues with our CPR (Compact Position Reporting) decoding implementation. ### Symptoms - Aircraft appearing at incorrect locations (100+ km offset) - Position jumps that are physically impossible - Decoded positions that don't match known airport/route locations ### Likely Causes 1. **CPR Global/Local Decoding Issues**: - Incorrect reference position for local CPR decoding - Wrong latitude/longitude zone calculation - Missing or incorrect odd/even frame pairing 2. **Reference Position Problems**: - Using wrong reference position for relative decoding - Not properly handling position ambiguity near zone boundaries - Incorrect handling of surface vs airborne position formats 3. **Implementation Bugs**: - Integer overflow in position calculations - Incorrect scaling factors for lat/lon conversion - Wrong constants in CPR algorithm implementation ### Critical Areas to Review - **modes/cpr.go**: Complete review of CPR decoding algorithm - Position reference selection logic - Odd/even frame synchronization - Zone boundary handling - Surface position vs airborne position differentiation This position error issue is likely related to the filtering problem - we may be rejecting correctly decoded positions as 'impossible' while accepting incorrectly decoded ones.

olemd referenced this issue from a commit

2026-09-29 13:34:55 +02:00

fix: Correct CPR position decoding errors (refs #35, #24)

olemd referenced this issue from a commit

2026-09-29 13:53:45 +02:00

fix: Stop rejecting valid positions in merger validation (refs #24, #35)

olemd referenced this issue from a commit

2026-09-29 14:15:45 +02:00

fix: Handle Beast escaping across the whole frame (refs #24)

olemd referenced this issue from a commit

2026-09-29 14:33:18 +02:00

fix: Accept Mode S address/parity replies (refs #24)

olemd commented

2026-09-29 14:41:08 +02:00

Author

Owner

Fixed in v0.0.13

Several independent causes combined to hide targets that dump1090 shows:

Mode S surveillance replies were all discarded (612d65b). DF0/4/5/16/20/21 overlay the CRC with the aircraft address, so the CRC residue is never zero. The decoder required a zero residue and rejected every one of them, which was 58% of all messages in a 30 s capture. Aircraft without ADS-B never appeared, and squawk was almost always missing. The address is now recovered from the residue, and a reply is accepted only if that address was recently confirmed by a message with a clean CRC. The AC13 altitude and ID13 squawk fields were also mis-decoded and are fixed (c4fb68a).
CPR decoding dropped or misplaced positions. See #35 (30b21e9).
Merger validation rejected valid positions (42352e1). Reports from different receivers a few milliseconds apart looked like thousands of knots, out-of-order timestamps were rejected, and a bad first position blocked all later ones. This is the "rejecting correctly decoded positions" suspicion in the comment above.
ADS-B altitude was decoded with the wrong bit layout, which roughly halved it (33d5beb).
Beast framing misaligned frames that had 0x1A in the timestamp or signal field, about 0.3% of frames (0dcd913).

Live comparison with dump1090 on the same feed:

	v0.0.12	v0.0.13
Messages decoded	~42%	99.7%
Aircraft shown	–	20/20
Altitude vs dump1090	about half the real value	median 25 ft, max 175 ft
Squawk	rarely available	17/17 match

Released in v0.0.13. Local (single-frame) CPR decoding, which gets positions onto the map faster, follows in a separate commit.

## Fixed in v0.0.13 Several independent causes combined to hide targets that dump1090 shows: 1. **Mode S surveillance replies were all discarded** (612d65b). DF0/4/5/16/20/21 overlay the CRC with the aircraft address, so the CRC residue is never zero. The decoder required a zero residue and rejected every one of them, which was 58% of all messages in a 30 s capture. Aircraft without ADS-B never appeared, and squawk was almost always missing. The address is now recovered from the residue, and a reply is accepted only if that address was recently confirmed by a message with a clean CRC. The AC13 altitude and ID13 squawk fields were also mis-decoded and are fixed (c4fb68a). 2. **CPR decoding dropped or misplaced positions.** See #35 (30b21e9). 3. **Merger validation rejected valid positions** (42352e1). Reports from different receivers a few milliseconds apart looked like thousands of knots, out-of-order timestamps were rejected, and a bad first position blocked all later ones. This is the "rejecting correctly decoded positions" suspicion in the comment above. 4. **ADS-B altitude** was decoded with the wrong bit layout, which roughly halved it (33d5beb). 5. **Beast framing** misaligned frames that had `0x1A` in the timestamp or signal field, about 0.3% of frames (0dcd913). **Live comparison** with dump1090 on the same feed: | | v0.0.12 | v0.0.13 | |---|---|---| | Messages decoded | ~42% | 99.7% | | Aircraft shown | – | 20/20 | | Altitude vs dump1090 | about half the real value | median 25 ft, max 175 ft | | Squawk | rarely available | 17/17 match | Released in [v0.0.13](https://kode.naiv.no/olemd/skyview/releases/tag/v0.0.13). Local (single-frame) CPR decoding, which gets positions onto the map faster, follows in a separate commit.

olemd closed this issue

2026-09-29 14:41:08 +02:00

olemd commented

2026-09-29 14:50:38 +02:00

Author

Owner

Follow-up: local (single-frame) CPR decoding landed in fa01e71. Once an aircraft has a position from a global decode, a single frame is decoded relative to it if that position is less than a minute old. This helps when frames are missed at the edge of coverage or when a pair straddles an NL boundary. On the strong test feed it made no measurable difference (183/189 position frames decoded either way), since valid pairs were almost always available.

Follow-up: local (single-frame) CPR decoding landed in fa01e71. Once an aircraft has a position from a global decode, a single frame is decoded relative to it if that position is less than a minute old. This helps when frames are missed at the edge of coverage or when a pair straddles an NL boundary. On the strong test feed it made no measurable difference (183/189 position frames decoded either way), since valid pairs were almost always available.