# SkyStream independent generic-detector stress check

Run date: 3 August 2026

This report records a reproducible image-only baseline on the held-out,
privacy-redacted SkyStream benchmark. It supports one narrow conclusion: the
benchmark contains out-of-domain sky imagery that can expose false triggers and
missed small targets in a stock detector. It does not establish the performance
of a buyer model or operational alerting system.

## Method

- Test material: 314 held-out images
- Reviewed negatives: 122 images with no supported benchmark box
- Ground truth: 297 boxes, including 273 with a narrow COCO mapping
- Model: stock TorchVision Faster R-CNN ResNet-50 FPN V2 with
  `FasterRCNN_ResNet50_FPN_V2_Weights.COCO_V1`
- Training or tuning on SkyStream: none
- Match rule: IoU 0.50
- Mapped-class recall: aircraft to airplane and wildlife to bird only
- [Official TorchVision model documentation](https://docs.pytorch.org/vision/0.25/models/generated/torchvision.models.detection.fasterrcnn_resnet50_fpn_v2.html)

## Results

| Score threshold | Detections | Reviewed negatives triggered | Negative trigger rate | Mapped-class recall at IoU 0.50 |
|---:|---:|---:|---:|---:|
| 0.05 | 1,359 | 86 / 122 | 70.5% | 2.2% |
| 0.25 | 367 | 37 / 122 | 30.3% | 1.8% |
| 0.50 | 131 | 12 / 122 | 9.8% | 0.7% |
| 0.75 | 56 | 6 / 122 | 4.9% | 0.7% |

At score 0.25, the model emitted 66 detections across the 37 triggered
reviewed-negative images and matched 5 of 273 narrowly mapped boxes at IoU 0.50.
Its most frequent output labels on the reviewed-negative detections were person
(24), kite (18), bird (12), bear (7), and airplane (3). Those are model outputs,
not SkyStream ground-truth labels.

## Quality and privacy checks

- All 15 private run artifacts match their recorded byte sizes and SHA-256 hashes.
- All 12 error-gallery cases trace to reviewed-negative records in the redacted
  test split.
- The top cases were visually audited and include cloud texture, glare, treeline
  or camera-edge clutter, and small ambiguous sky structures.
- None of the 66 reviewed-negative detections at score 0.25 had its box center
  inside a fixed privacy mask. Eleven boxes touched a mask, but the largest mask
  overlap was 5.1% of a predicted box.
- No exact timestamp, source location, private source map, unredacted image, or
  private operator identity is published with this report.

## Interpretation limits

A reviewed negative means that the image has no supported benchmark box; it does
not mean every pixel is empty. This run used a generic COCO image detector, not a
temporal tracker, counter-UAS system, species model, or buyer model. A production
pipeline may suppress image-level detections with tracking, zones, persistence,
or business rules. The corpus comes from one fixed conventional-RGB viewpoint,
so this is a site-specific stress test rather than evidence of broad deployment
performance.

Reproducibility versions: Torch 2.10.0, TorchVision 0.25.0. Model-weights
SHA-256: `dd69338a24b8d7381807e247652bdc356325bcbaf1cd3e092e00e0a1a58706bf`.

[Return to the SkyStream dataset overview](https://skystream-evaluation.pages.dev/)