Counting bicycles part 2: recall and counting

Part 1 ended on two open questions: whether a doorbell sees enough of the street to count anything, and whether bikes would come back labelled person with the bicycle box dropped.

The first is yes. The second is also yes, and it turned out to be the whole problem.

This is what got built on top, and what the numbers said.

What mobilenet actually did

Over 11 hours, the bundled SSD MobileNet produced:

It sees the rider and drops the bike. This wasn’t a threshold problem — the bicycle hits it did produce scored 0.67–0.97, so there was no weak-box population sitting just under min_score waiting to be recovered by loosening a filter. The boxes simply weren’t there.

Swapping the model

YOLOv9-s, exported to ONNX with the Ultralytics container:

docker run --rm -v "$PWD:/work" ultralytics/ultralytics:latest-cpu \
  yolo export model=yolov9s.pt format=onnx imgsz=320

Measured by replaying saved snapshots through each candidate:

Plus four passes mobilenet missed entirely. 640 is worse than 320 — the motion regions Frigate crops are only ~300×300 native, so upscaling adds interpolation, not detail.

Inference went 6.85 ms → 12.19 ms against a 100 ms budget at 10 fps. The .onnx is 28 MB, so it isn’t in git and isn’t in the image — it sits on the Synology and mounts read-only:

- name: models
  nfs:
    server: 192.168.1.22
    path: /volume1/data/frigate-models
    readOnly: true

Read-only works because Frigate sets no OpenVINO CACHE_DIR — it recompiles the model in memory on every start, so nothing ever gets written next to the weights. No initContainer, no copy-to-PVC.

The counting layer

Detection produces events. Counting them is a separate job, and it lives in Home Assistant:

flowchart TB
FRIG["🧠 Frigate · near_lane zone<br/>bicycle + person"]
MQTT["📡 frigate/events<br/>Mosquitto · LXC 105"]
AUTO["⚙️ MQTT automation<br/>matches end/voordeur/bicycle/True"]
CNT["🔢 counter.bicycles_passed"]
BRIDGE["🌉 Template sensor<br/>state_class: total_increasing"]
METERS["📊 utility_meters<br/>hour · day · week · month"]
DASH["📈 Dashboard<br/>totals + hour-of-day profile"]
PROBE["❓ Miss-rate probe<br/>short person events"]
QUEUE["🖼️ Judge queue<br/>input_text of event ids"]
SUB["🏷️ Frigate sub_label<br/>bike / not_bike"]

FRIG --> MQTT
MQTT --> AUTO
AUTO --> CNT
CNT --> BRIDGE
BRIDGE --> METERS
METERS --> DASH
MQTT --> PROBE
PROBE --> QUEUE
QUEUE --> SUB
SUB --> CNT

The zone matters. near_lane covers the part of the frame where detection is reliable, and both counting and the probe filter on it — otherwise you count whatever wanders past the hedge.

Filtering happens in the MQTT trigger rather than a condition, because no native Home Assistant condition inspects a JSON payload:

triggers:
  - trigger: mqtt
    topic: frigate/events
    value_template: >-
      {{ value_json.type }}/{{ value_json.after.camera }}/{{
      value_json.after.label }}/{{ 'near_lane' in
      (value_json.after.entered_zones or []) }}
    payload: end/voordeur/bicycle/True

Counting on type: end matters too — Frigate publishes new, update and end per tracked object, and only end fires once.

The judge queue

A count is worthless without a miss rate, and passive observation only shows what Frigate caught. So the probe counts short (<5 s) person events in the near lane as candidate cyclists whose bike was never labelled.

Duration is the discriminator because average_estimated_speed reads 0 for every event — it needs zone distance calibration that isn’t configured. The measured split:

Each candidate shows up on the dashboard as a snapshot with two buttons — yes, a bike and no, on foot. Pressing one writes a sub_label back onto the Frigate event via rest_command, and increments the matching counter.

Storing the verdict on the Frigate event rather than in Home Assistant was the right call for three reasons: it’s unlimited, it survives Home Assistant restarts, and ?sub_labels=bike doubles as the shortlist for a future fine-tune. An earlier version tracked an “everything older than this timestamp is judged” watermark, which was simply wrong — candidates fall off the end of the queue unjudged, making them permanently unreachable.

The ratio is the point: judged-bike ÷ (judged-bike + judged-not-bike) is the fraction of short near-lane person events that are really cyclists. Multiply by the day’s candidate count and you have an estimate of bikes actually missed.

Class-agnostic NMS

With ground truth from the judge queue, the missed bikes could finally be analysed properly. Ruled out first, with data:

• Not resolution. Of 47 confirmed-missed bikes, 46 were ≥43 px wide (median 75, max 155). Only one sat in the band where a higher-resolution detect stream would help.
• Not region clipping. Widening the crop 1.5×, 2× and to full frame rescued 0 of 8.
• Not thresholds. The boxes score well above min_score and are deleted before any Frigate filter sees them.

The cause is in __post_process_nms_yolo:

indices = cv2.dnn.NMSBoxes(boxes, scores, 0.4, 0.4)

Non-maximum suppression over every box regardless of class. Two overlapping boxes are treated as duplicate detections of one object. That’s correct for two overlapping people. It’s wrong for a person on a bicycle, where the overlap is two real objects — and the rider’s person box outscores the bicycle box every time (measured 0.84–0.91 against 0.48–0.80, at IoU 0.42–0.52).

A model swap can’t fix this. post_process_dfine, post_process_rfdetr, __post_process_nms_yolo, __post_process_multipart_yolo and post_process_yolox all call the same class-agnostic NMSBoxes. RF-DETR needs no NMS at all — Frigate applies one anyway. The only NMS-free path is ssd, which is mobilenet, which is worse.

Dropping person from objects.track doesn’t help either: detect() runs the detector (NMS inside) and only then applies the track-list filter. The bicycle box is already gone.

OpenCV 4.11 ships NMSBoxesBatched, which does per-class NMS natively. Replayed offline against 56 manually-confirmed cyclists:

19 passes — 33% — recovered by one function call. That shipped as a usercustomize.py shim mounted via ConfigMap, with PYTHONPATH=/opt/frigate-patch:/opt/frigate.

And then it did nothing

Matched-window measurement in production, 10:15 –15:00 local on consecutive days, both sides after the model swap and after the zone existed, split by the patch going live:

Flat. An earlier 23-minute check agreed. Resolution is roughly ±5 pp, so a small effect could hide in the noise — but recovering 19 of 56 missed bikes would have moved share to 75–80%, and an effect that size is excluded.

The patch was verified running before measuring — [nms-patch] class-aware NMS installed for yolo-generic in the logs, pod up 23 h, zero restarts. Not a silent fail-open.

Why the offline test overpromised: it replayed one frame per event — the peak-person snapshot, the single frame where the rider’s box most dominates. Production sees 10–30 frames per pass and needs only one of them to yield a bicycle box above threshold. Stock NMS was already winning on the easier frames. The patch fixed a constraint that wasn’t binding.

So it got reverted. It forked upstream by rebinding a private function, failed open (meaning a silent regression after any Frigate upgrade would look like nothing at all), and cost ~3.5 ms/frame. Inference went back from 15.7 ms to 12.27 ms.

The 25% of passes producing no bicycle box at all were always the bigger number. Only a better model fixes those.

Gotchas

model_type must be yolo-generic, not yolov9

The latter is accepted and silently falls back to CPU. Check /api/stats for the detector’s inference_speed — CPU inference is obvious once you look.

yolo-generic hard-codes a 0.4 score floor

Both the filter and NMSBoxes’ score_threshold in __post_process_nms_yolo sit at 0.4, before Frigate applies its own filters, capped at 20 detections per frame. Setting min_score below 0.4 is inert with this model type. That limit didn’t exist on the ssd/mobilenet path.

Replaying past events needs the crop, not the frame

To test a new model against historical events, crop each event’s stored data.region (normalised x, y, w, h) out of its snapshot — that’s the exact tile the detector saw. Running full frames instead shrinks distant objects to nothing and produces a meaningless zero. Preprocess as RGB and /255 to match _transform_input for input_dtype: float; feeding 0–255 gives confident nonsense like hair drier 1.0.

Frigate 0.17 also ignores crop, height and bbox query params on /api/events/<id>/snapshot.jpg — you always get the full detect-resolution frame.

utility_meter refuses non-sensor sources

source: counter.x fails validation with “belongs to domain counter, expected [‘sensor’]”. Hence a template sensor bridging counter → sensor. There’s no dedicated helper for that conversion.

Never lower a counter feeding a total_increasing sensor

A decrease is read as a meter reset, so the entire new value is booked as fresh growth. Four downward re-baselines in one afternoon produced a per-hour bar of 192 against a real 19. The giveaway is the statistic’s state going down while change spikes.

Don’t backfill statistics into a live-recorded sensor

Home Assistant recomputes each hour’s sum from the sensor’s own state deltas and doesn’t reliably chain onto imported sums. The running total collapses and the chart renders a large negative bar — observed state=228 but sum=41, so change came out at −131.

Diagnose with statistic_types: ["change","state","sum"]. If state and sum disagree, the chain is broken. Backfilling only works for statistic ids Home Assistant does not live-record.

A fresh utility_meter has no unit until its first reading

Creating a meter and calibrating it immediately writes statistics rows with a blank unit. When the first source update arrives the meter inherits the unit, and Home Assistant raises a repair about the change. Harmless — the values were always right, only the label was missing, so update the units is the correct answer rather than delete. Avoid it by calibrating after the first reading.

apexcharts-card’s data_generator replaces the fetch

For the hour-of-day chart. The generator signature is (entity, start, end, hass, moment) — there is no data parameter, and the call site is if (data_generator) …generate… else …fetch history…. A statistics: block alongside a data_generator is dead config that never runs.

The generator is async and gets hass, so it fetches its own:

const res = await hass.callWS({
  type: 'recorder/statistics_during_period',
  start_time: new Date(Date.now() - 30*24*3600*1000).toISOString(),
  end_time: new Date().toISOString(),
  statistic_ids: [id],
  period: 'hour',
  types: ['change']
});

Two more things there. The card is time-series only — category and numeric x-axes both fail, one silently drawing axes with no bars, the other hanging on a spinner. The fix is to decouple the windows: graph_span: 24h with span: {start: 'day'} sets the display axis to today, while the generator does its own 30-day lookback internally and maps each hour onto today’s date.

And plot at the middle of each hour. Home Assistant’s hourly buckets are labelled by their start, but ApexCharts centres a column on its x value — so a point at 05:00 draws a bar straddling 04:30 –05:30 that reads as the wrong interval. Adding 30 minutes makes the bar span the hour it actually represents.

Where it stands

Detection runs on the medium substream at 960×720, 10 fps, zero skipped frames, 12.27 ms inference on the iGPU. Everything is upstream Frigate again — the only local change is the model.

Counts to date: 776 bicycles, of which 545 detected automatically and 231 recovered by judging. That ratio is the honest headline. Roughly 30% of the count comes from a human pressing a button, which is a fine way to build a training set and a poor way to run a sensor.

The hour-of-day profile has a shape already, and it isn’t the one I expected:

No morning commute peak at all — 08:00 is one of the quietest active hours. The busiest is midday, with a second bulge late afternoon. That may well be August school holidays; the first weekend day and the end of the holidays will both test it.

Unproven: everything about the shape of that curve, on two days of data. And the 14 of 56 passes that produce no bicycle box under any NMS scheme — that’s the number a fine-tune on the sub_label=bike set would have to move, and it hasn’t been attempted.