Probabilistic Structured Event Modeling

Video Event Recognition under Perceptual Uncertainty

Asian Conference on Computer Vision (ACCV) 2026
Nhan Thanh Ho1,*, Anh Dong Nguyen2,*, Thuan Duc Tran3,*,†, Thang Van Nguyen1, Bui Anh Thinh1, Nguyen Bao Tran4, and Long Nhat Ho5
1Faculty of Computer Science and Engineering, HCMUT – VNU-HCM 2School of Industrial Management, HCMUT – VNU-HCM 3Ho Chi Minh City University of Industry and Trade 4Faculty of Mechanical Engineering, HCMUT – VNU-HCM 5Van Lang University

* Equal contribution  ·  † Corresponding author

Overview of the confidence-aware event retrieval pipeline.

Figure 1: Video observations are converted into structured events E = (T, L, A, S) — time, location, actor and action — through detection, tracking and fixed-length track windows, and stored in an ExplicitCell memory with calibrated per-attribute confidences. Deterministic Hard-AND retrieval discards an event after a single attribute mismatch; our probabilistic model lets uncertain attributes shrink while reliable ones dominate, so the event is still retrieved.

3.3×higher Recall@5 than Hard-AND at the highest attribute noise (0.394 vs. 0.120, CAVIAR)
0.552 → 0.000location-confidence calibration error after isotonic regression
0.221 vs. 0.085Recall@5 on outdoor VIRAT at the highest noise, without recalibration

Abstract

Long-duration video understanding systems summarize continuous observations as structured event representations to enable semantic retrieval. However, these events are produced by imperfect perceptual pipelines (e.g., detectors, trackers, and action classifiers), meaning that every extracted attribute carries inherent uncertainty. Existing retrieval frameworks often ignore this uncertainty, relying on deterministic hard matching that fails when a single cue is misrecognized, or on embedding-based retrieval methods that do not explicitly preserve structured event attributes, particularly spatial information. This paper models structured events as probabilistic entities in which the correctness of each perceptual attribute is treated as a latent variable. A marginal-likelihood retrieval formulation integrates attribute-level uncertainty, allowing reliable attributes to compensate for uncertain ones instead of discarding an event after a single erroneous observation. For the spatial axis, detection covariance is propagated through the homography Jacobian to estimate uncertainty on the ground plane.

Evaluated on an uncertainty-driven event retrieval benchmark using both the indoor CAVIAR and outdoor VIRAT datasets, the framework consistently outperforms deterministic hard matching under increasing attribute noise, reaching 3.3× higher Recall@5 (0.394 vs. 0.120) at the highest corruption level. Controlled ablations identify the per-event confidence as the operative signal precisely when perception is degraded, and delineate the boundary between robustness to attribute noise and failures caused by events missing from memory.

Reference

Nhan Thanh Ho, Anh Dong Nguyen, Thuan Duc Tran, Thang Van Nguyen, Bui Anh Thinh, Nguyen Bao Tran, and Long Nhat Ho. Probabilistic Structured Event Modeling for Video Event Recognition under Perceptual Uncertainty. In Proceedings of the Asian Conference on Computer Vision (ACCV) 2026.

BibTeX citation

@inproceedings{ho2026probabilistic,
  title     = {Probabilistic Structured Event Modeling for Video Event
               Recognition under Perceptual Uncertainty},
  author    = {Ho, Nhan Thanh and Nguyen, Anh Dong and Tran, Thuan Duc and
               Nguyen, Thang Van and {Bui Anh Thinh} and {Nguyen Bao Tran} and
               Ho, Long Nhat},
  booktitle = {Asian Conference on Computer Vision (ACCV)},
  year      = {2026}
}

Method

CIRCUIT memory architecture and ExplicitCell representation.
Figure 2: CIRCUIT memory architecture. Detection (YOLOv8 + homography), tracking (ByteTrack) and action recognition feed unified event nodes — ExplicitCells — that store each attribute together with a calibrated confidence and are indexed along temporal, topic, identity and spatial axes. At query time every attribute is a latent-correctness observation, P(qk|ek) = ĉk·sk + (1 − ĉk)·bk, and events are ranked by the marginal likelihood Σk log P(qk|ek).

Results

Recall@5 under increasing attribute noise and event omission on CAVIAR.
Figure 3: Recall@5 on CAVIAR (S+L+A queries) as perception degrades. (a) Under attribute noise our model degrades gracefully and reaches 3.3× the Recall@5 of Hard-AND at the highest level. (b) When events are missing from memory, all methods converge — no retrieval rule can rank events that were never stored.
Comparison with CLIP on CAVIAR and transfer to VIRAT.
Figure 4: (a) On clean CAVIAR, the structured model roughly doubles a CLIP appearance baseline on location-aware queries, while CLIP stays stronger on isolated action queries. (b) Applied unchanged to outdoor VIRAT scenes, the model keeps its advantage (0.22 vs. 0.09 Recall@5 at the highest noise).

Qualitative example

Query example event
Query · CAVIAR “OneShopOneWait1cor”
Find events where the same person (track 12) is browse in zone g00. Memory holds 85 ExplicitCells of this clip, 7 of which are truly relevant. Stored attributes are corrupted with the paper's heaviest noise level (s3, p = 0.6).
Oursmarginal likelihood, Eq. (6)5 / 5relevant in top-5
Rank 1: event at 33.6–36.6 s (relevant)
#1 · 34–37s✓ relevant
action browse0.96
zone g100.20
track 120.87
Rank 2: event at 36.6–39.6 s (relevant)
#2 · 37–40s✓ relevant
action browse0.77
zone g030.20
track 120.90
Rank 3: event at 30.6–33.6 s (relevant)
#3 · 31–34s✓ relevant
action browse0.77
zone g100.20
track 120.86
Rank 4: event at 24.6–27.6 s (relevant)
#4 · 25–28s✓ relevant
action browse0.83
zone g001.00
track 260.18
Rank 5: event at 39.6–42.6 s (relevant)
#5 · 40–43s✓ relevant
action inactive0.13
zone g100.20
track 120.87
Hard-ANDdeterministic, STEM-style1 / 5relevant in top-5
Rank 1: event at 42.6–45.6 s (not relevant)
#1 · 43–46s✗ wrong
action moving0.96
zone g001.00
track 120.91
Rank 2: event at 12.6–15.6 s (not relevant)
#2 · 13–16s✗ wrong
action browse0.67
zone g101.00
track 120.81
Rank 3: event at 6.6–9.6 s (not relevant)
#3 · 7–10s✗ wrong
action moving0.96
zone g200.95
track 120.90
Rank 4: event at 24.6–27.6 s (relevant)
#4 · 25–28s✓ relevant
action browse0.83
zone g001.00
track 260.18
Rank 5: event at 11.6–14.5 s (not relevant)
#5 · 12–15s✗ wrong
action moving0.96
zone g101.00
track 120.18
queryrelevant (ground truth)not relevant value attribute corrupted by the benchmarkvalue perceived mismatch numbers = calibrated confidence ĉ

Figure 5: Top-5 retrieval for one real query, produced by our evaluation code on the paper's s3 protocol (seed 4). No stored event matches all three cues, so Hard-AND falls back to ranking by confidence and returns only one relevant event. Our model sees that each corrupted cue carries low confidence, lets the remaining reliable cues decide, and all of its top-5 events are relevant.