Live control
The controller runs from screen capture through preprocessing, tokenization, dynamics prediction, and action dispatch.
Geometry Dash
Latent world model control at game speed
DashVMC uses an FSQ tokenizer, an action-conditioned transformer world model, and a lightweight actor-critic trained from behavioural cloning plus PPO in latent imagined rollouts. The controller is deployed on live Geometry Dash at the same 30 FPS cadence used by the captured training data; optimized wall-clock latency averages 16.3 ms, leaving compute headroom beyond the deployment rate.
Deployment cadence
30 FPS
Training data cadence
30 FPS
Full-loop latency
16.3 ms
Compute headroom
~61 FPS
Token grid
8x8
The 30 FPS setting matches the capture/training cadence, not the maximum inference rate.
Latent training
PPO updates are performed in token-space rollouts; decoded frames are not part of the policy optimization loop.
Visual rollouts
The decoder is used to inspect generated futures, including plausible level continuations from real gameplay prefixes.
Contributions
A compact Vision-Model-Controller stack
Discrete visual state
Sobel frames are encoded into an 8x8 FSQ token grid that is fast enough for live control.
Action-conditioned dynamics
A causal transformer predicts next-frame tokens from recent token grids and interleaved jump/idle actions.
Latent-rollout policy
A lightweight actor-critic is warm-started by demonstrations and optimized with PPO in model-generated latent rollouts.
Training loop
Control is learned in latent rollouts
The tokenizer and dynamics model define the latent environment used by the controller. After behavioural cloning on demonstrations, PPO improves the actor-critic inside autoregressive token rollouts generated by the action-conditioned world model.
Pixel reconstruction is used to train and inspect the tokenizer, and decoded rollouts are useful for demos. The controller itself consumes token grids and hidden states, so policy learning does not depend on reconstructing pixels at every imagined step.
Application system
Real-time Vision-Model-Controller
Discrete visual state
FSQ levels [8, 5, 5, 5] produce 1000 implicit codes on an 8x8 token grid.
Latent policy training
BC warm-starts the actor-critic, then PPO fine-tunes it in autoregressive latent rollouts.
Level continuations
Real gameplay prefixes can seed action-conditioned visual rollouts of plausible future segments.
30 FPS deployment
The optimized loop averages 16.3 ms on an RTX 2060, so deployment is set to the 30 FPS data cadence rather than a compute ceiling.
Interactive dreams
Human-controlled level continuations
A recorded gameplay prefix initializes the learned world model, then a human supplies jump or idle actions while the world model generates the continuation autoregressively.
Recorded continuations
Human actions, model-generated framesObserved failure modes
Limits of long autoregressive generationResults
Controlled live evaluation
The frozen policy was evaluated over 100 consecutive attempts on each of the first three official levels at 30 FPS with Auto-Retry enabled. Confidence intervals are percentile-bootstrap intervals for the mean over attempts.
| Level | Mean frames [95% CI] | Mean time | Median | Maximum |
|---|---|---|---|---|
| 1 - Stereo Madness | 279.6 [270.5, 289.2] | 9.32 s | 289 | 439 |
| 2 - Back on Track | 263.3 [239.8, 287.2] | 8.78 s | 239 | 457 |
| 3 - Polargeist | 64.3 [59.3, 70.0] | 2.14 s | 51 | 199 |
The evaluator measures acted frames survived rather than level percentage. Results are level-specific: difficulty increases across the sequence, and Polargeist introduces a yellow-orb mechanic requiring an additional timed jump while airborne.
Model snapshot
| FSQ validation SSE / frame | 1.595 |
| Token accuracy | 29.74% |
| Death F1 | 0.7941 |
| BC action accuracy | 79.76% |
| PPO latent survival | 29.71 / 45 |
Selection-aligned values: FSQ epoch 920, world model epoch 139, BC epoch 9, and PPO iteration 3250. The branch ships the PPO checkpoint downstream of BC; the FSQ value is squared error summed per frame, not pixelwise MSE.
Optimized deployment latency
Per-stage means and measured wall-clock latency over 5,000 optimized-path frames on an RTX 2060.
| Screen capture | 2.2 ms |
| Crop | 0.5 ms |
| Grayscale | 0.5 ms |
| Sobel | 2.5 ms |
| Downscale | 2.2 ms |
| FSQ encode | 2.1 ms |
| Transformer | 4.9 ms |
| Controller | 0.5 ms |
| Component-sum mean | 15.3 ms |
| Measured end-to-end | 16.3 ms |
End-to-end median is 16.3 ms and p95 is 18.7 ms; 2 of 5,000 frames exceed the 33.3 ms budget. The measured mean corresponds to roughly 61 FPS of compute headroom.
Reference
Cite this draft
@misc{tariolle2026dashvmc,
title = {DashVMC: Real-Time Discrete World Model Control in Geometry Dash},
author = {Florent Tariolle},
year = {2026},
url = {https://tariolle.github.io/dash-vmc/}
}