Preprint · 2026
GeoGuesser Bench
Benchmarking Vision-Language Agents on Interactive Geolocation
282 tasks in an offline Street View environment, where agents rotate, zoom and move to report coordinates, name places and find objects.
- University of California, Berkeley
- ∗Equal contribution
- Correspondence to: huanzhimao@berkeley.edu
Leaderboard
23 VLMs with identical prompts, tools and a 40-step budget, scored in GeoGuessr points (0–5000) or accuracy.
Demo
Step through an example of each task type, see each tool at work, then watch real agents’ logged tool calls.
Three task types, from knowing to looking
Pick a type to step through an illustrative example with the arrow keys, the slider or the log.
By design, Precise Localization depends mainly on knowledge, Place Naming on both, and Navigation Challenge mainly on exploration.
The environment
Camera tools change the view (a scripted API test); move tools step to an adjacent panorama.
In benchmark episodes
- Start facing north with a 90° view
- Up to 40 tool-calling steps
- No map, coordinates or web search
- Offline panoramas: views never drift
Real agent traces
Logged tool calls from evaluation runs. The logs don’t keep the returned images.
Benchmark
How answers are scored, and where the 282 tasks start.
Scoring
In Precise Localization, points fall exponentially with distance and halve about every 1,034 km.
S(d) = 5000 · exp(−10 d / D)
D = 14,916.862 km
Place Naming and Navigation Challenge need an exact match with an accepted answer.
Where the tasks start
Starting locations of all 282 entries. Click a marker to explore nearby Google Street View (opens in a new tab).
Findings
Five results from evaluating 23 VLMs; open one to see the data behind it.
Knowing where you are is not the same as finding things
Across 23 models, Precise Localization scores correlate only weakly with Navigation Challenge accuracy (Spearman ρ = 0.39 for both splits). GPT-5.4 is third on Synthetic but ties for 18th on Navigation Challenge.
Each dot is one model; hover or focus a dot for its values.
Show the values as a table
Agents trail the experts, and often fail to commit
The best agent, Claude Opus 4.8, averages 4352.4 points on Competition against the experts’ 4770.3. Pooled over models, 45.7% of Competition and 38.6% of Synthetic episodes score zero, mostly at the step limit or by abstaining.
Share of all episodes, pooled over the 23 models.
Show the counts as a table
Stronger agents tolerate restricted views
With half of each view cut away, Claude Opus 4.8 keeps 98% of its score, Gemini 3.1 Pro only 33%. Across 13 models, the share kept rises with main-setting score (Spearman ρ = 0.73).
Averaged over both halves and both splits; “Main” is the mean main-setting score.
Show the values as a table
Agents rarely inspect the full panorama
Without movement, about two-thirds of episodes with a tool call never tilt the camera. Yet Claude Opus 4.8 scores 4646 on Competition without moving or tilting, covering a median 0.31 of the viewing sphere.
Navigation is the main failure mode
In failed episodes of four models, navigational reasoning failure (FM2) is the most common mode on every task, appearing in 60–80% of them. It accounts for most failures of three of the four models.
An efficient harness
Three models share the work, and a short brief replaces the raw history that grows every step.
Raw history · reread every step
Median input, Gemini 3 Flash on Navigation Challenge.
Brief · read in its place
brief+focus_hint+key frames
Citation
If you use GeoGuesser Bench or its environment, please cite the paper.
@article{mao2026geoguesserbench,
title = {{GeoGuesser Bench}: Benchmarking Vision-Language Agents on Interactive Geolocation},
author = {Mao, Huanzhi and Wang, Yichuan and Koo, Michael and Huang, Wenyuan and Doan, Huy and Xia, Jinghan and Wang, Ziyao and Gonzalez, Joseph E.},
journal = {arXiv preprint arXiv:2611.02698},
year = {2026},
eprint = {2611.02698},
archivePrefix = {arXiv},
primaryClass = {cs.AI},
url = {https://arxiv.org/abs/2611.02698}
}