← Toutes les actualitésLire l'article ↗
When two maps disagree
Synthèse
Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either. So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0.34; the grounded sweep, ≈ 0.85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surpri
Now the recursive turn: If difficulty is something we generate, then we need to evaluate the generator itself; we should not trust it blindly either. So we built the same ambiguity sweep two ways: steering terms from a pure-LLM guess, versus terms grounded in TF-IDF surprisal. The two disagreed violently. At high ambiguity, the LLM-built sweep scored the agent at F1 ≈ 0. 34; the grounded sweep, ≈ 0.
85. One of these maps is badly distorted. The grounded one, predictably, is the more robust: surprisal gives it a footing the free-running LLM lacks. This is "evaluate your evals," made concrete. The information-theoretic lens does not only grade the agent along a continuous axis; it grades the benchmark's own construction, and adjudicates between the two.
Source originale
en.wikipedia.org