Does it work well?

Your browser does not support the video tag. humanoidmaze Your browser does not support the video tag. puzzle To see whether our method scales well to complex tasks, we directly evaluated TRL on some of the most challenging tasks in OGBench, a benchmark for offline goal-conditioned RL. We mainly used the hardest versions of humanoidmaze and puzzle tasks with large, 1B-sized datasets. These tasks are highly challenging: they require performing combinatorially complex skills across up to 3,000 environment steps.

TRL achieves the best performance on highly challenging, long-horizon tasks. The results are quite exciting! Compared to many strong baselines across different categories (TD, MC, quasimetric learning, etc. ), TRL achieves the best performance on most tasks. TRL matches the best, individually tuned TD-$n$, without needing to set $\boldsymbol{n}$.

This is my favorite plot. We compared TRL with $n$-step TD learning with different values of $n$, from $1$ (pure TD) to $\infty$ (pure MC). The result is really nice. TRL matches the best TD-$n$ on all tasks, without needing to set $\boldsymbol{n}$! This is exactly what we wanted from the divide-and-conquer paradigm.

By recursively splitting a trajectory into smaller ones, it can naturally handle long horizons, without having to arbitrarily choose the length of trajectory chunks. The paper has a lot of additional experiments, analyses, and ablations. If you’re interested, check out our paper!