Search Results for author:
Found 6 papers, 5 papers with code
Scaling Laws for Fine-Grained Mixture of Experts
Our findings not only show that MoE models consistently outperform dense Transformers but also highlight that the efficiency gap between dense and MoE models widens as we scale up the model size and training budget.
Mixture of Tokens: Efficient LLMs through Cross-Example Aggregation
The operation of matching experts and tokens is discrete, which makes MoE models prone to issues like training instability and uneven expert utilization.
Fast and Precise: Adjusting Planning Horizon with Adaptive Subgoal Search
Complex reasoning problems contain states that vary in the computational cost required to determine a good action plan.
Thor: Wielding Hammers to Integrate Language Models and Automated Theorem Provers
Thor increases a language model's success rate on the PISA dataset from $39\%$ to $57\%$, while solving $8. 2\%$ of problems neither language models nor automated theorem provers are able to solve on their own.
Ranked #2 on
Automated Theorem Proving
on miniF2F-test
Subgoal Search For Complex Reasoning Tasks
In this paper, we implement kSubS using a transformer-based subgoal module coupled with the classical best-first search framework.
Trust, but verify: model-based exploration in sparse reward environments
We propose $\textit{trust-but-verify}$ (TBV) mechanism, a new method which uses model uncertainty estimates to guide exploration.
Model-based Reinforcement Learning
reinforcement-learning
+1
