An Empirical Study of Data Ability Boundary in LLMs' Math Reasoning
Large language models (LLMs) are displaying emergent abilities for math reasoning tasks,and there is a growing attention on enhancing the ability of open-source LLMs through supervised fine-tuning (SFT).In this paper, we aim to explore a general data strategy for supervised data to help optimize and expand math reasoning ability.Firstly, we determine the ability boundary of reasoning paths augmentation by identifying these paths' minimal optimal set.Secondly, we validate that different abilities of the model can be cumulatively enhanced by Mix of Minimal Optimal Sets of corresponding types of data, while our models MMOS achieve SOTA performance on series base models under much lower construction costs.Besides, we point out GSM-HARD is not really hard and today's LLMs no longer lack numerical robustness.Also, we provide an Auto Problem Generator for robustness testing and educational applications.Our code and data are publicly available at https://github.com/cyzhh/MMOS.
PDF AbstractCode
GSM8K
MATH
SVAMP
Results from the Paper
Ranked #2 on
Math Word Problem Solving
on ASDiv-A
(using extra training data)
| Task | Dataset | Model | Metric Name | Metric Value | Global Rank | Uses Extra Training Data |
Benchmark |
|---|---|---|---|---|---|---|---|
| Math Word Problem Solving | ASDiv-A | MMOS-DeepSeekMath-7B(0-shot) | Execution Accuracy | 87.6 | # 2 | ||
| Math Word Problem Solving | ASDiv-A | MMOS-CODE-34B(0-shot) | Execution Accuracy | 85.1 | # 4 | ||
| Math Word Problem Solving | ASDiv-A | MMOS-CODE-7B(0-shot) | Execution Accuracy | 78.6 | # 8 | ||
| Arithmetic Reasoning | GSM8K | MMOS-DeepSeekMath-7B(0-shot,k=50) | Accuracy | 87.2 | # 32 | ||
| Parameters (Billion) | 7 | # 10 | |||||
| Arithmetic Reasoning | GSM8K | MMOS-CODE-7B(0-shot) | Accuracy | 73.9 | # 84 | ||
| Parameters (Billion) | 7 | # 10 | |||||
| Arithmetic Reasoning | GSM8K | MMOS-CODE-34B(0-shot) | Accuracy | 80.4 | # 65 | ||
| Parameters (Billion) | 34 | # 72 | |||||
| Arithmetic Reasoning | GSM8K | MMOS-DeepSeekMath-7B(0-shot) | Accuracy | 80.5 | # 64 | ||
| Parameters (Billion) | 7 | # 10 | |||||
| Math Word Problem Solving | MATH | MMOS-CODE-34B(0-shot) | Accuracy | 49.5 | # 28 | ||
| Parameters (Billions) | 34 | # 26 | |||||
| Math Word Problem Solving | MATH | MMOS-DeepSeekMath-7B(0-shot,k=50) | Accuracy | 63.7 | # 6 | ||
| Parameters (Billions) | 7 | # 58 | |||||
| Math Word Problem Solving | MATH | MMOS-DeepSeekMath-7B(0-shot) | Accuracy | 55.0 | # 19 | ||
| Parameters (Billions) | 7 | # 58 | |||||
| Math Word Problem Solving | MATH | MMOS-CODE-7B(0-shot) | Accuracy | 44.3 | # 44 | ||
| Parameters (Billions) | 7 | # 58 | |||||
| Math Word Problem Solving | SVAMP | MMOS-DeepSeekMath-7B(0-shot) | Execution Accuracy | 79.3 | # 6 | ||
| Math Word Problem Solving | SVAMP | MMOS-CODE-7B(0-shot) | Execution Accuracy | 76.4 | # 7 | ||
| Math Word Problem Solving | SVAMP | MMOS-CODE-34B(0-shot) | Execution Accuracy | 80.6 | # 5 |
