This AI Paper Introduces ReasonEval: A New Machine Learning Method to Evaluate Mathematical Reasoning Beyond Accuracy

Mathematical reasoning is important for problem-solving and decision-making, significantly in massive language fashions (LLMs). Evaluating LLMs’ mathematical reasoning normally focuses on the ultimate consequence somewhat than the reasoning course of intricacies. Present methodologies, just like the OpenLLM leaderboard, primarily use total accuracy, doubtlessly overlooking logical errors or inefficient steps. Enhanced analysis approaches are essential to uncover underlying points and enhance LLMs’ reasoning.

Current approaches usually consider mathematical reasoning in LLMs by evaluating ultimate solutions with floor reality and computing total accuracy. Nonetheless, some strategies assess reasoning high quality by evaluating generated resolution steps with reference ones. Regardless of datasets offering floor reality, numerous reasoning paths to the identical reply problem reliance on any single reference. Prompting-based strategies instantly ask LLMs, typically GPT-4, to evaluate generated options, however their excessive computational price and transparency points hinder the practicality of iterative mannequin improvement.

Researchers from Shanghai Jiao Tong College, Shanghai Synthetic Intelligence Laboratory, Yale College, Carnegie Mellon College, and Generative AI Analysis Lab (GAIR) launched REASONEVAL, a brand new method to evaluating reasoning high quality past final-answer accuracy. It makes use of validity and redundancy metrics to characterize reasoning steps’ high quality, which is mechanically assessed by accompanying LLMs. REASONEVAL depends on base fashions with sturdy mathematical information, skilled on high-quality labeled information, to instantiate its analysis framework.

REASONEVAL focuses on multi-step reasoning duties, assessing the standard of reasoning past final-answer accuracy. It evaluates every reasoning step for validity and redundancy, categorizing them into optimistic, impartial, or damaging labels. Step-level scores are computed based mostly on validity and redundancy after which aggregated to generate solution-level scores. The tactic makes use of varied LLMs with totally different base fashions, sizes, and coaching methods. Coaching information is sourced from PRM800K, a dataset of labeled step-by-step options collected by human annotators.

REASONEVAL achieves state-of-the-art efficiency on human-labeled datasets and may precisely detect totally different errors generated by perturbation. It reveals that enhanced final-answer accuracy doesn’t constantly enhance the standard of reasoning steps for advanced mathematical issues. The tactic’s evaluation additionally aids in information choice. Observations spotlight vital decreases in validity scores for logical and calculation errors, whereas redundancy scores stay steady. REASONEVAL distinguishes between errors affecting validity and people introducing redundancy.

In conclusion, the analysis introduces REASONEVAL, an efficient metric for assessing reasoning step high quality based mostly on correctness and effectivity. Experimentation confirms its capability to establish numerous errors and aggressive efficiency in comparison with present strategies. REASONEVAL exposes inconsistencies between final-answer accuracy and reasoning step high quality whereas additionally proving efficient in information choice for coaching.

Try the Paper. All credit score for this analysis goes to the researchers of this challenge. Additionally, don’t overlook to comply with us on Twitter. Be part of our Telegram Channel, Discord Channel, and LinkedIn Group.

In the event you like our work, you’ll love our e-newsletter..

Don’t Overlook to hitch our 40k+ ML SubReddit