Improving LLM Generalization Through Self-Certainty

🤔 What if, instead of using prompts, you could fine-tune LLMs to incorporate self-feedback and improvement mechanisms more effectively? Self-feedback and improvement have been shown to be highly beneficial for LLMs and agents, allowing them to reflect on their behavior and reasoning and correct their mistakes as more computational resources or interactions become available. The authors mention that frequently used test-time methods like prompt tuning and few-shot learning that are used for self-improvement, often fail to enable models to correct their mistakes in complex reasoning tasks. ⛳ The paper introduces RISE: Recursive Introspection, an approach to improve LLMs by teaching them how to introspect and improve their responses iteratively. ⛳ RISE leverages principles from online imitation learning and reinforcement learning to develop a self-improvement mechanism within LLMs. By treating each prompt as part of a multi-turn Markov decision process (MDP), RISE allows models to learn from their previous attempts and refine their answers over multiple turns, ultimately improving their problem-solving capabilities. ⛳It models the fine-tuning process as a multi-turn Markov decision process, where the initial state is the prompt, and subsequent states involve recursive improvements. ⛳It employs a reward-weighted regression (RWR) objective to learn from both high- and low-quality rollouts, enabling models to improve over turns. The approach uses data generated by the learner itself or more capable models to supervise improvements iteratively. RISE significantly improves the performance of LLMs like LLaMa2, LLaMa3, and Mistral on math reasoning tasks, outperforming single-turn strategies with the same computational resources. Link: https://lnkd.in/e2JDQr8M

Groundbreaking Research Alert: Rethinking Adaptive Retrieval in Large Language Models A comprehensive study by researchers from Skolkovo Institute of Science and Technology, AIRI, and other leading institutions has revealed fascinating insights about adaptive retrieval methods in LLMs. The study analyzed 35 different approaches, including 8 recent methods and 27 established uncertainty estimation techniques, across 6 diverse datasets. Key Technical Insights: - The research shows that simple uncertainty estimation methods often outperform complex retrieval pipelines while being significantly more compute-efficient. - Internal-state based uncertainty methods excel at simple tasks, while reflexive methods perform better on complex reasoning tasks. The study found that SeaKR demonstrates strong self-knowledge identification on single-hop datasets by inspecting LLM internal states. Under the Hood: - The study implements a hybrid approach combining multiple uncertainty features, including logit-based, consistency-based, and internal-based methods. - Researchers used LLaMA 3.1-8b-instruct model with BM25 retriever and Wikipedia corpus for evaluation. - The analysis covered 10 different metrics across QA performance, self-knowledge capabilities, and computational efficiency. Notable Findings: - Uncertainty methods achieve comparable performance to recent adaptive retrieval approaches while requiring fewer compute resources. - The study revealed that consistency-based methods excel in downstream performance but lag in self-knowledge assessment. - The research identified a significant gap between ideal and current uncertainty estimators, highlighting room for improvement. This work represents a significant step forward in understanding how to balance between LLMs' intrinsic knowledge and external information retrieval, potentially leading to more efficient and accurate AI systems.

LLMs can now train themselves to reason better – without any external judges or reward models. But there's a problem: teaching models to improve their reasoning on open-ended tasks like document revision has been nearly impossible. Unlike math problems where answers are clearly right or wrong, how do you score a thoughtful paragraph revision? Microsoft researchers just cracked this puzzle with Direct Reasoning Optimization (DRO). The challenge they tackle is fundamental: while we've made huge strides teaching LLMs to solve math and code (where correctness is verifiable), extending these techniques to tasks like revising documents, writing analytical reports, or crafting responses to complex feedback has remained elusive. Their solution is elegantly simple yet powerful. Instead of building separate reward models or using other LLMs as judges, DRO lets the model evaluate its own reasoning by measuring how confident it is about producing the correct outcome. But here's the clever part – they discovered that only certain "reasoning-reflective tokens" actually matter. These are the specific words whose likelihood changes dramatically based on the quality of the preceding reasoning. By focusing the reward signal on these key tokens rather than averaging across all tokens, the model learns to genuinely improve its logical thinking. On scientific paper revision tasks, their approach outperformed GPT-4 despite using a much smaller model, while reducing training costs by 45%. Even more impressively, it achieved performance comparable to having a perfect verifier on financial QA tasks – all without any external validation. This might enable a future where LLMs can continuously improve their reasoning on complex, open-ended tasks – from legal document analysis to creative writing – without needing armies of human evaluators or specialized reward models. ↓ 𝐖𝐚𝐧𝐭 𝐭𝐨 𝐤𝐞𝐞𝐩 𝐮𝐩? Join my newsletter with 50k+ readers and be the first to learn about the latest AI research: llmwatch.com 💡

Exciting new research from Anthropic introduces Internal Coherence Maximization (ICM), a breakthrough method that allows #largelanguagemodels (LLMs) to #finetune themselves using only their own outputs - potentially reducing or even replacing the need for human oversight in complex tasks. The model evaluates the consistency of its own responses and optimizes itself by comparing and correcting inconsistent statements. In benchmarks such as TruthfulQA and GSM8K, ICM achieved similar or better results than models with classic supervised fine-tuning. https://lnkd.in/d3Kbdsts