LLM Model Training Using Hidden Labels

Can a large language model (LLM) teach itself to get smarter? In a new paper by Meta researchers, we developed a method we call "instruction backtranslation" to fine-tune LLMs to better follow instructions and learn more tasks. Humans are the bottleneck. A key challenge to enable LLMs to perform general instruction-following is gathering demonstration examples for finetuning. Existing high-quality instruction-following LLMs rely on human annotations in various steps, including writing instructions, writing model responses, providing preferences to indicate desired response, etc. Those instruction sets are often proprietary. Overall, the human annotation approach is difficult to scale since collecting annotations on a wide range of tasks is expensive, time consuming and requires expertise in different domains. Self-Alignment with Instruction Backtranslation Link: https://lnkd.in/g7AsHmxz ABSTRACT: We present a scalable method to build a high quality instruction following language model by automatically labelling human-written text with corresponding instructions. Our approach, named instruction backtranslation, starts with a language model finetuned on a small amount of seed data, and a given web corpus. The seed model is used to construct training examples by generating instruction prompts for web documents (self-augmentation), and then selecting high quality examples from among these candidates (self-curation). This data is then used to finetune a stronger model. Finetuning LLaMa on two iterations of our approach yields a model that outperforms all other LLaMa-based models on the Alpaca leaderboard not relying on distillation data, demonstrating highly effective self-alignment.

😅 If not properly configured, LLM judges can cause more trouble than they solve. LLM judges are quickly becoming a go-to for evaluating LLM results, cutting down on human effort. However, they must be carefully configured, either through training, proper cues, or human annotations. Here’s a nice paper from Meta that shows how to achieve this using only synthetic training data, without relying on human annotations. Some Insights: ⛳ The paper uses unlabeled instructions and prompting to generate synthetic preference pairs, where one response is intentionally made inferior to the other. ⛳ An LLM is then used to generate reasoning traces and judgments for these pairs, creating labeled data from the synthetic examples. ⛳ This labeled data is used to retrain the LLM-as-a-Judge, with the process repeated in cycles to progressively improve the model’s evaluation capabilities. ⛳ On the Llama-3-70B-Instruct model, the approach obtains accuracy on RewardBench from 75.4 to 88.7 (with majority vote) or 88.3 (without majority vote). The method matches or even outperforms traditional reward models trained on human-annotated data, demonstrating the potential of using synthetic data for model evaluation without relying on human input. Link: https://lnkd.in/eRhF4ykx

Probing-RAG: RAG approach with efficient adaptive retrieval pipeline using LLM Hidden States for Selective Document Retrieval This paper propose Probing-RAG, which utilizes hidden state representations from the intermediate layers of language models to adaptively determine the necessity of additional retrievals for a given query. 𝗠𝗲𝘁𝗵𝗼𝗱 Similar to the conventional retrieval-augmented generation pipeline, this approach comprises a generating language model and a retriever. Different from the general pipeline, the generator of Probing-RAG leverages the output from the prober and adaptively calls the retriever based on the model’s internal hidden state. i) Prober - Given the LLM’s hidden state during answer generation, the prober assesses whether an additional retrieval step is necessary - designed prober as a feed-forward network with a single hidden layer and an output layer for binary classification - prober utilizes the hidden states corresponding to the model-generated rationale (r) and answer ii) Training Prober - requires pair of two data points: input derived from hidden states, output denoting whether additional retrieval is needed - Chain-of-Thought (CoT) prompting is used to generate these pairs - final dataset consists of 26,060 training and 500 validation samples. iii) Probing based Retrieval-Augmented Generation - After generating the initial rationale and answer, the prober assesses whether retrieval is necessary. - To do this, extract hidden state representation, feed them into probers assigned to each layer to generate logit values. - If difference between logit for retrieval necessity and logit indicating no need for retrieval is higher than the threshold, additional documents are retrieved 𝗥𝗲𝘀𝘂𝗹𝘁𝘀 - Probing-RAG demonstrates best performance, with improvements of approximately 6.59% points and 8.35% points in accuracy compared to no-retrieval and single-step approaches, respectively -  Probing-RAG outperforms all of these previous adaptive retrieval methods by avoiding redundant retrieval 𝗔𝗻𝗮𝗹𝘆𝘀𝗶𝘀 - LLM-based, FLARE, Adaptive-RAG, and DRAGIN perform 1.17, 2.67, 1.54, and 6.83 times more retrieval calls, respectively, compared to Probing-RAG - prober trained on just 1k data points outperforms all of the previous methods, indicating that that it is possible to effectively train the prober using a small dataset. 𝗕𝗹𝗼𝗴: https://lnkd.in/egY7w5v9 𝗣𝗮𝗽𝗲𝗿: https://lnkd.in/eHtMBTdM