Large Language Models Insights

For the last couple of years, Large Language Models (LLMs) have dominated AI, driving advancements in text generation, search, and automation. But 2025 marks a shift—one that moves beyond token-based predictions to a deeper, more structured understanding of language.  Meta’s Large Concept Models (LCMs), launched in December 2024, redefine AI’s ability to reason, generate, and interact by focusing on concepts rather than individual words.  Unlike LLMs, which rely on token-by-token generation, LCMs operate at a higher abstraction level, processing entire sentences and ideas as unified concepts. This shift enables AI to grasp deeper meaning, maintain coherence over longer contexts, and produce more structured outputs.  Attached is a fantastic graphic created by Manthan Patel How LCMs Work:  🔹 Conceptual Processing – Instead of breaking sentences into discrete words, LCMs encode entire ideas, allowing for higher-level reasoning and contextual depth.  🔹 SONAR Embeddings – A breakthrough in representation learning, SONAR embeddings capture the essence of a sentence rather than just its words, making AI more context-aware and language-agnostic.  🔹 Diffusion Techniques – Borrowing from the success of generative diffusion models, LCMs stabilize text generation, reducing hallucinations and improving reliability.  🔹 Quantization Methods – By refining how AI processes variations in input, LCMs improve robustness and minimize errors from small perturbations in phrasing.  🔹 Multimodal Integration – Unlike traditional LLMs that primarily process text, LCMs seamlessly integrate text, speech, and other data types, enabling more intuitive, cross-lingual AI interactions.  Why LCMs Are a Paradigm Shift:  ✔️ Deeper Understanding: LCMs go beyond word prediction to grasp the underlying intent and meaning behind a sentence.  ✔️ More Structured Outputs: Instead of just generating fluent text, LCMs organize thoughts logically, making them more useful for technical documentation, legal analysis, and complex reports.  ✔️ Improved Reasoning & Coherence: LLMs often lose track of long-range dependencies in text. LCMs, by processing entire ideas, maintain context better across long conversations and documents.  ✔️ Cross-Domain Applications: From research and enterprise AI to multilingual customer interactions, LCMs unlock new possibilities where traditional LLMs struggle.  LCMs vs. LLMs: The Key Differences  🔹 LLMs predict text at the token level, often leading to word-by-word optimizations rather than holistic comprehension.  🔹 LCMs process entire concepts, allowing for abstract reasoning and structured thought representation.  🔹 LLMs may struggle with context loss in long texts, while LCMs excel in maintaining coherence across extended interactions.  🔹 LCMs are more resistant to adversarial input variations, making them more reliable in critical applications like legal tech, enterprise AI, and scientific research.  

Many companies have started experimenting with simple RAG systems, probably as their first use case, to test the effectiveness of generative AI in extracting knowledge from unstructured data like PDFs, text files, and PowerPoint files. If you've used basic RAG architectures with tools like LlamaIndex or LangChain, you might have already encountered three key problems: 𝟭. 𝗜𝗻𝗮𝗱𝗲𝗾𝘂𝗮𝘁𝗲 𝗘𝘃𝗮𝗹𝘂𝗮𝘁𝗶𝗼𝗻 𝗠𝗲𝘁𝗿𝗶𝗰𝘀: Existing metrics fail to catch subtle errors like unsupported claims or hallucinations, making it hard to accurately assess and enhance system performance. 𝟮. 𝗗𝗶𝗳𝗳𝗶𝗰𝘂𝗹𝘁𝘆 𝗛𝗮𝗻𝗱𝗹𝗶𝗻𝗴 𝗖𝗼𝗺𝗽𝗹𝗲𝘅 𝗤𝘂𝗲𝘀𝘁𝗶𝗼𝗻𝘀: Standard RAG methods often struggle to find and combine information from multiple sources effectively, leading to slower responses and less relevant results. 𝟯. 𝗦𝘁𝗿𝘂𝗴𝗴𝗹𝗶𝗻𝗴 𝘁𝗼 𝗨𝗻𝗱𝗲𝗿𝘀𝘁𝗮𝗻𝗱 𝗖𝗼𝗻𝘁𝗲𝘅𝘁 𝗮𝗻𝗱 𝗖𝗼𝗻𝗻𝗲𝗰𝘁𝗶𝗼𝗻𝘀: Basic RAG approaches often miss the deeper relationships between information pieces, resulting in incomplete or inaccurate answers that don't fully meet user needs. In this post I will introduce three useful papers to address these gaps: 𝟭. 𝗥𝗔𝗚𝗖𝗵𝗲𝗸𝗲𝗿: introduces a new framework for evaluating RAG systems with a focus on fine-grained, claim-level metrics. It proposes a comprehensive set of metrics: claim-level precision, recall, and F1 score to measure the correctness and completeness of responses; claim recall and context precision to evaluate the effectiveness of the retriever; and faithfulness, noise sensitivity, hallucination rate, self-knowledge reliance, and context utilization to diagnose the generator's performance. Consider using these metrics to help identify errors, enhance accuracy, and reduce hallucinations in generated outputs. 𝟮. 𝗘𝗳𝗳𝗶𝗰𝗶𝗲𝗻𝘁𝗥𝗔𝗚: It uses a labeler and filter mechanism to identify and retain only the most relevant parts of retrieved information, reducing the need for repeated large language model calls. This iterative approach refines search queries efficiently, lowering latency and costs while maintaining high accuracy for complex, multi-hop questions. 𝟯. 𝗚𝗿𝗮𝗽𝗵𝗥𝗔𝗚: By leveraging structured data from knowledge graphs, GraphRAG methods enhance the retrieval process, capturing complex relationships and dependencies between entities that traditional text-based retrieval methods often miss. This approach enables the generation of more precise and context-aware content, making it particularly valuable for applications in domains that require a deep understanding of interconnected data, such as scientific research, legal documentation, and complex question answering. For example, in tasks such as query-focused summarization, GraphRAG demonstrates substantial gains by effectively leveraging graph structures to capture local and global relationships within documents. It's encouraging to see how quickly gaps are identified and improvements are made in the GenAI world.

If you’re an AI engineer trying to optimize your LLMs for inference, here’s a quick guide for you 👇 Efficient inference isn’t just about faster hardware, it’s a multi-layered design problem. From how you compress prompts to how your memory is managed across GPUs, everything impacts latency, throughput, and cost. Here’s a structured taxonomy of inference-time optimizations for LLMs: 1. Data-Level Optimization Reduce redundant tokens and unnecessary output computation. → Input Compression:  - Prompt Pruning, remove irrelevant history or system tokens  - Prompt Summarization, use model-generated summaries as input  - Soft Prompt Compression, encode static context using embeddings  - RAG, replace long prompts with retrieved documents plus compact queries → Output Organization:  - Pre-structure output to reduce decoding time and minimize sampling steps 2. Model-Level Optimization (a) Efficient Structure Design → Efficient FFN Design, use gated or sparsely-activated FFNs (e.g., SwiGLU) → Efficient Attention, FlashAttention, linear attention, or sliding window for long context → Transformer Alternates, e.g., Mamba, Reformer for memory-efficient decoding → Multi/Group-Query Attention, share keys/values across heads to reduce KV cache size → Low-Complexity Attention, replace full softmax with approximations (e.g., Linformer) (b) Model Compression → Quantization:  - Post-Training, no retraining needed  - Quantization-Aware Training, better accuracy, especially <8-bit → Sparsification:  - Weight Pruning, Sparse Attention → Structure Optimization:  - Neural Architecture Search, Structure Factorization → Knowledge Distillation:  - White-box, student learns internal states  - Black-box, student mimics output logits → Dynamic Inference, adaptive early exits or skipping blocks based on input complexity 3. System-Level Optimization (a) Inference Engine → Graph & Operator Optimization, use ONNX, TensorRT, BetterTransformer for op fusion → Speculative Decoding, use a smaller model to draft tokens, validate with full model → Memory Management, KV cache reuse, paging strategies (e.g., PagedAttention in vLLM) (b) Serving System → Batching, group requests with similar lengths for throughput gains → Scheduling, token-level preemption (e.g., TGI, vLLM schedulers) → Distributed Systems, use tensor, pipeline, or model parallelism to scale across GPUs My Two Cents 🫰 → Always benchmark end-to-end latency, not just token decode speed → For production, 8-bit or 4-bit quantized models with MQA and PagedAttention give the best price/performance → If using long context (>64k), consider sliding attention plus RAG, not full dense memory → Use speculative decoding and batching for chat applications with high concurrency → LLM inference is a systems problem. Optimizing it requires thinking holistically, from tokens to tensors to threads. Image inspo: A Survey on Efficient Inference for Large Language Models ---- Follow me (Aishwarya Srinivasan) for more AI insights!

Chain-of-Thought has been a fundamental architecture driving LLM performance. Now 'Chain of Continuous Thought' (Coconut) significantly improves reasoning performance through working in latent space rather than language space. This paper from Meta's AI research group lays out the logic and results: 💡 Continuous Reasoning Unlocks Efficiency: Large Language Models (LLMs) traditionally reason in "language space," where reasoning steps are expressed as explicit tokens, leading to inefficiencies. The Coconut (Chain of Continuous Thought) paradigm instead reasons in a continuous latent space by feeding the model’s hidden state back as input. This reduces reliance on explicit tokens and improves reasoning efficiency, especially for complex tasks requiring backtracking. 📊 Higher Accuracy in Complex Reasoning Tasks: Coconut achieves significant accuracy improvements on complex tasks requiring planning and logic. In ProsQA, a reasoning-intensive task, Coconut attains 97.0% accuracy, far exceeding Chain-of-Thought (CoT) at 77.5%. Similarly, in logical reasoning tasks like ProntoQA, it achieves near-perfect performance at 99.8% accuracy, outperforming or matching other baselines while demonstrating superior planning capabilities. ⚡ Greater Efficiency with Fewer Tokens: Coconut enhances reasoning efficiency by reducing the number of generated tokens while maintaining accuracy. For example, in GSM8k (math reasoning), Coconut achieves 34.1% accuracy using just 8.2 tokens, compared to CoT's 42.9% accuracy which requires 25 tokens. This token efficiency indicates that reasoning in latent space allows the model to process fewer explicit steps without sacrificing performance. 🌟 Parallel Reasoning Explores Multiple Alternative Steps: Coconut enables LLMs to simultaneously explore multiple reasoning paths by encoding alternative next steps in the continuous latent space. This parallel reasoning behavior mimics breadth-first search (BFS), allowing the model to avoid premature decisions and progressively narrow down the correct solution. 🔄 Multi-Stage Training Accelerates Learning: Coconut leverages a curriculum-based training strategy, where the reasoning chain is gradually replaced with latent thoughts. This phased approach facilitates model learning, improving performance on math problems (GSM8k) and logical tasks, outperforming baselines like No-CoT and iCoT. 🔍 Latent Reasoning Improves Planning and Focus: By reasoning in latent space, the model avoids premature decisions and progressively narrows down possibilities. Coconut shows reduced hallucinations and improved accuracy compared to CoT, demonstrating its ability to prioritize promising reasoning paths while pruning irrelevant ones. New model architectures are consistently improving LLM performance and efficiency. Even without more training data and underlying model progress we are seeing consistent advances. Link to paper in comments.

Are you tired of LLMs providing you generic answers with little or sometimes no contextual alignment and thus poor domain adoption? University of California, Berkeley researchers have pioneered a novel training method, RAFT (Retrieval Augmented Fine-Tuning), which bolsters a language model’s ability to respond to domain-specific queries using an “open-book” technique. This prompts us to ponder - Do language models truly comprehend and infer from provided documents, or do they simply memorize and echo information? The Challenge of Domain-Specific Question Answering - The task of tailoring large language models (LLMs) to respond to queries in specialized areas, such as biomedical research or API documentation, is an expanding yet demanding endeavor. Conventional techniques encompass:-  👉Retrieval-augmented generation (RAG): Supplying pertinent documents to the model during inference.  👉Supervised fine-tuning on domain-specific data. 📍Nonetheless, RAG in isolation doesn’t fully leverage the potential for in-domain learning, while standard fine-tuning doesn’t coach the model to effectively utilize retrieved documents. 💫The Fusion of the Best Approaches. RAFT overcomes these shortcomings by fine-tuning the model to respond to queries using a blend of relevant & irrelevant documents. Its key attributes include:- 👩🏽💻Training on a mix of question-document pairs, some with the “oracle” document that holds the answer and some with only “distractor” documents.  Generating responses in a chain-of-thought style that cites the pertinent sections of the reference documents. ✨Remarkable Outcomes Across Diverse Domains - The researchers tested RAFT on multiple question-answering datasets, covering Wikipedia articles, biomedical papers, and API documentation. In all these specialized domains, RAFT consistently surpassed both standard fine-tuning and RAG benchmarks. 🌟Significantly, RAFT achieved substantial improvements of up to 35% on the HotpotQA Wikipedia dataset and 76% on the Torch Hub API documentation dataset compared to the base RAG model. These outcomes validate RAFT’s capacity to genuinely comprehend and infer from domain-specific documents. ⚡️Way forward towards Efficient Domain Adaptation - RAFT represents a thrilling progression towards more proficient and effective customization of language models to specialized domains. By learning to selectively read and cite pertinent information from domain-specific documents, RAFT lays the groundwork for compact, dedicated models that can compete with much larger generic language models on niche question-answering tasks. As the need for deploying LLMs to domain-specific applications continues to surge, methods like RAFT is likely to be vital for facilitating practical, cost-efficient solutions! Kudos to Tianjun Zhang Shishir Patil @Naman Jain Sheng Shen Matei Zaharia Ion Stoica Joseph E. Gonzalez for this amazing work! #llm #ai #aiadoption #genai

A simple prompt hack that makes reasoning models 3x more effective has been hiding in plain sight - and it's not "think step by step." While many developers instinctively add chain-of-thought prompts to their LLM calls, new research shows this actually degrades performance with modern reasoning models like OpenAI's o1 and DeepSeek's recent R1. Key insights on effective prompting: (1) Developer-first architecture - modern reasoning models like o3 use a new paradigm where instructions are passed as what's called "developer messages" rather than system prompts, enabling more precise control over model behavior (2) Zero-shot superiority - these models often perform better without examples, contrary to traditional LLMs (3) Constraint clarity - explicitly outline limitations (e.g., "solution under $500") instead of asking for step-by-step thinking (4) Structured input control - use delimiters (XML tags, markdown, section headers) to organize your prompts, significantly improving the model's understanding of distinct input components (5) Markdown handling - newer versions of reasoning models require explicit activation of markdown formatting through a simple header flag, so don't expect markdown to be generated otherwise My take: Clearly, reasoning models operate differently from traditional LLMs such as Claude Sonnet and GPT-4o. Comparing these models is like judging a coding model, such as Mistral’s Codestral, by its ability to write poetry—it’s an apples-to-oranges comparison. This isn't just about better prompts - it's about fundamentally rethinking how we interact with AI systems designed for complex reasoning.

Generative AI cannot write what hasn’t been written. This is a subtle but profound truth the tech industry is just fully coming to understand. Ask a question of any large language model – from ChatGPT to Grok go Gemini – and you notice that LLMs cannot deliver reliable answers for topics that haven’t been covered. Gen AI programs work well when you are researching big companies like Nvidia and famous people like Tim Cook but not at all for the obscure or unsung. Since there is no way humans can keep up, the somewhat counterintuitive solution to this problem is to tap Gen AI to create a vast library of digital content. Effectively, we need machines to write articles so other machines can read them. To illustrate the challenge, consider Perplexity, an AI search engine, which is building a finance vertical. The biggest challenge won’t be the speed or depth of the LLMs that they leverage, but the lack of historical news about companies. In many cases, these are small or mid-sized companies that were never covered by reporters at Bloomberg or Reuters or the New York Times. Remember: Generative AI cannot write what hasn’t been written. The importance of specialized content was driven home by the recent announcement by DeepSeek, a Chinese LLM, when it said it had unveiled a high-performing open source large language model at a fraction of the cost of the version created by OpenAI. The DeepSeek announcement signaled that model enhancements will continue to come fast and furious, each leap-frogging the previous one and driving down inference costs.  Developers seeking to build moats around their applications will rely less on model performance and more and more on the quality, reliability and comprehensiveness of content. 

Large Language Models face a critical challenge: how to enhance factual accuracy without sacrificing either inference speed or general capabilities. Current solutions fall short-RAG systems suffer from high latency and shallow integration, while fine-tuning methods like LoRA risk catastrophic forgetting. Researchers from Shanghai Jiao Tong University and Shanghai AI Laboratory Lab propose MLP Memory, a parametric memory module that learns retrieval patterns during pretraining without requiring explicit document access at inference time. How it works: The system trains a lightweight MLP network to mimic the behavior of k-nearest neighbor retrieval across an entire pretraining corpus. During training, the MLP learns to map hidden representations from a frozen language model to probability distributions that match what a kNN retriever would produce-essentially compressing 40TB of datastore information into a 4GB parametric module. The architecture uses stacked feed-forward layers without token-mixing operations, leveraging recent findings that FFN layers function as key-value memories within transformers. The training objective combines KL divergence loss to match retrieval distributions with cross-entropy loss to maintain grounding in actual next-token predictions. At inference, the MLP Memory processes hidden states from approximately 70% network depth (not the final layer, as conventional kNN-LM does) and interpolates its output with the base model's predictions through simple probability mixing. Performance gains: On question-answering benchmarks, MLP Memory achieves 12.3% relative improvement over base models, outperforming both RAG and continued pretraining. On HaluEval, it reduces hallucinations by up to 10 points. Critically, it delivers 2.5x faster time-to-first-token than RAG and maintains constant inference speed regardless of corpus size-a fundamental advantage over retrieval-based methods whose latency scales with datastore size. The approach demonstrates that learning retrieval patterns parametrically bridges the efficiency-effectiveness gap, offering a practical alternative that combines the knowledge access benefits of RAG with the speed of purely parametric methods.

GenAI’s black box problem is becoming a real business problem. Large language models are racing ahead of our ability to explain them. That gap (the “representational gap” for the cool kids) is no longer just academic, and is now a #compliance and risk management issue. Why it matters: • Reliability: If you can’t trace how a model reached its conclusion, you can’t validate accuracy. • Resilience: Without interpretability, you can’t fix failures or confirm fixes. • Regulation: From the EU AI Act to sector regulators in finance and health care, transparency is quickly becoming non-negotiable. Signals from the frontier: • Banks are stress-testing GenAI the same way they test credit models, using surrogate testing, statistical analysis, and guardrails. • Researchers at firms like #Anthropic are mapping millions of features inside LLMs, creating “control knobs” to adjust behavior and probes that flag risky outputs before they surface. As AI shifts from answering prompts to running workflows and making autonomous decisions, traceability will move from optional to mandatory. The takeaway: Interpretability is no longer a nice-to-have. It is a license to operate. Companies that lean in will not only satisfy regulators but also build the trust of customers, partners, and employees. Tip of the hat to Alison Hu Sanmitra Bhattacharya, PhD, Gina Schaefer, Rich O'Connell and Beena Ammanath's whole team for this great read.

AI models are reasoning, creating, and evolving. The evidence is no longer theoretical; it's peer-reviewed, measurable, and, in some domains, superhuman. In the last 18 months, we’ve seen LLMs move far beyond next-token prediction. They’re beginning to demonstrate real reasoning, hypothesis generation, long-horizon planning, and even scientific creativity. Here are six breakthroughs that redefine what these models can do: Superhuman Clinical Reasoning (Nature Medicine, 2025) In a rigorous test across 12 specialties, GPT-4 scored 89% on the NEJM Knowledge+ medical reasoning exam, outperforming the average physician score of 74%. This wasn’t just Q&A; it involved multi-hop reasoning, risk evaluation, and treatment planning. That’s structured decision-making in high-stakes domains. Creative Research Ideation (Zhou et al., 2024 – arXiv:2412.10849) Across 10 fields from physics to economics, GPT-4 and Claude generated research questions rated more creative than human-generated ones in 53% of cases. This wasn’t trivia; domain experts blindly compared ideas from AI and researchers. In over half the cases, the AI won. Falsifiable Hypotheses from Raw Data (Nemati et al., 2024) GPT-4o was fed raw experimental tables from biology and materials science and asked to propose novel hypotheses. 46% of them were judged publishable by experts, outperforming PhD students (29%) on the same task. That’s not pattern matching, that’s creative scientific reasoning from scratch. Self-Evolving Agents (2024) LLM agents that reflect, revise memory, and re-prompt themselves improved their performance on coding benchmarks from 21% → 34% in just four self-corrective cycles, without retraining. This is meta-cognition in action: learning from failure, iterating, and adapting over time. Long-Term Agent Memory (A-MEM, 2025) Agents equipped with dynamic long-term memory (inspired by Zettelkasten) achieved 2× higher success on complex web tasks, planning across multiple steps with context continuity. Emergent Social Reasoning (AgentSociety, 2025) In a simulation of 1,000 LLM-driven agents, researchers observed emergent social behaviors: rumor spreading, collaborative planning, and even economic trade. No hardcoding. Just distributed reasoning, goal propagation, and learning-by-interaction. These findings span healthcare, science, software engineering, and multi-agent simulations. They reveal systems that generate, reason, and coordinate, not just predict. So when some argue that “AI is only simulating thought,” we should ask: Are the tests capturing how real reasoning happens? The Tower of Hanoi isn’t where science, medicine, or innovation happens. The real test is: 1. Can a model make a novel discovery? 2. Can it self-correct across steps? 3. Can it outperform domain experts in structured judgment? And increasingly, the answer is: yes. Let’s not confuse symbolic puzzles with intelligence. Reasoning is already here, and it’s evolving.