Ben Burtenshaw’s Post

🔥 Want to know more about a new technique that combines instruction tuning and reinforcement learning into one step to aligning LLMs to follow the behaviour you want? Read our latest blog with Argilla 👇

🔎 What if you only need a base model for preference alignment? Welcome to ORPO, summarized in our latest blog with MantisNLP. ORPO eliminates the complexity of multi-stage RLHF and leverages the power of an odds ratio-based loss approach. The result? Better performance and significant savings in resources and memory! 🌟 And the cherry on top? To showcase this, one of our favorite datasets, Argilla's UltraFeedback Binarized, was used.

Today we presented our paper “The Skipped Beat: A Study of Sociopragmatic Understanding in LLMs for 64 Languages”. Work by Chiyu Zhang, Khi Duy Doan, and Qisheng Liao at #EMNLP2023 Preprint:

This is a great example of what an application of neurosymbolic principles in practice will look like. The paper it's based on, too, does a great job to treat neurosymbolic principles as an abstract, and one whose implementation is not the same as its concept. The questions raised here are really exciting: what might be gained and lost by representing knowledge as AMR? What would it look like for AMR to be learned automatically, that is, for an AMR-based NeSy engine to be adaptable to new syntactic and semantic forms? Does the AMR of a sentence change based on the domain context that the reader is primed to understand? Or is it objective? https://lnkd.in/gUqsdZN7

Happy to share our work on large language model selection using random matrix theory. Please check the two-minute KDD 2023 video: https://lnkd.in/ePSxcDQX The take-home message is that one can measure the weight matrices of LLM models (without data) to predict their generalizability. Previous papers in this field mostly focus on predicting the "generalization gap", but that metric does not tell much about the quality of LLMs. Our paper shows that generalization metrics motivated by the "heavy-tail self-regularization" theory significantly outperform those focused on the generalization gap. #llm #generalization #randommatrix #KDD2023 Full paper: https://lnkd.in/euDzYDin

Excellent presentation from Andrej Karpathy from last month.  Many insights and intuition. - A fantastic explanation of how Reward Modeling and Reinforcement Learning on LLMs works. - The Retrieval (web search) vs. Memory (LLMs) spectrum. The best systems will be in the middle. - System 1 vs. System 2. LLMs are good at System 1; we’re still learning how to teach them System 2 thinking. - “Mode collapse”: Fine-tuned models lose entropy (and thus diversity that leads to “magic”). - RLHF: we don’t know why it works, but his intuition is because it’s easier to discriminate than generate. - RLHF is still in the research phase. Prefer SFT (supervised fine-tuning) as your default for optimizing. https://lnkd.in/eQmH6c3w