LinkedInS Next-Gen Recommendation System: Why Prompting Wasn’t the Answer
LinkedIn, a pioneer in AI-powered recommendation systems with over 15 years of experience, faced a important challenge in building a next-generation system for job seekers. Achieving the required accuracy,latency,and efficiency demanded a novel approach,one that moved beyond relying on readily available models and prompting techniques.
The Limitations of prompting
According to erran Berger, VP of Product Engineering at LinkedIn, prompting was immediately dismissed as a viable solution for their next-gen recommender systems. “There was just no way we were gonna be able to do that through prompting,” Berger stated in a recent Beyond the Pilot podcast. “We didn’t even try that… because we realized it was a non-starter.”
A New Approach: Fine-Tuning and Distillation
Instead of prompting, LinkedIn’s team focused on a meticulous process of fine-tuning and model distillation.This involved:
- Developing a complete product policy document to guide the AI’s behavior.
- Starting with a large 7-billion-parameter model.
- Distilling this model into smaller “teacher” and “student” models,optimized to hundreds of millions of parameters.
This distillation process allowed LinkedIn to create models that were both highly accurate and efficient, addressing the key challenges they faced.
A Repeatable Framework for AI Products
The success of this technique has led to the creation of a standardized, repeatable framework.This “cookbook” is now being leveraged across various AI products within LinkedIn, promising widespread quality improvements.
“Adopting this eval process end to end will drive substantial quality improvement of the likes we probably haven’t seen in years,” Berger explained.
Key Takeaways
- Prompting alone is insufficient for building highly sophisticated recommender systems.
- Fine-tuning large models and distilling them into smaller,optimized versions is a powerful technique.
- A well-defined product policy is crucial for guiding AI behavior.
- Establishing a repeatable framework can accelerate AI growth and improve quality across an organization.
linkedin’s experience demonstrates that achieving truly next-generation AI capabilities often requires a commitment to custom development and a willingness to move beyond off-the-shelf solutions. As AI continues to evolve, we can expect to see more organizations adopting similar strategies to unlock the full potential of this transformative technology.
