The Engineering behind LinkedIn Products "You May Like"
March 2, 2011
This is a part of our continuing series on engineering and analytics at LinkedIn. If this isn’t your cup of Java, check back tomorrow for regular LinkedIn programming. - Ed.
We are approaching the first anniversary of LinkedIn’s Recommendation Engine. Our team’s goal is to create a single platform to power all recommendations - ‘you may like this’ - on LinkedIn. A single platform eliminates the need to reinvent the wheel for:
- Extracting features from Semi Structured Data
- Message flows between different system to do real-time recommendations
- Blend Content Based, Collaborative Filtering, popularity and graph based approaches to recommendations in an algorithmic manner.
Once done this enables rapid iteration on many different recommendation products for our users, and iterate we did. Over the course of 2010, we released several different products and laid the foundation for many more to come.
Here is a quick walk-through of three of those major products that utilize the recommendation engine:
1. Groups You May Like
LinkedIn Groups provide a venue for professionals to share insights and discuss ideas relevant to their professional interests. With millions of active groups, finding groups relevant to your interests can be an exercise in keyword searching. This feature solves this problem by recommending groups relevant to you.
We accomplish this by building a virtual profile per group by selecting the most representative features of group members using Information Theory techniques like Mutual Information and KL Divergence. We then map the user’s attributes to the group’s virtual profile to generate a list of recommended groups. We top this off by adding more recommendations based on collaborative filtering.
2. Jobs You May Be Interested In
“Everyone’s looking for an opportunity, even if they don’t know it”. Wise words from a recruiter that inspired this functionality, which surfaces job recommendations based on your profile features, behavior, location and attributes of people like you. We released the product on the Jobs homepage, where it got sufficient traction that we moved it to the LinkedIn homepage.
Though the feature is still in beta, we are proud and enthused with the results thus far. "Jobs you may be interested in" continues to evolve in different ways: for e.g. it powers the “Work with us at Google” widget on the profile of Google employees (see example and screenshot below), recommending jobs to the user.
3. Talent Match
Our team's flagship product of 2010, this feature helps connect job posters, both small businesses to larger companies, with the most relevant talent. Given a job posting, there could be hundreds or thousands of potential candidates for the job.
We use a hybrid of Content and Collaborative Filtering approaches to bubble up the best candidates in real time (matches are shown before you even finish posting the job). Making this product work involved many challenges such as standardizing job titles (e.g. technical yahoo == software engineer) and standardizing company names (BING search == Microsoft); separating skills and salient phrases from resume fillers using Statistical Language Models; and cranking the prediction wheel using Logistic Regression to namea few.
The cherry on top is that this all happens in real-time. Try posting a Job @ LinkedIn and see the magic unroll.
These are just a few examples of products that speak to the power of building a common platform to rapidly iterate on data collection, modeling, and A/B testing across multiple products. Improving relevance across these multiple products will continue to remain a focus for us in 2011. But, we feel confident with the foundation we’ve set that we can extend similar features across our many products. Stay tuned for more.