Recommender system basics

The advance of artificial intelligence is super exciting and also somewhat scary. More and more, AI is taking over defining how we interact with computers and the people connected with us. One of the most used types of AI is the recommender system. Recommender systems are everywhere. Working behind the scenes, they recommend products, music and films.

They make algorithmic timelines on social platforms and decide what ads we see. Chances are, you used one to discover this post. Or was a recommender using you?

What recommender systems are

‘RecSys’ engineers are just like designers who can’t agree on what UX, IxD and design actually mean. Wikipedia limits the definition to systems that predict ratings and preferences. Yanir Seroussi, in his article about the technical basics of recommender systems says: “Recommender is a misnomer—calling it a discovery assistant is better.”

Based on his ideas, I’ve settled on this working definition:

Recommender systems present people items that are best guesses for what they may need, like, want to consume or interact with.

Those items can be anything from banner ads to groceries and partners for a date.

Recommender systems are applications of data science methods like machine learning. Simple rules like “You’re visiting this page from Germany. Do you want to view this page in German?” may technically be a system too, but are not considered a recommender system.

Recommender systems are not about search results either. Search is another type of filtering problem altogether. With search, users actively define specific criteria for the items they’re looking for. Where good search results are the end of a search, good recommendations can be the start of a new discovery phase.

There are no ‘perfect’ songs, movies or vacuum cleaners. And even if there were, users would find it hard to describe them in a search query. Recommenders aren’t about finding perfect matches—They’re about finding great matches and surprising users with things they don’t know yet.

How recommenders work

The two basic approaches to filtering items for recommendations are item-based and user-based. 

Item-based filtering

This approach uses the properties of the items of which the system knows that a user likes them. Properties can be anything, like genre, author and price. Example: You bought a red and a blue medium-sized cotton T-shirt, so maybe you like a black medium-sized cotton T-shirt?

User-based filtering

With this approach, the recommender takes behavior and preferences of users, like purchases and ratings as the input. With that, it finds users with similar preferences and recommends what they like. Example: People who like Taylor Swift, Charli XCX and Troye Sivan, also like Kim Petras and Rob the Heart.

(That’s what Gnoosic told me.)

Hybrid filtering

Typically, user-based recommendations work better than item-based recommendations. With a few ratings, they provide more relevant recommendations, because they include items that the user may never have heard of before.

User-based filtering has an important limitation though. Let’s say my taste in literature overlaps very well with people who like the original Don Quixote. That doesn’t necessarily make it a good recommendation for me; I can’t read Spanish. The best recommenders combine user-based filtering (ratings) with item-based filtering (language).

A good recommender doesn’t need users to be explicit about their preferences. Such a system also collects indicators for what users prefer. These indicators can be whether or not you skip a song before it ends or how often you buy a certain product. Facebook’s shadow profiles are an extreme example of how having a lot of different types of data can make people’s preferences predictable.

What you need to make a recommender work well

Recommender systems can compare many different properties of users (like age, time spent online, location) and items (style, weight, price). The weight that the recommender assigns to each depends on what the recommendation is for.

For music, it’s likely that users of a similar age have similar taste, but the color of the album cover is likely less relevant. For graphic prints it can be the other way around. There are many item-based and user-based factors a recommender system can take into consideration. For most applications, it would be infeasible to manually define weights for these factors. But that’s precisely the kind of problem machine learning algorithms can solve.

Data

Like with any machine learning project, a recommender needs loads of data. The more high-quality data, the better. High quality means, among other things:

  • It’s unbiased
  • Represents a good part (or all) of the user base
  • Reliable indicators for user preferences

Only a minority of users may leave a review on your store, but the system can still track what they buy frequently. Similarly, liking a post on a social platform can indicate preference, but the time spent reading a linked article too.

Understanding of users and the domain

Although the algorithms used for recommenders may be similar between platforms, they vary in how they’re applied. Without thorough understanding of your target group and the items that are recommended, it’s hard to create a recommender system and even harder to evaluate it.

Here are some questions that your team should be able to answer:

  • Does it make sense to recommend an item that the user knows already? Not so much when they just bought a washing machine, but it could be helpful to be reminded of a movie one watched years ago.
  • How should the recommender handle diversity of users? Does it take demographics into account (possibly through proxies, like preference for pink or blue-colored products)? Does it reinforce stereotypes?
  • What’s the right amount of serendipity in the recommendations? Looking for new discoveries, similarity to known items can be good, but also get boring.
  • How much do you want to promote freshness of items? New things may excite users initially and make the recommender appear to perform well. Too much of it can be alienating, making users less likely to return to your platform.

What this means for your product

Any app or website that contains a list of items (that’s nearly all of them!) has the potential to get better with recommendations. Provided there’s enough data to make a guess about what the user is looking for, many lists can be accompanied by recommendations. A photo album could infer what photos the user would like to see based on ratings and the content of the photos. Shopping in brick and mortar stores is all about getting recommendations—online shopping can learn from that.

When working on new products and features, it’s essential to keep in mind that these serve users. Designers should be involved in that, as they know the target group, their challenges, goals and desires. Therefore, its important designers are aware of the possibilities. This post should help designers go beyond black box thinking and get an impression of what’s possible.

The following pages helped me writing these posts. I recommend them for more details:

Want to know more concretely how recommenders can be applied? Read What designers should know about recommender systems.