How to Build Your Own Search Ranking Algorithm with Machine Learning
“Any sufficiently advanced technology is indistinguishable from magic.” – Arthur C. Clarke (1961)
This quote couldn’t apply better to general search engines and web ranking algorithms.
Think about it.
You can ask Bing about mostly anything and you’ll get the best 10 results out of billions of webpages within a couple of seconds. If that’s not magic, I don’t know what is!
Sometimes the query is about an obscure hobby. Sometimes it’s about a news event that nobody could have predicted yesterday.
Sometimes it’s even unclear what the query is about! It all doesn’t matter. When users enter a search query, they expect their 10 blue links on the other side.
To solve this hard problem in a scalable and systematic way, we made the decision very early in the history of Bing to treat web ranking as a machine learning problem.
As early as 2005, we used neural networks to power our search engine and you can still find rare pictures of Satya Nadella, VP of Search and Advertising at the time, showcasing our web ranking advances.
This article will break down the machine learning problem known as Learning to Rank. And if you want to have some fun, you could follow the same steps to build your own web ranking algorithm.
Why Machine Learning?
A standard definition of machine learning is the following:
“Machine learning is the science of getting computers to act without being explicitly programmed.”
At a high level, machine learning is good at identifying patterns in data and generalizing based on a (relatively) small set of examples.
For web ranking, it means building a model that will look at some ideal SERPs and learn which features are the most predictive of relevance.
This makes machine learning a scalable way to create a web ranking algorithm. You don’t need to hire experts in every single possible topic to carefully engineer your algorithm.
Instead, based on the patterns shared by a great football site and a great baseball site, the model will learn to identify…
Read More Here