In this video I’ll give you a very quick introduction to natural language understanding and it’s applications for e-commerce search.
Natural language understanding, or NLU is formally described as machine based reading comprehension. It’s a discipline within the broader field of Artificial Intelligence, and it’s becoming increasingly important. So what is it?
Essentially Natural Language Understanding allows a system to fully understand the meaning of text. It works on entire articles, sentences or short fragments of text. Like all A.I. an NLU model is able to generalise and make predictions about text it hasn’t seen before.
At the lowest level NLU provides three core capabilities:
The first is Named Entity Recognition - Given some text, an NLU model is able to identify products, brands, sizes, prices and other entities. Next we have something known as Part of Speech tagging - which is the ability to identify nouns, verbs, adjectives in a sentence. And finally, Dependency Parsing - This is the capacity to understand how words relate to each other. As I’ll explain a little later this is really powerful.
So in the context of e-commerce search, why do we need Natural Language Understanding? Product data is already structured and users typically search using keywords and search filters. The current systems work.
Well times are changing. I want to digress a little, and look to a historical precedent. Those of you of a certain age will remember the old cell phones with numeric keypads. Writing full sentences was a pain so vendors invested heavily in predictive text technology. Consumers also changed their behaviour, adapting to the technical constraints of the time - so called “text speak was born”
Then along came the iPhone, Blackberrys and other devices with full qwerty keyboards. The previous constraints were removed, and consumer behaviour shifted. All of a sudden auto-correct replaced predictive text and emojis replaced text speak. Nokia, Motorola and others found themselves behind the curve and they weren’t unable to catch up with Apple.
The same thing is happening today with search. Users have become accustomed to using keywords and relying on search filters, not through choice but through necessity. Typing long sentences is time consuming, and anyway search engines only understand keywords and filters.
All that is changing, and once again it’s being led by the US tech giants. Google, Apple, Facebook and others are investing heavily in natural language understanding, along with speech to text technology. Consumer behaviour is already starting to change.
Let me show you a real world example, which illustrates the threat facing merchants today. Imagine I’m looking for a Gant jacket in a size medium. I’ll kid myself that I am still a size medium. Anyway here are the search results on the John Lewis site. Not particularly relevant are they. The problem is that the full text search system used by John Lewis doesn’t understand that in this context medium refers to a size. To the system, medium is just another keyword. I’m not having a pop at John Lewis because they’re one of the best e-commerce merchants out there. This scenario is likely to be repeated across most e-commerce sites.
The problem for John Lewis and other merchants is Google. Let’s perform the same search in google shopping. As you see the results are far more relevant. If I scroll down you can see that google has correctly identified the size I’m looking for. They even understand that in this context M and Medium are synonymous. Google allows me as consumer to specify exactly what I’m looking for. If typing long phrases is tedious I can just tell it what I’m looking for.
So if you’re responsible for e-commerce or digital here’s a question for you. Are you happy for consumers to drop your own search, in favour of Google. Surprisingly many merchants are, typically those with little brand recognition or a limited product catalogue. However, if this scenario does concern you, it’s time to start thinking about raising your game.
So how exactly would an e-commerce merchant apply Natural Language Understanding to search?
By really understanding what shoppers are asking for, we can offer far more relevant search results. Let me go into this in a bit more detail.
Firstly let me explain the problem with traditional full text search as employed on most e-commerce sites. Search doesn’t understand language and grammar, it works at a term or token level. Search results are ranked using the TF-IDF algorithm along with predefined field weightings. TF-IDF attaches more weight to infrequently used terms within the corpus/product catalogue and this is a problem for long tail search. Let me give you an example.
Imagine you manage a store selling outdoor clothing and equipment. A user searches for a “black packable jacket”. TF-IDF will attach more weight to documents containing the word packable than black or jacket, simply because there are likely less instances of this word in the corpus. If you sell black packable jackets this is great. But what if you don’t, but you do sell lightweight black jackets.
As humans we understand that a lightweight or summer jacket would be a good match, because ultimately the shopper wants a jacket, ideally something packable. However the TF-IDF algorithm believes the shopper is looking for something packable, ideally a jacket. So from it’s perspective a packable mosquito net is a good candidate.
Natural language understanding allows us to understand that the user is looking for a jacket first and foremost. This is possible because of something I mentioned at the beginning of this video, Named Entity Recognition. The ability to infer specific entities, without resorting to keyword matching.
Having inferred the user is looking for a jacket we can then turn to part of speech tagging and dependency parsing to understand that packable is an adjective or attribute of the product itself.
Think back to the John Lewis search results I showed earlier. Named Entity Recognition, along with dependency parsing would have allowed the system to understand that I’m looking for a jacket, not a mattress or anything else containing the word medium.
In a later video I’ll cover some of the tooling we can use to implement NLU. I’ll also explain how we can compliment traditional full text search with NLU capabilities. Stay tuned.
Do you have questions or comments about this article? Drop me a line: