Google is improving 10 percent of searches by understanding language context

You can foun additiona information about ai customer service and artificial intelligence and NLP. It’s our job to figure out what you’re searching for and surface helpful information from the web, no matter how you spell or combine the words in your query. While we’ve continued to improve our language understanding capabilities over the years, we sometimes still don’t quite get it right, particularly with complex or conversational queries. In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question. Google is currently rolling out a change to its core search algorithm that it says could change the rankings of results for as many as one in ten queries.

Particularly for longer, more conversational queries, or searches where prepositions like “for” and “to” matter a lot to the meaning, Search will be able to understand the context of the words in your query. To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence. However, it is not possible to train bidirectional models by simply conditioning each word on its previous and next words, since this would allow the word that’s being predicted to indirectly “see itself” in a multi-layer model.

What Makes BERT Different?

Since BERT is trained on a giant corpus of English sentences, which are also inherently biased, it’s an issue to keep an eye on. BERT builds upon recent work in pre-training contextual representations — including Semi-supervised Sequence Learning, algorithme nlp Generative Pre-Training, ELMo, and ULMFit. However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).

• Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional.
• However, unlike these previous models, BERT is the first deeply bidirectional, unsupervised language representation, pre-trained using only a plain text corpus (in this case, Wikipedia).
• Any time Google signals a change in its search algorithm, the entire web sits up and takes notice.
• Google is currently rolling out a change to its core search algorithm that it says could change the rankings of results for as many as one in ten queries.
• So we can take models that learn from improvements in English (a language where the vast majority of web content exists) and apply them to other languages.

Here are some of the examples that showed up our evaluation process that demonstrate BERT’s ability to understand the intent behind your search. For featured snippets, we’re using a BERT model to improve featured snippets in the two dozen countries where this feature is available, and seeing significant improvements in languages like Korean, Hindi and Portuguese. If there’s one thing I’ve learned over the 15 years working on Google Search, it’s that people’s curiosity is endless. We see billions of searches every day, and 15 percent of those queries are ones we haven’t seen before–so we’ve built ways to return results for queries we can’t anticipate.

Understanding searches better than ever before

We’re always getting better and working to find the meaning in– and most helpful information for– every query you send our way.

The open source release also includes code to run pre-training, although we believe the majority of NLP researchers who use BERT will never need to pre-train their own models from scratch. The BERT models that we are releasing today are English-only, but we hope to release models which have been pre-trained on a variety of languages in the near future. Everything that we’ve described so far might seem fairly straightforward, so what’s the missing piece that made it work so well? Cloud TPUs gave us the freedom to quickly experiment, debug, and tweak our models, which was critical in allowing us to move beyond existing pre-training techniques.

To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training). The pre-trained model can then be fine-tuned on small-data NLP tasks like question answering and sentiment analysis, resulting in substantial accuracy improvements compared to training on these datasets from scratch. Pre-trained representations can either be context-free or contextual, and contextual representations can further be unidirectional or bidirectional. Context-free models such as word2vec or GloVe generate a single word embedding representation for each word in the vocabulary. For example, the word “bank” would have the same context-free representation in “bank account” and “bank of the river.” Contextual models instead generate a representation of each word that is based on the other words in the sentence. Account” — starting from the very bottom of a deep neural network, making it deeply bidirectional.

• In fact, that’s one of the reasons why people often use “keyword-ese,” typing strings of words that they think we’ll understand, but aren’t actually how they’d naturally ask a question.
• To understand why, consider that unidirectional models are efficiently trained by predicting each word conditioned on the previous words in the sentence.
• To help close this gap in data, researchers have developed a variety of techniques for training general purpose language representation models using the enormous amount of unannotated text on the web (known as pre-training).
• That so-called “black box” of machine learning is a problem because if the results are wrong in some way, it can be hard to diagnose why.

Improving Search in more languagesWe’re also applying BERT to make Search better for people across the world. A powerful characteristic of these systems is that they can take learnings from one language and apply them to others. So we can take models that learn from improvements in English (a language where the vast majority of web content exists) and apply them to other languages. All changes ChatGPT App to search are run through a series of tests to ensure they’re actually improving results. One of those tests involves using Google’s cadre of human reviewers who train the company’s algorithms by rating the quality of search results — Google also conducts live live A/B tests. To launch these improvements, we did a lot of testing to ensure that the changes actually are more helpful.

The company also says that it doesn’t anticipate significant changes in how much or where its algorithm will direct traffic, at least when it comes to large publishers. Any time Google signals a change in its search algorithm, the entire web sits up and takes notice. Google says that it has been ChatGPT rolling the algorithm change out for the past couple of days and that, again, it should affect about 10 percent of search queries made in English in the US. While this idea has been around for a very long time, BERT is the first time it was successfully used to pre-train a deep neural network.

It’s based on cutting-edge natural language processing (NLP) techniques developed by Google researchers and applied to its search product over the course of the past 10 months. That so-called “black box” of machine learning is a problem because if the results are wrong in some way, it can be hard to diagnose why. Google says that it has worked to ensure that adding BERT to its search algorithm doesn’t increase bias — a common problem with machine learning whose training models are themselves biased.

Making BERT Work for You

Another example Google cited was “parking on a hill with no curb.” The word “no” is essential to this query, and prior to implementing BERT in search Google’s algorithms missed that. Doing so allows it to realize that the words “for someone” shouldn’t be thrown away, but rather are essential to the meaning of the sentence. Here are some other examples where BERT has helped us grasp the subtle nuances of language that computers don’t quite understand the way humans do. When people like you or I come to Search, we aren’t always quite sure about the best way to formulate a query. We might not know the right words to use, or how to spell something, because often times, we come to Search looking to learn–we don’t necessarily have the knowledge to begin with. Language understanding remains an ongoing challenge, and it keeps us motivated to continue to improve Search.

Here’s a search for “2019 brazil traveler to usa need a visa.” The word “to” and its relationship to the other words in the query are particularly important to understanding the meaning. Previously, our algorithms wouldn’t understand the importance of this connection, and we returned results about U.S. citizens traveling to Brazil. With BERT, Search is able to grasp this nuance and know that the very common word “to” actually matters a lot here, and we can provide a much more relevant result for this query. The old Google search algorithm treated that sentence as a “bag of words,” according to Pandu Nayak, Google fellow and VP of search. So it looked at the important words, medicine and pharmacy, and simply returned local results. The new algorithm was able to understand the context of the words “for someone” to realize it was a question about whether you could pick up somebody else’s prescription — and it returned the right results.

Open Sourcing BERT: State-of-the-Art Pre-training for Natural Language Processing

One of the biggest challenges in natural language processing (NLP) is the shortage of training data. Because NLP is a diversified field with many distinct tasks, most task-specific datasets contain only a few thousand or a few hundred thousand human-labeled training examples. However, modern deep learning-based NLP models see benefits from much larger amounts of data, improving when trained on millions, or billions, of annotated training examples.

Well, by applying BERT models to both ranking and featured snippets in Search, we’re able to do a much better job  helping you find useful information. In fact, when it comes to ranking results, BERT will help Search better understand one in 10 searches in the U.S. in English, and we’ll bring this to more languages and locales over time. The models that we are releasing can be fine-tuned on a wide variety of NLP tasks in a few hours or less.

The Transformer model architecture, developed by researchers at Google in 2017, also gave us the foundation we needed to make BERT successful. The Transformer is implemented in our open source release, as well as the tensor2tensor library. The way BERT recognizes that it should pay attention to those words is basically by self-learning on a titanic game of Mad Libs. Google takes a corpus of English sentences and randomly removes 15 percent of the words, then BERT is set to the task of figuring out what those words ought to be. Over time, that kind of training turns out to be remarkably effective at making a NLP model “understand” context, according to Jeff Dean, Google senior fellow & SVP of research.