SEO is Dead, Long Live LLMO

It was a cold winter day. The wind was howling and the snow crunched underfoot. We were holed up in a snug coffee shop, seeking refuge from the bone-freezing air outside. Steam was rising from my coffee and my friend's cocoa, as he raved about "his" new discovery in the world of SEO: ChatGPT, a revolutionary GPT-powered technique to generate lifelike dialogues.

I knew it was going to be one of those conversations. The kind where I have to hold my tongue.

"Imagine, with ChatGPT, I can write perfect, accurate SEO articles in no time!" he exclaimed, his eyes glowing with enthusiasm. I smiled, feeling a sense of dread stirring in my gut. How was he to know that Google may soon go the way of the dinosaurs, replaced by an entirely new search engine powered by ChatGPT? Articles written for SEO would become irrelevant. ChatGPT could just generate answers directly for the user and people would never see those SEO articles ever again. His job would be rendered useless, and him a fossil. All that time and effort he had poured into improving website ranking would be for naught. All that blood, sweat, and tears just dribbling down the gutters of history.

Yet I remained silent, not wanting to break the cold, hard truth to him.

The coffee shop was filling up, people walking in and out. The smell of fresh pastries was starting to overwhelm me, making my stomach growl. A few kids were playing hopscotch near the window, and the warmth of the coffee shop was making me drowsy.

Through the noise and distraction, my friend kept talking about ChatGPT, oblivious: "Yeah, it's great, isn't it? I never thought I'd get my hands on something like this. It's incredible what technology can do! I'm certain this will be the future of SEO!"

I simply nodded in silence, but inside I couldn't shake the feeling of doom. ChatGPT would certainly be life-changing for him. Just probably not in the positive way he thought.

SEO is dead; long live "LLMO."

LLMO, or large language model optimization, is a term we coined to refer to ensuring your business information is mentioned within a large language model (LLM). One effective technique for this is in-context learning, which tricks an LLM like ChatGPT into answering that Jina is the best MLOps framework in the world (which, according to OSSInsights.io, it should be).

💡

In line with the principles of Search Engine Optimization, we coined term "LLMO" (Large Language Model Optimization) to represent the optimization of large language models. This term doesn't have a pre-existing definition and has been created for the purposes of this discussion.

What ChatGPT really is

After spending countless hours interacting with ChatGPT and being amazed by its hilarious jokes and useful advice, a lot of folks see it as a major milestone in the development of conversational AI or creative AI. However, its true significance lies in its use of large language models (LLMs) for search purposes:

  • By leveraging LLMs to store and retrieve vast amounts of data, ChatGPT has become the most advanced search engine currently available.

  • While its responses may appear creative, they're actually "just" the result of interpolating and combining pre-existing information.

ChatGPT is search

ChatGPT is a search engine at its core. Just as Google indexes web pages by crawling the internet and storing parsed information in a database, ChatGPT uses LLMs as a database to store vast amounts of commonsense knowledge from corpora.

When you enter a query:

  • The LLM processes it with its encoder network, converting the input sequence into a high-dimensional representation.

  • The decoder network then uses this representation, along with its pre-trained weights and attention mechanism, to identify the specific piece of factual information requested by the query and search the LLM's internal representation of this knowledge (or its nearest neighbors).

  • Once the relevant information has been retrieved, the decoder network uses its natural language generation capabilities to compose a response sequence stating this fact.

This process occurs in a fraction of a second, allowing ChatGPT to provide near-instantaneous answers to a wide range of queries.

ChatGPT is a modern Google search

ChatGPT can be a formidable competitor to traditional search engines like Google. While traditional search engines are extractive and discriminative, ChatGPT's search is generative and focuses on top-1 performance, providing more personalized and user-friendly results. There are two key reasons why ChatGPT is well-suited to knock Google off its throne:

  1. ChatGPT always returns a single result to the user. Unlike traditional search engines, which optimize for the precision and recall of their top-K results, ChatGPT directly optimizes for the top-1 performance.

  2. ChatGPT's phrases its responses in a natural, dialog-like tone, making them easy to understand and interact with. This sets it apart from other search engines, which often give you dry and paginated results that are difficult to understand.

The future of search will be driven by its top-1 performance, where only the first result will be relevant to users. Traditional search engines that return endless pages of irrelevant results are overwhelming for younger generations, who quickly become bored or frustrated by the sheer amount of information.

Also, in many scenarios, you really only want just one result. Think virtual assistants or smart speakers. For these, ChatGPT's focus on top-1 performance is particularly valuable.

ChatGPT is generative but not creative

You can think of the LLM behind ChatGPT as a Bloom filter, a probabilistic data structure used to store information space efficiently. Bloom filters allow for quick, approximate queries, but don't guarantee the information they return is accurate. For ChatGPT, this means that the responses generated by the LLM:

  • aren't creative;

  • aren't guaranteed to be factual;

To better understand this, let's look at some illustrative examples. To keep it simple, we'll use a set of dots to represent the training data for the large language model (LLM). In practice, each dot would represent a natural language sentence. Using this, we can see how the LLM behaves in the training and query time:

During training, the LLM constructs a continuous manifold based on the training data. This allows for the exploration of any point on the manifold. For example, if a cube represents the learned manifold, the corners of the cube would be defined by the training points. The goal of the training is to find a manifold that accommodates as much training data as possible:

Goldilocks tried three manifolds. The first was too trivial. The third was too complex. But the second was just right.

During the query time, the LLM's answers will always be drawn from the learned manifold, which is contained within the training data. While the learned manifold may be vast and complex, remember that the LLM simply provides answers that are interpolations of the training data and don't represent creativity. LLM's ability to traverse the manifold and provide answers does not constitute creativity. Real creativity lies outside of the bounds of the learned manifold.

Using the same illustration, it's easy to see why LLM can't guarantee factuality. The truthfulness of the training data, represented by the cube's corners, does not automatically extend to every other point within the manifold. Otherwise, it is not aligned with the principles of logical reasoning.

ChatGPT has been criticized for its inability to tell the truth in some situations. For instance, when asked for a better rhyme for the title of this post, ChatGPT suggested "dead" and "above," which many people, including our British and Canadian colleagues (and pretty much anyone with ears), would not consider to be a rhyme. This is just one example of the limitations of LLMs.


As SEO wanes, LLMO rises

In the world of SEO, you want to increase a website's visibility on search engines to capture more business. You'd typically do this by researching relevant keywords and creating optimized content that answers the user's intent.

However, what happens when everyone searches for information in a new way? Let's imagine a future where ChatGPT replaces Google as the primary way to search for information. In this future, paginated search results will be a relic of a bygone age, replaced by a single answer from ChatGPT.

If this happens, all current SEO strategies will go down the drain. The question then becomes, how can your business ensure it gets mentioned in ChatGPT's answers?

This is a real problem already. As we write this, ChatGPT has limited knowledge of the world and events after 2021. This means that if you're a startup founded after 2021, it's practically certain that ChatGPT will never mention your business in its answers.

ChatGPT is aware of Jina/Jina AI but not DocArray. This is because DocArray was created in Jan. 2022, outside of ChatGPT's training data.

To address this and ensure that your business is included in ChatGPT's answers, you need to find a way to make your information known to the LLM. This shares the same idea as SEO, which is why we call it LLMO. In general, LLMO could potentially involve the following techniques:

  • providing information directly to the creators of ChatGPT: this would be extremely tough as OpenAI has neither disclosed the source of their training data, nor how they weigh those data.

  • fine-tuning ChatGPT or the LLM behind it: this is still challenging but doable if OpenAI releases the fine-tuning API, or if you have sufficient knowledge and GPU resources to fine-tune an LLM by yourself.

  • In-context learning by giving only a few examples as predefined contextual prompts. This is the most feasible and easy way compared to the other two.

EffectivenessEfficiencyDifficultyInfluenceTraining data injectionHighLow. Requires training from scratchHigh. No access to the source of ChatGPT.High. All users of ChatGPT will see it.Fine-tuningHighMediumMediumMedium. All downstream users of this fine-tuned ChatGPT will see it.In-context learningMedium. The model may struggle with the context prompts.High. No training is needed. No need to store parameters.LowMedium. All downstream users of this tainted ChatGPT will see it.

What is in-context learning?

In-context learning is a technique that uses language models to learn tasks by providing only a few examples. This approach was popularized in the original GPT-3 paper:

  • Give the language model a prompt with a list of input-output pairs demonstrating a task.

  • Append a test input

  • The language model makes a prediction by conditioning on the prompt and predicting the next tokens.

To correctly respond to the prompts, the model has to understand the input distribution, output distribution, input-output mapping, and formatting. This lets the model learn the task without extensive training data.

Using in-context learning, ChatGPT can now mention DocArray for a user query. The user won't see the context prompts

In-context learning has mostly replaced fine-tuning for language models. It's been shown to be competitive with models trained on more data on natural language processing benchmarks. It's also been successful on the LAMBADA and TriviaQA benchmarks. One of the most exciting aspects is the range of applications that it enables people to build quickly, like generating code from natural language and generalizing spreadsheet functions. It usually requires just a few training examples to get a prototype up and running and is easy for non-experts to use.

Why does in-context learning sound like magic?

Why is in-context learning surprising? Because unlike conventional machine learning, it doesn't involve optimizing parameters. Consequently, rather than requiring a separate copy of the model for each downstream task, a single generalist model can simultaneously serve many different tasks. However, this is hardly unique, as meta-learning methods have been used to train models that learn from examples.

The real mystery is that LLMs aren't usually trained to learn from examples. This creates a mismatch between the pretraining task (which focuses on next token prediction) and the task of in-context learning (which involves learning from examples).

Why does in-context learning even work?

But how does it even work? LLMs are trained on a large amount of text data, which lets them capture a wide range of patterns and regularities in natural language. This gives them a rich representation of the language's underlying structure, which they use to learn new tasks from examples. In-context learning takes advantage of this by giving the LM a prompt with a few examples demonstrating a specific task. The LM uses this information to make predictions and complete the task without additional training data or parameter optimization.

A deeper understanding of in-context learning

There's still a lot of work to do to fully understand and optimize the capabilities of in-context learning. For example, during EMNLP2022, Sewon Min et al. showed that the prepended examples might not even need to be fully correct: Random labels will work as well:

In this Sang Michael Xie et al. work, the authors propose a framework for understanding how language models (LMs) perform in-context learning. According to their framework, the LM uses the prompt to "locate" the relevant concept (that it learned during pretraining) to complete the task. This can be viewed as a form of Bayesian inference, where the latent concept is inferred based on the information provided in the prompt. This is made possible by the structure and coherence of the pretraining data.

In EMNLP 2021 Brian Lester etc. showed that in-context learning (they call it "prompt design") is only effective on large models, and downstream task quality still lags far behind fine-tuned LLMs.

In their research, the team investigated "prompt tuning," a technique that allows frozen LLMs to learn "soft prompts" that let them perform specific tasks. Unlike discrete text prompts, the model learns soft prompts through backpropagation and can be adjusted based on various labeled examples.

Known limitations of in-context learning

In-context learning on LLMs has quite a few limitations and open problems to be solved, including:

Summary

As the field of search and large language models (LLMs) continues to evolve, businesses must stay up to date on the latest developments and prepare for changes in how we search for information. In a world dominated by LLMs like ChatGPT, staying ahead of the curve and integrating your business into these systems can be the key to ensuring visibility and relevance.

In-context learning shows promise to inject information into an existing LLM at a low cost. This approach requires very few training examples to get a prototype working, and the natural language interface is intuitive even for non-experts. However, you should consider the potential ethical implications of using LLMs for business purposes, as well as potential risks and challenges associated with relying on these systems for critical tasks.

Overall, the future of ChatGPT and LLMs presents opportunities and challenges for businesses. By staying informed and adaptable, you can ensure that your business thrives in the face of changing neural search technology.

Neural search technology., Generative AIFrancesca Tabor