Revisiting Google’s AI moat after I/O 2023

This article is part of our series that explores the business of artificial intelligence

Google has seen several setbacks in the past few months as advances in artificial intelligence seem to trigger major shifts in the tech landscape. First was the successful launch of OpenAI’s ChatGPT, which caught Google off guard and caused speculation about the disruption of its search business. Then was the hasty and failed demo of Bard, Google’s ChatGPT rival, which shaved off $100 billion from its market cap and caused doubt about whether it could catch up with Microsoft and OpenAI.

Next, the advent of highly efficient open-source large language models (LLM) brought the dominance of big tech in the generative AI market under question. A leaked internal Google memo—now known as the “moat document”—further highlighted the ambivalence around the company’s future in applied AI.

But this week’s Google I/O event proved that the search giant is adapting to the changes in the field. Although Google—nor Microsoft, for that matter—will probably dominate the LLM market, it is still making sure its products are relevant as the new wave of AI sets in.

Mass integration effort

Google I/O did not introduce any fundamentally new generative AI technology. Rather than new and larger models that can do weird things, the event’s more interesting highlights were how Google is enhancing its current products with LLMs.

Bard is now pretty much on par with ChatGPT in tasks such as writing articles and generating code. But its biggest upgrade is integration with Google products, including Search, Gmail, Docs, Sheets, Colab, Lens, and Maps. You can pull and transfer information from different apps in the Google ecosystem seamlessly through the Bard interface. For example, you can prompt Bard for advice on a coding problem, iterate on the output several times, and then directly transfer the results to a Colab notebook.

There are two ways to integrate AI. One is to bring applications to the AI, such as plugins for Bard. The other is bringing AI to the apps. Google is moving in this direction too, integrating generative AI features directly into some of its apps. This is very much like Microsoft’s 365 Copilot. For example, you can directly use an LLM assistant in Docs to write an article or in Gmail to draft an email.

Trailing behind Microsoft?

For the moment, Google’s playbook looks a lot like Microsoft and OpenAI. It is even releasing an API service that is very much like OpenAI API and Microsoft’s Azure OpenAI Service. The API service will provide access to PaLM 2, the LLM that powers Bard and other generative AI models used in its apps. Like the OpenAI API, PaLM 2 will also support fine-tuning. At I/O, Google demonstrated two fine-tuned versions of the model, tailored for medical and cybersecurity applications. Unfortunately, the PaLM 2 paper (titled “technical report” like OpenAI’s GPT-4 paper) contains very little detail on model architecture and training data (again, a la GPT-4). We also know that the next generation of Google’s AI, called Gemini, will be multimodal language models.

Through AI integration, Google is protecting its products against Microsoft’s aggressive AI offense. Google Search + Bard might eventually be much better than Bing Chat given Google’s superior search engine, helping Google maintain its dominance in online search. And the Workspace integrations will be at least on par with the 365 Copilot, which will help Google keep its share of collaboration tools.

It’s not clear what the monetization model of these services will be, but there will be fierce competition between the two tech giants in adding value to their services. Part of it will come down to competing on pricing models.

It seems to me that for the moment, big tech companies are fighting over existing markets. Their growth and defensibility mechanisms are around the classic factors such as added value and switching costs. They’re adjusting their moats to the shifts caused by the new wave of AI.

But the market for LLMs and generative AI is still going through major upheavals and it is not clear who will dominate the new segments of the market.

The new moats

In a recent post, I explained that previously, it was assumed that the moats around LLMs were built on three things: training data, compute costs, and model weights. Big tech companies could create defensible products and businesses around huge LLMs that required massive datasets and millions of dollars to train and run.

Developments in the open-source LLM movement showed that those moats are not as impregnable as they seem. Models such as Alpaca and Dolly 2 showed that if you have a pretrained LLM with 7-13 billion parameters (e.g., LLaMA or Pythia), you will be able to fine-tune it for instruction following (like ChatGPT) with a modest amount of data and a few hundred dollars. And many of them can run on consumer-grade GPUs.

This makes it possible for companies to create totally new applications. They no longer need 175-billion-parameter LLMs for every application. They can take a much smaller model, fine-tune it with their own data, and run it on their own servers. Naturally, a very large model will be better at generalizing to a wide range of tasks. But in reality, most applications are specialized. Many companies will prefer a small model that can be fine-tuned to perform a few tasks very well over a very large model that has average performance on a wide range of tasks.

In line with Google’s internal warning, at least some parts of the LLM market currently have no moats. Companies that are not locked into a specific cloud provider and have more flexibility in choosing their app ecosystem are more likely to explore open-source solutions.

However, I think Google and Microsoft still have an advantage in the enterprise sector. Big companies that are already using the Google or Microsoft cloud will find the AI integrations much more convenient than implementing their own LLMs across different data stores and applications.