Skip to main content
AI

Tech Brew test-drove ChatGPT’s new search feature

How the new feature stacks up in a hot market.
article cover

Anna Kim

5 min read

From Microsoft Bing’s sometimes devious character Sydney to Google’s early Bard bot, tech companies have spent much of the past two years attempting to sculpt generative AI into a usable and up-to-date search product.

OpenAI made its long-awaited foray into this space with the recent rollout of ChatGPT Search, which augments the bot’s conversational capabilities with realtime web access.

It marks a big expansion of the kinds of queries the system can field; GPT-4o otherwise has a broad training knowledge cutoff of October 2023. The search leans heavily on the startup’s growing collection of media partners, which include the Associated Press, News Corp, Vox Media, Axel Springer, and many others. (Axel Springer acquired a majority stake in Morning Brew in 2020.)

But while OpenAI has enjoyed a first-mover advantage with ChatGPT, rivals have already laid stakes in the AI-powered search market. One of Google’s key selling points with Gemini has been its ability to ground AI in years of search dominance. Well-heeled upstarts like Perplexity have also amassed a substantial share of users.

These efforts have sometimes been met with lawsuits from publishers whose information has been swallowed up. Dow Jones and the New York Post recently sued Perplexity, and the New York Times has taken action against OpenAI, despite a roster of its peers joining forces with the company.

Against this backdrop, Tech Brew wanted to try out ChatGPT Search and see how it stacks up against its dominant search rival across a set of use cases suggested in the announcement.

Better weather?

When asked about the weather forecast for New York City, ChatGPT Search gave a comprehensive rundown of the week, with some further analysis about an incoming storm system. The sources cited included the National Weather Service and the New York Post, owned by OpenAI partner News Corp.

That all seems straightforward, but one added advantage of AI search is the ability to converse about results. So, we asked what that rain would mean for the city’s drought warning conditions. The bot spit back a balanced answer, again pulled exclusively from a New York Post article.

Fair enough, but does that work outside of the country’s biggest media market, where the weather gets outsize attention in national news? ChatGPT was able to pull some similar guidance on drought-stricken southeastern Ohio from a local NPR affiliate, which does not appear to have partnered with OpenAI. (The company can—and does—crawl non-partner websites, though publishers can install blockers.)

Google uses generative AI with search in at least two big ways—through its AI Overviews that pop up sporadically in web searches and within its Gemini chatbots. This particular Google search didn’t trigger one of these overviews, so we looked to Gemini. The paid version gave a similar answer to ChatGPT, though with a wider range of works cited, including a Dartmouth study, a transcript of a mayoral press conference, and a local news article.

Keep up with the innovative tech transforming business

Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.

Perplexity’s answers here weren’t much different, either, with a range of news articles cited.

The free version of Gemini gave the wrong answer—no rain in the forecast—but immediately corrected and contradicted that text with a Google result below, an example of the company’s search grounding at work.

To news or not to news

Perhaps emboldened by legacy news backing, ChatGPT Search will readily give you an analysis of, say, Trump’s staffing picks, pulled more or less directly from partners like Business Insider and New York Magazine. Perplexity also gave a similar answer.

Both free and paid Gemini models, on the other hand, are still skittish about political news since the election and respond to such questions with a blanket refusal to engage.

Gemini Advanced will mention vague, sometimes-outdated items like “speculation is rife about who Donald Trump might pick for his cabinet”—he’s already announced several picks—and President Biden attending the G20 Summit. But it will refuse any follow-up questions about any of those items related to politics.

Gemini’s free version—perhaps not a fair comparison since a paid subscription is required to use ChatGPT Search—returned several badly out of date news updates, like an economic outlook from November of 2023.

All bots tested had mixed utility for local news outside of big metros, scraping a hodgepodge of news articles and municipal press releases.

Restaurant recs

All AI search engines showed an impressive ability to break down the dining scenes of major cities, where food blogs are plentiful, as well as smaller cities and towns, based on restaurant websites and Tripadvisor reviews, including answering detailed questions.

This is the kind of use where this navigation seems to make sense and the internet’s bank of user-generated and business content can flesh out answers.

Overall, though, OpenAI’s search product shone most beyond rivals when tapping its premium content partnerships. Some of these queries did, however, seem more readily attainable through a simple Google Search, which has the added benefit of being free, though OpenAI has promised to roll search out to free users “over the coming months.” It’s also worth watching how these search offerings might diverge further as publishers and platforms push back on AI training.

Keep up with the innovative tech transforming business

Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.