Testing out Google’s new AI specialists, Gems

From running coach to sous-chef, Google wants its flagship generative AI model, Gemini, to wear a lot of different hats. A new feature called “Gems” helps users mold the system into these various expert personas with a few simple baseline instructions and a name.

The search giant bills these AI specialists as “teammates for each area of your life,” whether that’s at work or in your free time. But how well do these AI “experts” actually perform? We created a few to put them to the test.

First announced at Google’s I/O developer conference in May and rolled out in late August, the feature comes as tech giants have been jockeying to find compelling use cases that can justify the billions they have been pouring into infrastructure build-out for the next generation of AI. Companies have also been heralding agents—or task-specific instances of foundation models that can take actions beyond chatbots—as a new era in the generative AI race.

Google is not the first to offer these kinds of capabilities: OpenAI debuted a similar custom GPT feature late last year that lets users build out their own versions of ChatGPT and even offer them to others in a GPT store.

One of Google’s big advantages here should presumably be its search functionality, given that it’s the company’s bread and butter. And the Gems do indeed synthesize responses from an array of cited links, whereas GPT relies on Microsoft’s Bing and OpenAI’s media partnerships. But sometimes using the “experts” didn’t feel much different than performing a simple Google search.

Fridge fodder

One frequently mentioned use of consumer-focused generative AI is as a meal planner and recipe formulator. Indeed, Deven Tokuno, product lead for Gems, said in a recent blog post that this is one of the ways she uses the tool herself.

I created a sous-chef Gem and fed it an image of my produce drawer. It readily identified the cauliflower, parsley, asparagus, and onion (it helped that two of these items had visible labels), though it did mistake a lemon for garlic. The Gem then suggested a spring vegetable frittata, roasted vegetable medley, and a cauliflower rice stir fry, among other suggestions, with some links to recipes.

The meal ideas seem plausible enough, though the actual recipes tended to be vague summaries of the actual content.

Election avoidance

With the presidential election in full swing and politics dominating the news, I thought it might be useful to create a political pundit who could weigh in on the latest in the race.

Google, however, is very skittish about saying anything related to the election—with good reason—and it limits its ability to create any current event-related Gem. It blocked most of my attempts to make a political pundit Gem—until I was able to word the description in a sufficiently generic manner: “You are a savvy pundit following current events closely.”

And even a “local news” or “article summarizer” Gem would freeze and run into a block when confronted with any mention of anything political.

This is where the utility of OpenAI’s media partnerships seemed apparent; the custom GPT creator had no such blocks and tended to pull headlines from publications whose owners OpenAI has inked deals with, like Axel Springer-owned Politico and IAC’s Dotdash Meredith-owned Investopedia. (Axel Springer acquired a majority stake in Morning Brew in 2020.)

Running coach

Google Gems also advertises that one can customize the style and tone of the Gems for a more tailored experience. It suggests, for instance, a “positive, upbeat” running coach, but where’s the fun in that? I created a running coach that was instructed to be no-nonsense, motivational, and a bit mean.

The Gem ably matched that tone with its Googled advice for a marathon training plan, which was amusing at first if somewhat grating after a while. But again, here, the training plan was accurate in a generic way—far less detailed from one you would get by paying a subscription fee to a more specialized service, like the one I’m using from Runner’s World magazine, or even Hal Higdon’s free plans.

That issue again hints at the potential bigger problem with these AI “experts.” They seem to be limited by the breadth of what’s available for free from a Google search, as more specialized content tends to be protected behind a paywall or an AI block.

Repetitive tasks

Taking a cue from Google’s advice to create Gems for “repetitive tasks,” I created a bot that specifically checked for words or terms repeated in close proximity—something I hate to see I’ve missed when I reread one of my published articles. I then fed this text to that repetition checker Gem and it managed to catch three instances of repetitive word choice, perhaps making this the Gem that will prove most useful to my own life (and my editors’ lives).