Just as artificial intelligence notoriously lacks common sense, it also lacks an understanding of memes. But that doesn’t make its attempts to decode them any less comical.
Enter ResNeXtGuesser, the viral Twitter account that passes popular memes through a neural network and posts the results.
The neural network in question, ResNeXt, was trained on the ImageNet dataset, an AI touchstone that contains 14 million images and underpins tools ranging from military target identification to autonomous driving. But despite its sophisticated foundation, the neural network is hilariously off-base in some of its guesses, like classifying an image of a refrigerator full of eggs as “ping-pong ball” with 99.99% percent confidence. Active since June 2021, the account has amassed more than 450,000 followers.
We connected with the account’s hitherto anonymous creator, Glen Neumann, an embedded computer engineer based in San Jose, California, via email about what sparked the idea, what it’s like to go viral, and his favorite moments running the account.
Talk to me about how the idea came about. What made you first think of it—an idea you had with friends, something you’d been waffling on, maybe a background in a relevant space?
Throughout college, I took some courses related to machine learning, and I had a lot of assignments related to image recognition. For example, one assignment was to make and train a neural network that can differentiate [between] pictures of cats and dogs. But you can pass any image you want through a neural network, and I always found it super funny to pass through memes or any random image and watch the NN [neural network] try to classify it as a dog or cat.
In other assignments, I was dealing with more advanced neural networks like ResNeXt. In doing the same thing, the results were rarely correct, but you could start to see how the neural network thinks, and I thought that was super interesting. In some cases, you could identify what the machine was looking at and why it made its prediction (for example, the stacked cheese in this prediction really does look like the texture of a pineapple IMO).
[Since graduating], I’m now working in the embedded world, so I don’t get a chance to play with machine learning anymore. One uneventful evening, I thought that making the NNGM Twitter bot would be a good chance to both mess with NNs again and learn how to use the Twitter API, so I got to work on it that weekend.
Writing the NN prediction code was actually the easy part—I was able to do that in a few hours. I was just using PyTorch’s pre-trained ResNeXt model. Learning the Twitter API was the hard part, and that took up the bulk of the time of the project. But after a few weekends of work, I had an early version of the bot ready.
What made you choose ResNeXt?
ResNeXt was trained on the ImageNet data set, a huge assortment of labeled images with hundreds of classes. Each year, ImageNet has a competition among researchers to see who can make the most accurate NN. ResNeXt was the 2016 winner, having a better top-5 error percentage, beating even that of humans. When I saw that the Python library PyTorch had a pretrained model for ResNeXt, and I wouldn’t have to code it up and train it myself, it was an easy decision.
Keep up with the innovative tech transforming business
Tech Brew keeps business leaders up-to-date on the latest innovations, automation advances, policy shifts, and more, so they can make informed decisions about tech.
When did the account start to pick up speed? How many submissions were you receiving at its peak?
It was a while before the bot picked up. I would say that the first tweet to really take off was this one in October 2021, a picture of a plushie riding on top of a tank. The plushie is a character from the Touhou series, a video-game series with a cult following.
There’s a sizable Touhou fan community on Twitter, and that tweet found its way in there and blew up. After that, growth was very burst-y. Every now and then, the bot would produce a particularly funny prediction, which would get spread around, and the [account] would get a bunch of new followers all at once.
When the account was small, the number of submissions was small enough to where I could look in my DMs and download each image manually. Eventually, I wrote a script to take some load off of me, but eventually there were too many submissions in the DMs for even my script to handle. Once that happened, I shut down submissions for a while so I could set up a website dedicated to submissions.
By the time I got around to finishing the website, the account had exploded. I don’t remember how long I had the website up—maybe just a few weeks—but the amount of submissions I was getting was simply unmanageable. Even today, I still have a backlog of about 8,000 images [that] I have to look through and moderate.
Can you point me to some of your all-time favorite results?
Absolutely: Cheese that looks like a pineapple. And ResNeXt wasn’t trained to identify eggs, so it mistook a fridge full of eggs for ping-pong balls. Today’s misclassification of a door was particularly funny, IMO, [and] honestly one of my favorites. Giving CPR to a cat was misclassified as a harmonica. Mango ice cream and a picture of an avocado were classified as mashed potatoes. Jinx the cat with a supposed ping pong ball. And sometimes, just getting it right can be amusing.
Did you ever think about the tool’s purpose as both a joke and a way to show that neural nets aren’t as “smart”/advanced/infallible as the public sometimes thinks they are?
To a degree, yes. I ironically, though, hoped that the account would show just how good neural networks can be. Sure, it has a hard time getting the predictions right, but that’s kind of to be expected when I’m feeding it totally nonsensical memes. What I love about the account is that sometimes you can see how the bot thinks, you can sometimes see why it made the prediction it did. And that, to me, is a little bit spooky.
Tell me about a favorite memory that stands out to you from running the account.
Honestly, my favorite moments running the bot are seeing people root for it and get excited when the bot gets a prediction right. It’s cute watching people personify a neural network, and celebrate it when it succeeds and chastise it when it fails. I understand that the NNGM Twitter bot may not be much more than a meme page with a twist, but watching people get excited over a robot predicting nonsensical memes really makes my day sometimes.