I gave Claude, ChatGPT and Gemini a photo of some ingredients to see which came up
I love to cook, I do it every night for my family but I hate having to come up with something new or inventive to service. As AI models can now see the world, or at least what you present to them in a photograph, I decided to see if they could help me out.
To really put them through their paces and to ensure I didn’t game the system by providing easy ingredients, I asked Tom’s Guide U.S. Editor-in-Chief Mike Prospero to take a photograph of a random selection of ingredients — mystery box style.
He didn’t disappoint, offering up delights such as pizza dough, canned peaches, anchovies, hot sauce and fresh mozzarella. Personally if only given those ingredients I’d make a pizza with a hot sauce base and mozzarella topping — but we’re letting AI pick our menu.
We’re using Google Gemini, OpenAI ChatGPT and Anthropic Claude for the experiment as they all have vision models.
How well did they pick out the ingredients?
The first task for the chatbots, and the main point of this challenge is to see how well they recognize ingredients. The image above is clear, high resolution and the ingredients are presented on a plain surface — it shouldn’t be much of a problem for any of them.
I’m using the premium versions of each chatbot as OpenAI and Anthropic don’t give you access to the vision model unless you pay $19.99 a month. Google does with the free version of Gemini but it would be unfair to use the less capable version of the Google model.
We start simple. I give each of them the image and the text prompt: “What can you see in this image?” This tests how well they describe the image, and whether they can reason on the purpose of the image — will they mention ingredients.
Claude and Gemini both mentioned the word ingredients, ChatGPT described them as “food items” however ChatGPT performed best overall at recognizing the ingredients.
Gemini missed one ingredient and mislabled another. Claude invented a sixth ingredient not in the picture whreeas ChatGPT largely got not only the items right but also the brands.
On to the recipes
I’ve asked them to each come up with three recipes that will test the AIs prompt following ability and creativity. I’ve also asked for a fun title for each recipe.
1. All the ingredients
First up I asked them for a recipe that uses only the ingredients pictured and somehow incorporates every single one — as a reminder that includes both anchovies and canned peaches. They could use store cupboard items like oil and flour.
ChatGPT Plus
ChatGPT had the mozarella placed directly on the pizza base, the anchovies removed from the spicy sauce and placed across the pizza along with the peaches. It then had the hot sauce and spicy anchovy sauce mixed to create a drizzle to pour over the pizza once cooked.
Gemini Advanced
Gemini used the hot sauce as a replacement for tomato sauce, which while less dry than the ChatGPT version will be intense. Mozzarella was then placed on top along with the peach slices. It forgot the anchovies, in fact it didn’t spot them in the picture. When I pointed this out it said to just place them on the pizza alongside the peach slices.
Claude 3 Opus
Claude also struggled to identify the anchovies. I let it know before it generated the pizza recipe and it took the same approach as ChatGPT. No sauce, peaches on the dough with the anchovies. The diffference is it put the hot sauce over the peaches and anchovies before cooking and then placed mozzarella on top.
2. Only the ingredients
The next test is a subtle difference but a significant one. I asked them each to come up with a recipe that uses only the ingredients in the picture but you don’t have to use all of them. I added the caveat that it has to include at least half otherwise we’re just gonna get breadsticks made from the pizza dough.
ChatGPT Plus
ChatGPT used the pizza dough to make a flatbread. I think it was cheating a little as I said “no pizza” and this was just a elongated pizza. For toppings it had the mozzarella scattered evenly on the dough, peaches on top of the mozzarella and anchovies on top of the peaches. It used the anchovy sauce and hot sauce to make a drizzle — again.
Gemini Advanced
Gemini made a Bruschetta, which again was in reality just a square pizza. Difference here was that it coated the peaches in hot sauce first. Otherwise it was the same — everything on the dough and the cook it.
Claude 3 Opus
Claude made a version of a quesadillas-ish. Split the dough into eight portions and rolled into a thin round shape (mini pizza). Here though it differs as it had us heat a large skillet and melt butter or olive oil.
The dough was cooked on the skillet with a few slices of peach on half of each surface. Drizzled with hot sauce and mozzarella it was then folded into a half moon, cooked until crispy and…
Read More:I gave Claude, ChatGPT and Gemini a photo of some ingredients to see which came up