MIT is building a system that can identify a recipe using pictures of food

Mastering the ability to distinguish hot dogs from not hot dogs is truly one of mankind’s greatest achievements. Attempting to one-up that sort of masterstroke would surely be flying too close to the sun — but researchers at MIT are trying nonetheless.

The team’s pic2recipe system uses neural networks to determine the recipe of a given food, in an attempt to gain “valuable insight into health habits and dietary preferences,” based on images of food pulled from social media.

pic2recipe was built using the Food-101 Data Set, a food-identifying algorithm built by Swiss scientists in 2014, using a database of 101,000 food images. That information is cross-referenced with the CSAIL team’s own Recipe1M database of one million recipes pulled from popular sites like All Recipes and Food.com.

The system is far from perfect. At the moment, it’s able to correctly identify the recipe around 65-percent of the time. The biggest problem with the tech is the images themselves.

“It’s mostly an issue of getting the scale correct,” coauthor Nick Hynes told TechCrunch. “When people take pictures of food, there’s a lot of variation in style: whether taken from close up or far away; whether it’s of the single item, multiples, part of a complete dish. Of course, that’s not unreasonable, since even a human might think that a single cookie is a pancake when zoomed out.”

The system excelled with baked goods like cookies and muffins, according the research — while things like sushi and smoothies gave it a bit more trouble, marking the first (and hopefully last) time those two foodstuffs were mentioned in the same sentence. It also had problems when it came to identifying foods that had multiple different recipes in the system’s database — and as such, the team made sure to identify when it got at least part of the way there.

The team readily admits that there’s still a fair amount of work to be done on pic2recipe, and while CSAIL doesn’t have any plans to bring it directly to market, in spite of America’s current love affair with hot dog identification. Though the technologies developed here might actually have some real world uses, like checking nutritional value or identifying dietary restrictions.