Helping people manage their diet using AI models
In recent years, people have become more aware of their dietary choices and the impact of food on health and chronic diseases. Revathy Venkataramanan, a computer science and engineering Ph.D. student, is using artificial intelligence (AI) techniques to develop nutritional analysis from food images and meal recommendations based on a user’s health conditions and food preferences.
Venkataramanan’s core research at the AI Institute of South Carolina (AIISC) focuses on an explainable food recommendation system to help diet-conscious audiences and patients with their food choices. She hopes to deploy the nutrition management tools as a mobile application or chatbot.
“Most dietary assessment systems serve as a journal to log food intake. Similarly, many deep learning-based nutritional analysis models will estimate the calorie intake of food images, and recommender systems will recommend meals,” Venkataramanan explains. “But these models do not comprehensively analyze nutrition, ingredients used and cooking methods involved to recommend meals or provide the explanation for food choices made by the recommendation system. “We aim to close the gap by developing a recommender system that explains its decisions to the user, thereby making the AI system trustworthy for users.”
Venkataramanan’s explainable food recommender system also incorporates a personalized health knowledge graph to recommend meals based on a user’s health condition and food preferences. Her work aims to harness the generalization ability of deep learning models and the abstraction ability of knowledge graphs.
“The system doesn’t just tell the user whether the food is good for them,” she says. “It also explains why or why not so they can make an informed decision.”
Many of Venkataramanan’s projects have focused on health care initiatives designed to monitor patient vitals through Internet of Things-based sensors that can help clinicians make informed decisions. Her ideas for using AI models for nutrition management came after a meeting with a bariatric surgeon.
“We were discussing how to monitor the diet of bariatric surgery patients post-surgery, and the bariatric surgeon suggested it would be cool to take a picture of a food and receive the name of it. That was my bulb-glowing moment,” Venkataramanan says.
Venkataramanan is also developing a cooking simulator for enhanced cooking representation learning, a prerequisite to her explainable food recommender system. She explains that one of the major challenges in analyzing food from images is that many ingredients added during the cooking process are invisible in an image of the completed recipe. However, understanding what is added during the cooking process and the cooking method is essential for appropriate food recommendations.
Revathy offers a perfect example of the rich environment the AIISC offers to develop Ph.D. students as independent researchers.
- Amit Sheth, Founding Director, AIISC
Venkataramanan seeks to bridge the gap between the representations of the recipes and target food images by modeling the cooking process to include instruction, ingredient text and ingredient images. By understanding the cooking process and ingredients used, health-specific recipes can be generated and classified, and new flavor combinations can be explored based on cooking style and ingredients.
“By analyzing the effects of cooking methods and the type of ingredients used, we can evaluate whether a given recipe is suitable for a given diet,” Venkataramanan explains. “The cooking model can further aid users with food restrictions by finding alternate ingredients.”
The third project in development will aid carbohydrate estimation for type 1 diabetes patients. These patients must precisely track their carbohydrate intake to estimate their insulin dosage. But Venkataramanan says that many patients are known to guesstimate their carbohydrate consumption, resulting in inaccurate insulin dosage. Her solution is a mobile application that will track food intake and estimate carbohydrate content from meal images using a volume estimation algorithm. Venkataramanan’s team collaborates with nutrition database provider Edamam and Dr. Lisa Knight, a clinical endocrinologist at Prisma Health, who will use the application in clinical trials.
Working with colleagues from different departments has fostered an interdisciplinary approach that has helped cultivate Venkataramanan’s research. For example, she is collaborating with the physical sciences department on “Precision Nutrition,” which aims to create personalized meals for patients incorporating gut microbiota. “It’s a huge plus having input and perspectives from many different areas,” she says.
Amit Sheth, professor and founding director of the AIISC, is Venkataramanan’s research mentor. “Revathy offers a perfect example of the rich environment the AIISC offers to develop Ph.D. students as independent researchers,” Sheth says. “AIISC has around 20 translational research projects involving collaborations with most UofSC schools and colleges. Revathy’s project in precision nutrition, one of the top 20 technology areas, is a perfect example of such research.”
Venkataramanan is a graduate research assistant at AIISC and a research intern at Hewlett Packard Enterprise. She expects to complete the Ph.D. program in the spring of 2024.