This is a submission for the Open Source AI Challenge with pgai and Ollama

What I Built

NutritionAI is an AI-powered application that helps users track their dietary habits by allowing them to upload images of their meals. The app analyzes these images to provide detailed nutritional insights and personalized recommendations based on individual dietary goals and preferences.

How it Works

• Diet Preferences: Users first input their dietary goals and health needs, which are sent to the backend. This data is embedded using the Mistral model and stored in a pgvector database.

This is the schema for the user's diet preferences

CREATE TABLE users (
    id SERIAL PRIMARY KEY,
    age INT NOT NULL,
    height FLOAT NOT NULL,
    weight FLOAT NOT NULL,
    caloric_target INT NOT NULL,
    protein_target INT NOT NULL,
    dietary_preferences TEXT[],  
    complications TEXT[],      
    embedding VECTOR             
);

• Meal Upload & Image Analysis: Once on the main page, users can upload a meal image. The backend processes the image and sends it to the Llava model, which identifies food items and dishes.

• Nutritional Analysis: The identified food items are passed to the Mistral model to generate nutritional information. These stats are embedded as vectors, saved in the database for future use.

This schema represents the food items.

CREATE TABLE food_items (
    id SERIAL PRIMARY KEY,
    name VARCHAR(255) NOT NULL,   
    embedding VECTOR,              
    nutrition_info JSONB          
);

• Similarity Check: The new meal vector is compared with the user's dietary preferences using cosine similarity (via pgai) to evaluate alignment with the user's goals.

• Personalized Recommendations: Based on similarity scores, NutritionAI analyzes the nutritional data and suggests diet adjustments that best meet the user's preferences and health objectives.

Architecture

Demo

Here's the link to the working project
nutrition-ai-bay

Disclaimer: The generation of nutritional stats and personalized recommendations may take around 5-10 minutes, as the Node.js application is hosted on an Amazon EC2 t2.large instance.

Source code:

Screenshots

Diet Preferences Page

Main Food Analyzer Page

Tools Used

• React: Framework for building dynamic user interfaces.
• Tailwind CSS: Utility-first CSS framework for styling.
• Node.js: JavaScript runtime for backend development.
• Express: Framework for handling API requests and routing.
• Multer: Middleware for processing file uploads.
• Ollama Models:
  • Llava Model: Analyzes meal images to identify dishes.
  • Mistral Model: Generates nutritional content and embeddings.
• TimescaleDB (PostgreSQL): Database for storing user and nutritional data.
• pgvector: Utilized for storing and querying vector embeddings of user profiles and food items, enhancing the accuracy of recommendations.
• pgai: Integrated to leverage AI capabilities for generating insightful nutritional content and recommendations based on user data.
• Amazon EC2 (Ubuntu t2.large): Hosts the Node.js application.
• Vercel: Deploys the frontend application efficiently.

Final Thoughts

Building NutritionAI has been an exciting project(PS: Took me 5 whole days to build this), especially with the challenges of implementing real-time image analysis and generating personalized nutritional recommendations through advanced AI models. With Ollama’s models and PostgreSQL extensions, I was able to create an engaging, user-friendly experience that brings detailed nutritional insights to users' fingertips.

In the future, I plan to enhance NutritionAI with additional features, such as a personalized fitness plan that aligns with users' dietary habits, and tailored meal planning suggestions. Improvements could also include faster processing times and even more advanced dietary analytics.

Prize categories:
Open Souce Models from Ollama
All the extensions.