Chef R.A.M.S.A.I. (Reactive Augmented Masterchef Smart AI)

Inspiration

• We saw Zuck fail the Meta Ray-Bans cooking demo and we thought we could do better...
• Then, we enlisted our friend, Gordon Ramsay's help

What it does

An AI-powered AR cooking assistant that keeps tracks of TODOs for whatever recipe of choice and automatically helps you with the next step for your creation, in real-time.

How we built it

• Unity for the VR/AR software piece, with C# scripts
• Meta Quest for hardware with its SDK
• gpt-realtime for LLM realtime interaction
• Blender for animating Ramsay's talking mouth

Challenges we ran into

• Unity is painful
• Realtime AI is hard
• Cooking is difficult
• Lack of sleep
• Git conflicts with Unity files are very annoying

Accomplishments that we're proud of

• Getting it to work
• Learning about mug cakes
• Making a PB & J sandwich

What's next for Chef R.A.M.S.A.I.

Coming to a kitchen near you!