Let’s Cook Up a Story
"Let's Cook Up a Story" turns grammar into magic!
Learn sentence structure by placing handcrafted felt ingredients into an enchanted pot.
This interactive installation turns sentence building into Christmas magic. Inspired by PECS communication tools, visitors of all ages tap into childlike wonder as they learn grammar through play. By placing handcrafted felt ingredients into an enchanted cooking pot, they construct sentences with a subject, verb, and object that transform into whimsical Christmas stories.
Role: Arduino & Sensor Integration, Javascript Development, Visual Design, Fabrication
Timeline: 4 Weeks (Nov - Dec 2025)
Team: Minami Matsumoto
Our Process
The Inspiration
The inspiration came from working with children on the spectrum. I saw how building blocks and paint became languages of their own, each creation filled with meaning. I also learned about PECS (Picture Exchange Communication System), a tool that empowers non-verbal children to communicate through pictures. We wanted to capture that same magic: learning through touch, shape, and play.
Our Goal
We asked: how might we make sentence structure feel like play rather than a lesson?
Our solution maps grammar to geometry. Subject + Verb + Object becomes Circle + Triangle + Square. Children intuitively understand that different shapes serve different purposes, creating a mental model that sticks.
The Interaction
This sketch maps the four-step user journey. Visitors select felt pieces from an ingredient shelf, place them onto the cooking pot, close the lid to trigger the magic, and watch their custom story unfold on the puppet theater display.
How does the system know which piece is placed?
Each felt piece contains a unique resistor sewn inside. When placed on the pot, a voltage divider circuit produces a distinct analog reading that the Arduino identifies, matching each resistance value to a specific character, action, or object.
How does p5.js know what to show?
p5.js uses a state machine to manage the entire experience, listening for Arduino input at every step. The sketch transitions seamlessly from title screen to item selection to cooking animation to story playback, all driven by physical interaction and sensors.
How did we create 27 animations?
We turned static felt elements into 27 animations using Flora AI as our coordinator. By starting with three combinations and letting Flora work with Kling O3 Pro and Runway, we created a diverse motion content that still felt warm and handmade.
The final animations were uploaded to GitHub and pulled into our p5 sketch for the interactive build.
Reindeer + Bake + Gingerbread House
Putting it all together
While Minami worked on designing the physical interactive piece, I worked on devising the hardware structure and coding environment to support it.
Video Description: A time lapse of hands assembling the felt game pieces. Minami applies hot glue along the edges of colorful felt shapes, and presses them together. The video speeds through the process as soft, handcrafted characters quickly take form.
Video description: A time lapse shows Jackie working through the hardware schematics. Wires are inserted into a breadboard, LEDs are positioned, and magnetic pogo pin sensors are connected to an Arduino. The footage captures the iterative process of testing, adjusting, and troubleshooting until the circuit works.
On December 14th, we presented at NYU's ITP/IMA Winter Show. The reaction was more than we could have hoped for.
Within minutes of opening, a child ran up to our table, grabbed a felt Santa, and dropped it into the pot. She didn't wait for instructions. She just played. That moment repeated throughout the night as kids gravitated towards the bright colors and puppet theater design, eager to cook up their own stories.
Adults were just as engaged as kids!
Watching people interact with the project taught me that lowering the learning barrier doesn't just help children… it makes the experience more inviting and accesible for everyone.
A huge thank you to the people who taught me to think differently, experiment boldly, and challenge myself to design for the unexpected.
Thank you to Professor Danny Rozin, Professor Tom Igoe, Andre Lira, Proud Aiemruksa, Duan, William and Ryan.
