ChanHui Jung
All projects

Course project · Intelligent Robot Crash Lab · 2020

Latte Panda: a mobile robot that sorts cups and collects donations

Latte Panda is a service robot for public recycling. It finds a person holding a disposable cup, sorts the cup into a paper or plastic bin, and invites a donation through a QR code.

Team
6 students
Compute
LattePanda Alpha, Jetson TX2, Arduino Mega
Sensors
RealSense D435, touchscreen
Stack
ROS, C++, YOLOv5, TensorRT, PyQt
Watch the demo video

What a visit looks like

  1. 1 Approach The robot drives around, finds a person and greets them with a motion and a short introduction.
  2. 2 Sort It asks for a disposable cup, recognizes it as paper or plastic, and tilts the matching bin open.
  3. 3 Donate The touchscreen offers a donation by QR code and plays a thank-you video.

The system

ROS NODES color depth color detections mode + image TCP /yolo_mode /yolo_result /point /tilt, /motion /servo /speed_set /TerminalToPyqt RealSense D435 color + depth Jetson TX2 YOLOv5 + TensorRT YOLO server person · cup · hand sign Point manager depth grid · person distance Robot core state machine Dynamixel bin tilting panda motion Arduino Mega servo drive motors, PID Touchscreen UI PyQt
Detection runs on a separate Jetson TX2 reached over TCP; the rest runs as ROS nodes. The robot core is a state machine that calls the perception servers and drives the motors and the touchscreen UI.
DetectionColor frames go to a YOLO server, which sends them to the Jetson TX2 with a mode: find people, classify a cup as paper or plastic, or read a hand sign. YOLOv5 ran on the TX2 with TensorRT.
Obstacle avoidanceThe 640×480 depth image is low-pass filtered, averaged over 20 frames and split into a 3×5 grid. The nearest distance in each cell, weighted by a sigmoid, sets the left and right wheel speeds.
Following a personBearing comes from the horizontal position of the detection; distance is the mean depth over a 100×100 window at its center.
ActuationDynamixel motors play the robot's greeting motions and tilt the bins; an Arduino Mega runs the drive motors with PID control over rosserial.

My part

On a team of six I worked on the system design, motor control, communication, sensor control, the algorithms, the electrical system and the UI.

The robot in its white mug-shaped shell with a toy panda on top, on stage in front of a projection screen
The robot in its mug-shaped shell at the fall 2020 Crash Lab demo. Frame from the team's demo video.