ENPH 353 · Fall 2023 · Software & machine learning
A robot detective.
Powered by vision.
Read the clues. Navigate the course. React to traffic.
A software-only autonomous robot in a ROS simulation, combining CNN character recognition with image processing, PID navigation, and a competition state machine.
Full size- My focus
- CNN training, clue perception, state machine
- Team
- Sasan Ghasaei & Mahdi Shakouri
- Environment
- UBC ENPH 353 · Simulated competition
- 8 / 8
- Clues read correctly in competition
- 50 / 57
- Final competition score
- 37
- Character classes: A–Z, 0–9, space
Software architecture & ownership
A camera feed becomes useful information.
The course combined paved roads, off-road terrain, moving hazards, and eight clue boards. Our task was to navigate autonomously and submit the topic and value from each board. We separated character recognition from the modules that moved the robot and tracked its progress.
Generated and collected training data, trained the character CNN, and used failure cases to refine it.
Implemented ClueFinder for banner detection and perspective correction, and ClueGuesser for character-by-character recognition.
Implemented the competition state machine. Mahdi developed the driver, road perception, score keeper, confidence safeguard, and state-transition improvements.
That split gave us clear interfaces: camera images into perception, recognized clues into the score keeper, and state-dependent commands into the driver.
OpenCV · Contours · Perspective transforms
Read the board before reading the letters.
The CNN needs a consistent view of each character. ClueFinder first isolates the blue board, locates a four-corner contour, and rectifies its perspective. A second crop removes the border. A Laplacian-variance check rejects blurry frames before the letters reach the network.
Full size
Full size
Full sizeClueGuesser turns each crop into a 37-class probability vector and selects the most likely character. It checks the topic against the eight possible topics before processing the value, avoiding unnecessary work.
CNNs · Dataset design · Error analysis
Good validation scores were only the beginning.
Our first model performed well on generated characters, but struggled with images from the simulated robot’s camera. The important work was closing that data gap: collecting representative examples, identifying confusing characters, and fine-tuning without introducing new bias.
Start with synthetic characters
Generated A–Z, digits, and spaces on clue banners, with blur and horizontal shifts. Initial training used 50 epochs, batch size 16, and a learning rate of 0.0001.
Collect the camera’s actual view
Captured nearly 200 banner images across all eight locations in the simulation. Cropped their characters and fine-tuned for 15 epochs; validation exposed remaining weaknesses.
Target the confusing cases
Collected examples of O/Q/0, I/1, S/5, and P/R. A short five-epoch training round raised reported validation accuracy above 95%.
Refine from complete runs
Tracked the remaining character errors during runs and applied focused one-to-three-epoch updates. Short updates helped limit bias toward the new examples.
The character classifier
784,613 trainable parameters
A Keras sequential CNN, trained using Google Colab. Convolutional layers extract visual features; dense layers classify the character.
| Layer | Output shape |
|---|---|
| Conv2D · 32 filters | 98 × 43 × 32 |
| Max pooling | 49 × 21 × 32 |
| Conv2D · 64 filters | 47 × 19 × 64 |
| Max pooling | 23 × 9 × 64 |
| Conv2D · 128 filters | 21 × 7 × 128 |
| Max pooling | 10 × 3 × 128 |
| Conv2D · 128 filters | 8 × 1 × 128 |
| Flatten + dropout | 1,024 |
| Dense | 512 |
| Character output | 37 |
Explore the training examples and original learning curves
Full size
Full sizeROS · Classical vision · PID · State machines
Keep reading while the world keeps moving.
Navigation used color masks and contour geometry to estimate the road center, then PID-based velocity commands to keep the robot on track. Different masks and contour filters handled paved, dirt, and mountain sections. The CNN’s job was reading clues; road and hazard detection used classical image processing.
Full sizeAt a crossing, changes in a selected image region indicate the pedestrian’s passage before the robot resumes.
A color-based mask detects a nearby truck and triggers a waiting state until it moves away.
Pink-line detections switch driving modes. At the peak, the robot approaches the final clue board until it obtains a usable reading.
Evaluation & engineering trade-offs
Eight correct clues. Clear lessons from the misses.
In competition, the robot read all eight clues correctly and finished with 50 out of 57 points. It received 52 clue points and a two-point respawn deduction. We used a deliberate respawn to skip the tunnel/Yoda segment; the result does not represent a fully continuous traversal of every obstacle.
Full sizeView the final character confusion matrix
Full sizeWhat I took forward
Building this system connected neural-network training to the behavior of a complete autonomous agent. I learned to curate datasets around failure cases, inspect confusion matrices, correct image geometry, and create modular perception software that works with a state machine.
The next improvement would be faster clue inference so navigation can continue smoothly without slowing down to read. Completing the skipped course segment would also remove reliance on the respawn strategy.
Technical details, contribution attribution, figures, and results are drawn from our Fall 2023 final report.