UBC Engineering Physics Capstone · Project 2561 · Wayve

Small cars.
Real-world learning.

Multi-agent reinforcement learning for autonomous micro-scale driving.

We built a physical training platform around modified 1:64 RC cars: learn to drive in NVIDIA Isaac Lab, transfer PPO policies to the real track, then improve them using real-world experience.

Project: ENPH 479 · Sep 2025–Apr 2026
Team: Sasan Ghasaei + four teammates
Sponsors: Dr. Matt Brown & Dr. Daniele Reda
Wayve Technologies Ltd.

Reinforcement learningIsaac LabPPO · Stable Baselines3PyTorch · CNNsSim-to-realComputer visionEmbedded integration

Multiple agents on the physical trackModified RC cars driving together in the real environment. Simulation training and real-world fine-tuning are shown separately below.

Zero-shot deployment

Simulation-trained policies drive on the real track without additional training.

Real-world PPO

Single-agent fine-tuning improves driving using physical rollouts and automatic resets.

Multiple physical agents

A shared camera and individually addressed cars support simultaneous driving.

Systems integration · Embedded control

A miniature platform for a real research problem

Full-size autonomous-driving experiments are expensive and difficult to repeat. Our capstone goal was to create an accessible system for training multiple driving agents in real life. A 2.5 × 2.5 m modular track gives us a controlled place to explore the gap between simulated and physical driving.

Modified micro RC car carrying a large ArUco marker — The physical agent: a 1:64 Ackermann-steered RC car with an ArUco marker and upgraded battery mounting.

Three modified RC transmitters connected to ESP32 controller electronics — The control unit: dedicated transmitter circuits allow the central computer to command individual vehicles.

An overhead FLIR camera localizes the vehicles. The central computer computes observations and runs policy inference, then sends car-ID-prefixed commands over USB serial to a master ESP32. ESP-NOW routes each command to the corresponding receiver; DAC outputs drive the modified RC transmitter. Computation stays on the host computer.

Hardware platform diagram connecting overhead camera, central GPU computer, ESP32 and DAC transmitter circuits to RC cars — Original hardware diagram · Poster, page 3. The camera closes the loop around the computer, communication bridge, and physical vehicles. The poster lists a 118 fps camera; this is distinct from the complete control-loop rate.

Computer vision · Coordinate transforms · Observation design

Make simulation and reality speak the same language

The policy does not need the entire camera image. ArUco markers provide vehicle position and heading; HSV segmentation separates the track and obstacles. The pipeline crops and rotates these maps around each car so that observations are expressed in the agent’s own frame.

Vision pipeline showing two tracked cars and their individual track and obstacle image channels — Original computer-vision diagram · Poster, page 4. One overhead view becomes a different local observation for each car.

48 × 48 pixels

A binary track crop gives the line-following policy its local visual context.

Velocity input

A normalized velocity vector supplies motion information alongside the image.

Obstacle channel

A second binary image encodes obstacles and, for multi-agent tasks, nearby cars.

Matching crop scale, orientation, normalization, and action conventions across both environments makes the learned policy reusable. In the multi-agent simulation, stacked frames also give the policy temporal information about other cars’ motion.

CNN feature extraction · Actor–critic learning · PPO

From a track image to steering and throttle

We trained policies with Proximal Policy Optimization (PPO) using Stable Baselines3. The line-following model combines a convolutional image encoder with a velocity encoder, then uses an actor–critic architecture to learn continuous driving actions.

CNN actor-critic architecture: 48 by 48 image through three convolutional layers to 128 features, two velocity inputs to 32 features, concatenated 160 features feeding actor and critic — Original model architecture · Poster, page 5. Open the full-size figure to inspect every layer. This is the single-channel line-following model; obstacle and multi-agent observations extend the visual input.

Reading the network
Stage	Representation	Purpose
Visual encoder	1 × 48 × 48 → 16 × 24 × 24 → 32 × 12 × 12 → 32 × 6 × 6	Extract local track geometry through three convolutional layers.
Feature fusion	1,152 flattened values → 128 image features; 2 velocity inputs → 32 features	Concatenate vision and motion into a 160-dimensional representation.
Shared layers	160 → 128 → 128, with ReLU	Combine spatial context and vehicle motion.
Actor and critic	Separate branches, each with two 64-unit hidden layers	The actor produces steering and throttle; the critic estimates expected discounted return.

PPO alternates between collecting rollouts and updating the policy with a clipped objective. Reward design connects that optimization to useful driving: stay close to the line, move forward, remain on track, and discourage erratic steering. The critic supports training; the actor selects the driving action.

Isaac Lab · Vectorized environments · Reward design

Build driving skills in simulation first

Our Isaac Lab environment models an Ackermann vehicle, track geometry, and overhead observations. Parallel environments make it practical to test rewards and observations quickly. The report describes 12 track variants for line following, with static track images cached as tensors to avoid repeated rendering. Observations, rewards, actions, and terminations are vectorized for GPU execution.

Parallel Isaac Lab track environments and a timeline from simulation training to real-world adaptation — Original simulation diagram · Poster, page 3. Simulation builds the initial driving skill; physical training adapts it to the real system.

Line following

The foundation: learn to follow the visible path using the local track image and velocity. An episode ends when the track leaves the observation, and the simulated car resets for another attempt.

Obstacle avoidance

Add a second image channel for obstacle locations. The policy must leave the line when needed and rejoin it afterward, with collision handling added to the task.

Obstacle avoidance on the real trackPhysical deployment of the obstacle-aware driving task, from the project presentation.

Multi-agent driving

Other cars appear in each agent’s obstacle channel. Experiences from multiple simulated agents train a shared policy, with frame stacking to represent motion. This creates interactions such as following another vehicle or meeting an oncoming car.

Multi-agent policies in Isaac LabThe supplied simulation demonstration shows trained agents sharing a driving environment.

Sim-to-real transfer · Real-world rollouts · Automatic recovery

Deploy the policy. Then let the real world teach it.

Zero-shot transfer means deploying the simulation-trained policy without further learning. It worked well for basic driving on our physical track. But successful transfer does not eliminate the reality gap: motor response, tire slip, battery state, sensing noise, and control delays can still affect speed and smoothness.

The action interface stays consistent: normalized steering and throttle commands are mapped to simulated joint controls or to the physical controller’s 8-bit command range. Rather than trying to model every physical effect perfectly, we used the pretrained policy as the starting point for real-world PPO fine-tuning.

Real-world PPO loop from camera frame to observation, policy inference, ESP32 action, reward, PPO update and automatic reset — Original real-world training loop · Poster, page 5. Physical rollouts feed PPO updates. The diagram’s approximately 30 ms frame loop describes this setup, not a guaranteed timing bound.

The crucial difference: a real car cannot teleport

Resetting in simulation takes one command. Resetting in reality requires driving the car back. We trained a separate goal-reaching policy from a vector observation: displacement to the goal, nearest boundary, heading error, and speed. It learns to approach the target, align, and stop, allowing driving episodes to restart automatically.

Reset-agent task with goal vector, boundary vector, heading error, speed, reward and termination conditions — Poster, page 4 · A separate recovery task makes repeated real-world training possible.

Zero-shot reset agentA simulation-trained recovery policy deployed to the physical environment.

Real-world training timelapseRepeated physical rollouts and resets. This is an accelerated recording, not real-time footage.

Evaluation · Experiment design · Engineering tradeoffs

What the platform demonstrated

Results and their scope
Capability	Demonstrated outcome
Zero-shot transfer	Simulation-trained policies drove on the physical platform. Basic line following transferred successfully, while oscillations and higher-speed tracking remained motivations for adaptation.
Single-agent fine-tuning	The final report records approximately 170,000 real-world steps over 3–4 hours, with learning rate reduced from 3 × 10⁻⁴ to 1 × 10⁻⁴. It reports improved line-following performance after fine-tuning.
Physical multi-agent driving	Multiple cars drove in the real environment, as shown in the supplied demonstration. This is distinct from jointly fine-tuning multiple agents in reality.
Scalable control	The report documents three simultaneous vehicles commanded at 60 Hz. Camera frame rate, command rate, and end-to-end policy timing measure different parts of the system.

Line following in Isaac LabSimulation footage of the line-following task from the final presentation.

Multiple cars with shared policy inferenceAdditional presentation footage of multiple cars running a shared policy on the physical track.

The April 2026 report describes an earlier project snapshot. The physical multi-agent result above incorporates the subsequent demonstration supplied for this case study. The recordings show behavior; they are not a controlled before-and-after benchmark.

Skills I developed through this capstone

This five-person project connected reinforcement learning to a working physical system. The technical work strengthened my experience with Isaac Lab, PPO and actor–critic models, CNN observations, reward and termination design, simulation-to-real interfaces, and evaluating policies through real-world experiments.

It also developed my systems-integration experience: understanding how visual tracking, coordinate frames, communication, vehicle dynamics, and automatic recovery affect the learning loop as a whole.

Further research

From a working platform to richer interactions

The next extension is more complex paths, denser obstacles, and more physical agents trained and fine-tuned together. The infrastructure now supports asking harder questions about shared policies, interaction, recovery, and adaptation.

Introduce intersections and more challenging path geometry.
Increase obstacle density and evaluate robustness across layouts.
Scale real-world training and fine-tuning to multiple agents.
Measure collisions, completion rate, tracking error, and recovery time across controlled experiments.

People behind the platform

A team-built research testbed

Developed by Sasan Ghasaei, Felipe Garavelli, William Gibbs, Itai Boss, and Mahdi Shakouri Ganjavi for UBC Engineering Physics, sponsored by Dr. Matt Brown and Dr. Daniele Reda at Wayve Technologies Ltd.

The architecture and results presented here reflect the team’s capstone work.

Project poster · PDF Final report · PDF

Capstone team and sponsor standing at the physical RC-car testbed — Left to right: Dr. Daniele Reda, Felipe Garavelli, Sasan Ghasaei, William Gibbs, Itai Boss, and Mahdi Shakouri Ganjavi.

Project recognition

Roy Nodwell Prize

Our capstone team received the Roy Nodwell Prize, recognizing a project of the highest standard for its originality and industrial relevance.

The prize commemorates Roy Nodwell’s retirement in 1983 as head of UBC’s Department of Physics and Astronomy.

The five-member Wayve capstone team standing with the Roy Nodwell Prize plaque — Our team with the Roy Nodwell Prize, awarded for the capstone research testbed.

Small cars.Real-world learning.