Aron Petau - coral

AIRASPI Build Log

2024-01-30T00:00:00+00:00

AI-Raspi Build Log</h2>
This document chronicles the process of building a custom edge computing device for real-time image recognition and object detection. The goal was to create a portable, self-contained system that could operate independently of cloud infrastructure.</p>
Project Goals:</strong></p>
Build an edge device with image recognition and object detection capabilities that can process video in real-time, targeting 30fps at 720p resolution. Portability and autonomous operation are critical requirements—the device must function without an active internet connection and maintain a compact form factor suitable for installation environments. All computation happens locally on the device itself, making it a true edge computing solution with no cloud dependency.</p>
This project was inspired by pose2art</a>, which demonstrated the creative potential of real-time pose detection for interactive installations.</p>

Hardware</h2>

Raspberry Pi 5</a></li>
Raspberry Pi Camera Module v1.3</a></li>
Raspberry Pi GlobalShutter Camera</a></li>
2x CSI FPC Cable (needs one compact side to fit pi 5)</li>
Pineberry AI Hat (m.2 E key)</a></li>
Coral Dual Edge TPU (m.2 E key)</a></li>
Raspi Official 5A Power Supply</li>
Raspi active cooler</li> </ul>
Setup</h2>
Primary Resources</h3>
This build wouldn't have been possible without the excellent documentation and troubleshooting guides from the community. The primary sources I relied on throughout this project were:</p>

coral.ai official documentation</a> - Google's official setup guide for the M.2 Edge TPU</li>
Jeff Geerling's blog</a> - Critical PCIe configuration insights for Raspberry Pi 5</li>
Frigate NVR documentation</a> - Comprehensive guide for the network video recorder software</li> </ul>
Raspberry Pi OS Installation</h3>
I used the Raspberry Pi Imager to flash the latest Raspberry Pi OS to an SD card. The OS choice is critical for camera compatibility.</p>

Needs to be Debian Bookworm. Needs to be the full arm64 image (with desktop), otherwise you will get into camera driver hell.</p> </blockquote>
Initial Configuration Settings:</strong></p>
Using the Raspberry Pi Imager's advanced settings, I configured the following before flashing:</p>

Used the default arm64 image (with desktop) - critical for camera driver compatibility</li>
Enabled custom settings for headless operation</li>
Enabled SSH for remote access</li>
Configured WiFi country code for legal compliance</li>
Set WiFi SSID and password for automatic network connection</li>
Configured locale settings for proper timezone and keyboard layout</li>
Set custom hostname: `airaspi</code> for easy network identification</li> </ul>`System Update</h3> After the initial boot, updating the system is essential. This process can take considerable time with the full desktop image, but ensures all packages are current and security patches are applied.</p> sudo apt update && sudo apt upgrade -y && sudo reboot </code></pre> Preparing the System for Coral TPU</h3> The Raspberry Pi 5's PCIe interface requires specific configuration to work with the Coral Edge TPU. This section was the most technically challenging, involving kernel modifications and device tree changes. A huge thanks to Jeff Geerling for documenting this process—without his detailed troubleshooting, this would have been nearly impossible.</p> # check kernel version uname -a </code></pre> # modify config.txt sudo nano /boot/firmware/config.txt </code></pre> While in the file, add the following lines:</p> kernel=kernel8.img dtparam=pciex1 dtparam=pciex1_gen=2 </code></pre> Save and reboot:</p> sudo reboot </code></pre> # check kernel version again uname -a </code></pre> should be different now, with a -v8 at the end</li> </ul> edit /boot/firmware/cmdline.txt</p> sudo nano /boot/firmware/cmdline.txt </code></pre> add pcie_aspm=off before rootwait</li> </ul> sudo reboot </code></pre> Modifying the Device Tree</h3> Initial Script Attempt (Deprecated)</h4> Initially, there was an automated script available that was supposed to handle the device tree modifications. However, this script proved problematic and caused issues during my build.</p> maybe this script is the issue? i will try again without it</p> </blockquote> curl https://gist.githubusercontent.com/dataslayermedia/714ec5a9601249d9ee754919dea49c7e/raw/32d21f73bd1ebb33854c2b059e94abe7767c3d7e/coral-ai-pcie-edge-tpu-raspberrypi-5-setup | sh </code></pre> Yes, it was the problematic script. I left a comment documenting the issue on the original gist: My comment on the gist</a></p> Manual Device Tree Modification (Recommended)</h4> Instead of relying on the automated script, I followed Jeff Geerling's manual approach. This method gives you complete control over the process and helps understand what's actually happening under the hood.</p> In the meantime the Script got updated and it is now recommended again.</p> </blockquote> The device tree modification process involves backing up the current device tree blob (DTB), decompiling it to a readable format, editing the MSI parent reference to fix PCIe compatibility issues, and then recompiling it back to binary format. Here's the step-by-step process:</p> 1. Back up and Decompile the Device Tree</strong></p> # Back up the current dtb sudo cp /boot/firmware/bcm2712-rpi-5-b.dtb /boot/firmware/bcm2712-rpi-5-b.dtb.bak # Decompile the current dtb (ignore warnings) dtc -I dtb -O dts /boot/firmware/bcm2712-rpi-5-b.dtb -o ~/test.dts # Edit the file nano ~/test.dts # Change the line: msi-parent = <0x2f>; (under `pcie@110000`) # To: msi-parent = <0x66>; # Then save the file. # Recompile the dtb and move it back to the firmware directory dtc -I dts -O dtb ~/test.dts -o ~/test.dtb sudo mv ~/test.dtb /boot/firmware/bcm2712-rpi-5-b.dtb # Reboot for changes to take effect sudo reboot </code></pre> Note: msi-parent seems to carry the value <0x2c> nowadays, cost me a few hours.</p> </blockquote> 2. Verify the Changes</strong></p> After rebooting, check that the Coral TPU is recognized by the system:</p> lspci -nn | grep 089a </code></pre> You should see output similar to: 0000:01:00.0 System peripheral [0880]: Global Unichip Corp. Coral Edge TPU [1ac1:089a]</code></p> Installing the Apex Driver</h3> With the device tree properly configured, the next step is installing Google's Apex driver for the Coral Edge TPU. This driver enables communication between the operating system and the TPU hardware.</p> Following the official instructions from coral.ai</a>:</p> echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add - sudo apt-get update sudo apt-get install gasket-dkms libedgetpu1-std sudo sh -c "echo 'SUBSYSTEM==\"apex\", MODE=\"0660\", GROUP=\"apex\"' >> /etc/udev/rules.d/65-apex.rules" sudo groupadd apex sudo adduser $USER apex sudo reboot </code></pre> This sequence:</p> Adds Google's package repository and GPG key</li> Installs the gasket DKMS module (kernel driver) and Edge TPU runtime library</li> Creates udev rules for device permissions</li> Creates an apex</code> group and adds your user to it</li> Reboots to load the driver</li> </ol> After the reboot, verify the installation:</p> lspci -nn | grep 089a </code></pre> This should display the connected Coral TPU as a PCIe device.</p> Next, confirm the device node exists with proper permissions:</p> ls -l /dev/apex_0 </code></pre> If the output shows /dev/apex_0</code> with appropriate group permissions, the installation was successful. If not, review the udev rules and group membership.</p> Testing with Example Models</h3> To verify the TPU is functioning correctly, we'll use Google's example classification script with a pre-trained MobileNet model:</p> # Install Python packages sudo apt-get install python3-pycoral # Download example code and models mkdir -p ~/coral && cd ~/coral git clone https://github.com/google-coral/pycoral.git cd pycoral # Run bird classification example python3 examples/classify_image.py \ --model test_data/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite \ --labels test_data/inat_bird_labels.txt \ --input test_data/parrot.jpg </code></pre> The output should show inference results with confidence scores, confirming the Edge TPU is working correctly.</p> Docker Installation</h3> Docker provides containerization for the applications we'll be running (Frigate, MediaMTX, etc.). This keeps dependencies isolated and makes deployment much cleaner.</p> Install Docker using the official convenience script from docker.com</a>:</p> curl -fsSL https://get.docker.com -o get-docker.sh sudo sh get-docker.sh sudo usermod -aG docker $USER </code></pre> After installation, log out and back in for group membership changes to take effect.</p> Configure Docker to start automatically on boot:</p> sudo systemctl enable docker.service sudo systemctl enable containerd.service </code></pre> Test the Edge TPU (Optional)</h3> To verify the Edge TPU works inside a Docker container, we can build a test image. This is particularly useful if you plan to use the TPU with containerized applications.</p> Create a test directory and Dockerfile:</p> mkdir coraltest cd coraltest sudo nano Dockerfile </code></pre> Into the new file, paste:</p> FROM debian:10 WORKDIR /home ENV HOME /home RUN cd ~ RUN apt-get update RUN apt-get install -y git nano python3-pip python-dev pkg-config wget usbutils curl RUN echo "deb https://packages.cloud.google.com/apt coral-edgetpu-stable main" \ | tee /etc/apt/sources.list.d/coral-edgetpu.list RUN curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | apt-key add - RUN apt-get update RUN apt-get install -y edgetpu-examples RUN apt-get install libedgetpu1-std CMD /bin/bash </code></pre> Build and run the test container, passing through the Coral device:</p> # build the docker container docker build -t "coral" . # run the docker container docker run -it --device /dev/apex_0:/dev/apex_0 coral /bin/bash </code></pre> Inside the container, run an inference example:</p> # run an inference example from within the container python3 /usr/share/edgetpu/examples/classify_image.py --model /usr/share/edgetpu/examples/models/mobilenet_v2_1.0_224_inat_bird_quant_edgetpu.tflite --label /usr/share/edgetpu/examples/models/inat_bird_labels.txt --image /usr/share/edgetpu/examples/images/bird.bmp </code></pre> You should see inference results with confidence values from the Edge TPU. If not, try a clean restart of the system.</p> Portainer (Optional)</h3> Portainer provides a web-based GUI for managing Docker containers, images, and volumes. While not required, it makes container management significantly more convenient.</p> This is optional, gives you a browser GUI for your various docker containers.</p> </blockquote> Install Portainer:</p> docker volume create portainer_data docker run -d -p 8000:8000 -p 9443:9443 --name portainer --restart=always -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer-ce:latest </code></pre> Access Portainer in your browser and set an admin password:</p> Navigate to: https://airaspi.local:9443</a></li> </ul> VNC Setup (Optional)</h3> VNC provides remote desktop access to your headless Raspberry Pi. This is particularly useful for testing cameras and debugging visual issues without connecting a physical monitor.</p> This is optional, useful to test your cameras on your headless device. You could attach a monitor, but I find VNC more convenient.</p> </blockquote> Enable VNC through the Raspberry Pi configuration tool:</p> sudo raspi-config </code></pre> Navigate to: Interface Options</strong> → VNC</strong> → Enable</strong></p> Connecting through VNC Viewer</h3> Install RealVNC Viewer</a> on your computer (available for macOS, Windows, and Linux).</p> Connect using the address: airaspi.local:5900</code></p> You'll be prompted for your Raspberry Pi username and password. Once connected, you'll have full remote desktop access for testing cameras and debugging.</p> Frigate NVR Setup</h2> Frigate is a complete Network Video Recorder (NVR) with real-time object detection powered by the Coral Edge TPU. It's the heart of this edge AI system.</p> Docker Compose Configuration</h3> This setup uses Docker Compose to define the Frigate container with all necessary configurations. If you're using Portainer, you can add this as a custom stack.</p> Important: you need to change the paths to your own paths.</p> </blockquote> version: "3.9" services: frigate: container_name: frigate privileged: true # this may not be necessary for all setups restart: unless-stopped image: ghcr.io/blakeblackshear/frigate:stable shm_size: "64mb" # update for your cameras based on calculation above devices: - /dev/apex_0:/dev/apex_0 # passes a PCIe Coral, follow driver instructions here https://coral.ai/docs/m2/get-started/#2a-on-linux volumes: - /etc/localtime:/etc/localtime:ro - /home/aron/frigate/config.yml:/config/config.yml # replace with your config file - /home/aron/frigate/storage:/media/frigate # replace with your storage directory - type: tmpfs # Optional: 1GB of memory, reduces SSD/SD Card wear target: /tmp/cache tmpfs: size: 1000000000 ports: - "5000:5000" - "8554:8554" # RTSP feeds - "8555:8555/tcp" # WebRTC over tcp - "8555:8555/udp" # WebRTC over udp environment: FRIGATE_RTSP_PASSWORD: "******" </code></pre> Key configuration points in this Docker Compose file:</p> Privileged mode</strong> and device mappings</strong>: Required for accessing hardware (TPU, cameras)</li> Shared memory size</strong>: Allocated for processing video frames efficiently</li> Port mappings</strong>: Exposes Frigate's web UI (5000) and RTSP streams (8554)</li> Volume mounts</strong>: Persists recordings, config, and database</li> </ul> Frigate Configuration File</h3> Frigate requires a YAML configuration file to define cameras, detectors, and detection zones. Create this file at the path you specified in the docker-compose file (e.g., /home/aron/frigate/config.yml</code>).</p> This is necessary just once. Afterwards, you will be able to change the config in the GUI.</p> </blockquote> Here's a working configuration using the Coral TPU:</p> mqtt: enabled: False detectors: cpu1: type: cpu num_threads: 3 coral_pci: type: edgetpu device: pci cameras: cam1: # <++++++ Name the camera ffmpeg: hwaccel_args: preset-rpi-64-h264 inputs: - path: rtsp://192.168.1.58:8900/cam1 roles: - detect cam2: # <++++++ Name the camera ffmpeg: hwaccel_args: preset-rpi-64-h264 inputs: - path: rtsp://192.168.1.58:8900/cam2 roles: - detect detect: enabled: True # <+++- disable detection until you have a working camera feed width: 1280 # <+++- update for your camera's resolution height: 720 # <+++- update for your camera's resolution </code></pre> This configuration:</p> Disables MQTT</strong>: Simplifies setup for local-only operation</li> Defines two detectors</strong>: A Coral TPU detector (coral</code>) and a CPU fallback</li> Uses default detection model</strong>: Frigate includes a pre-trained model</li> Configures two cameras</strong>: Both set to 1280x720 resolution</li> Uses hardware acceleration</strong>: preset-rpi-64-h264</code> for Raspberry Pi 5</li> Detection zones</strong>: Enable only when camera feeds are working properly</li> </ul> MediaMTX Setup</h2> MediaMTX is a real-time media server that handles streaming from the Raspberry Pi cameras to Frigate. It's necessary because Frigate doesn't directly support libcamera</code> (the modern Raspberry Pi camera stack).</p> Install MediaMTX directly on the system (not via Docker - the Docker version has compatibility issues with libcamera).</p> Double-check the chip architecture when downloading - this caused me significant headaches during setup.</p> </blockquote> Download and install MediaMTX:</p> mkdir mediamtx cd mediamtx wget https://github.com/bluenviron/mediamtx/releases/download/v1.5.0/mediamtx_v1.5.0_linux_arm64v8.tar.gz tar xzvf mediamtx_v1.5.0_linux_arm64v8.tar.gz && rm mediamtx_v1.5.0_linux_arm64v8.tar.gz </code></pre> MediaMTX Configuration</h3> Edit the mediamtx.yml</code> file to configure camera streams. The configuration below uses rpicam-vid</code> (Raspberry Pi's modern camera tool) piped through FFmpeg to create RTSP streams.</p> Add the following to the paths</code> section in mediamtx.yml</code>:</p> paths: cam1: runOnInit: bash -c 'rpicam-vid -t 0 --camera 0 --nopreview --codec yuv420 --width 1280 --height 720 --inline --listen -o - | ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -i /dev/stdin -c:v libx264 -preset ultrafast -tune zerolatency -f rtsp rtsp://localhost:$RTSP_PORT/$MTX_PATH' runOnInitRestart: yes cam2: runOnInit: bash -c 'rpicam-vid -t 0 --camera 1 --nopreview --codec yuv420 --width 1280 --height 720 --inline --listen -o - | ffmpeg -f rawvideo -pix_fmt yuv420p -s:v 1280x720 -i /dev/stdin -c:v libx264 -preset ultrafast -tune zerolatency -f rtsp rtsp://localhost:$RTSP_PORT/$MTX_PATH' runOnInitRestart: yes </code></pre> This configuration:</p> cam1</code> and cam2</code></strong>: Define two camera paths</li> rpicam-vid</code></strong>: Captures YUV420 video from Raspberry Pi cameras</li> ffmpeg</code></strong>: Transcodes the raw video to H.264 RTSP stream</li> runOnInitRestart: yes</code></strong>: Automatically restarts the stream if it fails</li> </ul> Port Configuration</h3> Change the default RTSP port to avoid conflicts with Frigate:</p> In mediamtx.yml</code>, change:</p> rtspAddress: :8554 </code></pre> To:</p> rtspAddress: :8900 </code></pre> Otherwise there will be a port conflict with Frigate.</p> Start MediaMTX</h3> Run MediaMTX in the foreground to verify it's working:</p> ./mediamtx </code></pre> If there are no errors, verify your streams using VLC or another RTSP client:</p> rtsp://airaspi.local:8900/cam1</code></li> rtsp://airaspi.local:8900/cam2</code></li> </ul> Note: Default RTSP port is 8554, but we changed it to 8900 in the config.</p> Current Status and Performance</h2> What's Working</h3> The system successfully streams from both cameras at 30fps and 720p resolution. The Coral Edge TPU performs object detection with minimal latency - the TPU itself is not breaking a sweat, maintaining consistently high performance.</p> According to Frigate documentation, the TPU can handle up to 10 cameras, so there's significant headroom for expansion.</p> Current Issues</h3> However, there are several significant problems hampering the system:</p> 1. Frigate Display Limitations</strong></p> Frigate limits the display FPS to 5, which is depressing to watch, especially since the TPU doesn't even break a sweat. The hardware is clearly capable of much more, but software limitations hold it back.</p> 2. Stream Stability Problems</strong></p> The stream is completely errant and drops frames constantly. I've sometimes observed detect FPS as low as 0.2, but the TPU speed should definitely not be the bottleneck here. One potential solution might be to attach the cameras to a separate device and stream from there.</p> 3. Coral Software Abandonment</strong></p> The biggest issue is that Google seems to have abandoned the Coral ecosystem, even though they just released new hardware for it. Their most recent Python build supports only Python 3.9.</p> Specifically, pycoral</code> appears to be the problem - without a decent update, I'm confined to Debian 10 with Python 3.7.3. That sucks. There are custom wheels available, but nothing that seems plug-and-play.</p> This severely limits the ability to use modern software and libraries with the system.</p> Reflections and Lessons Learned</h2> Hardware Decisions</h3> The M.2 E Key Choice</strong></p> The decision to go for the M.2 E key version to save money, instead of spending more on the USB version, was a huge mistake. Please do yourself a favor and spend the extra 40 bucks.</p> Technically, it's probably faster and better for continuous operation, but I have yet to feel the benefit of that. The USB version would have offered far more flexibility and easier debugging.</p> Future Development</h2> Several improvements and experiments are planned to enhance this system:</p> Documentation and Visual Aids</strong></p> Add images and screenshots to this build log to make it easier to follow</li> </ul> Mobile Stream Integration</strong></p> Check whether vdo.ninja</a> is a viable way to add mobile streams, enabling smartphone camera integration and evaluation</li> </ul> MediaMTX libcamera Support</strong></p> Reach out to the MediaMTX developers about bumping libcamera support, which would eliminate the current rpicam-vid</code> workaround. I suspect there's quite a lot of performance lost in the current pipeline.</li> </ul> Frigate Configuration Refinement</strong></p> Tweak the Frigate config to enable snapshots and potentially build an image/video database for training custom models later</li> </ul> Storage Expansion</strong></p> Worry about attaching an external SSD and saving the video files on it for long-term storage and analysis</li> </ul> Data Export Capabilities</strong></p> Find a way to export the landmark points from Frigate, potentially sending them via OSC (like in my pose2art</a> project) for creative applications</li> </ul> Dual TPU Access</strong></p> Find a different HAT that lets me access the other TPU - I have the dual version, but can currently only access 1 of the 2 TPUs due to hardware restrictions</li> </ul>