Make a quadcopter land on a frying pan
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Garrett Wilson c2c036b2f9 more TODOs 1 year ago
tests TF Lite OpenCL im2col conv2d 2 years ago
tflite Moved from gpu/ to / 2 years ago
.gitignore ignore png images 1 year ago added Youtube link 1 year ago more detailed README 1 year ago Couple scripts for viewing resulting video easily 2 years ago control that might work 1 year ago more TODOs 1 year ago
box_priors.txt Working implementation of tflite in numpy 2 years ago attempt at sending detections to autopilot 1 year ago Attempt at camera calibration 2 years ago don't have source system/component that conflicts with GCS 1 year ago
control.service attempt at sending detections to autopilot 1 year ago Photography mode 2 years ago Photography mode 2 years ago Cleanup / renames 2 years ago Couple scripts for viewing resulting video easily 2 years ago Moved from gpu/ to / 2 years ago
labels.txt TF Lite and non-TF Lite version 2 years ago output processed images if desired 1 year ago Photography mode 2 years ago Photography mode 2 years ago
photography.service Photography mode 2 years ago Stream via RSTP for QGroundControl app, use MAVProxy 2 years ago
schema.fbs Moved from gpu/ to / 2 years ago Photography mode 2 years ago
stream.service Changed to standard mavlink TCP port 2 years ago Photography mode 2 years ago Initial TFLite OpenCL version 2 years ago Initial TFLite OpenCL version 2 years ago Make it build on the Pi 2 years ago Initial TFLite OpenCL version 2 years ago send bounding boxes to on RPi 1 year ago
video.service Stream rtsp video at 300x300 2 years ago Photography mode 2 years ago Stream or shutdown on CH6 of R/C controller 2 years ago

Vision Landing

Use the model learned from the Detect Frying Pan code and now run that on live RPi Zero camera input. We want to get the drone to land on the frying pan.

See the demo video on Youtube.

Running Object Detection on another computer

Wiring the Raspberry Pi to the Pixhawk

Connect RPi UART/serial pins to the Serial 4 of the Pixhawk looking at the pinouts of TELEM2 or Serial 4/5 on this or this. Basically, just follow this (i.e. swapping TX and RX between above and RPi) and the RPi diagram (Note: the square pin on the RPi is pin 1). Note: no need for level converter since Pixhawk TX/RX says it's also 3.3V just like the RPi

In APM Planner, under Config/Tuning -> Full Parameter List -> ... set:

SERIAL4_PROTOCOL=1 # default is 5, but 1 is MAVLINK1 and 2 is MAVLINK2
                   # See:
SERIAL4_BAUD=115 # i.e. 115200 which is probably fast enough
                 # See:

Set up the Pi:

sudo raspi-config # Interfacing options -> Serial -> No login screen but Yes Serial Enabled

Edit /boot/config.txt, adding "dtoverlay=pi3-miniuart-bt" to the bottom (src).

Reboot to apply the above boot configuration, then install dependencies:

sudo systemctl reboot
sudo apt install screen python-wxgtk2.8 python-matplotlib python-opencv \
    python-pip python-numpy python-dev libxml2-dev libxslt-dev python-lxml
sudo pip install future pymavlink mavproxy

Raspberry Pi Setup

Since the Zero is really slow, I'll stream to another computer to do processing for now. Though, at least one person has used the GPU on the RPi Zero to get ~8 fps on face detection. So, it is possible, though he hasn't shared his code. I will return to this problem later.

sudo apt install python3-zmq

Also you need to build v4l2rtspserver and mavlink-router and set up a wireless access point.


Download and install:

git clone --recursive
cd v4l2rtspserver/
vim CMakeLists.txt # replace ONLY_CMAKE_FIND_ROOT_PATH with /usr/lib/arm-linux-gnueabihf /usr/local/lib
cmake . # for a total clean before: rm -rf CMakeCache.txt *.a CMakeFiles
sudo make install

Download and install:

git clone --recursive
cd mavlink-router
./ && ./configure CFLAGS='-g -O2' --sysconfdir=/etc --localstatedir=/var/local --libdir=/usr/local/lib --prefix=/usr/local
sudo make install

Create the config file /etc/mavlink-router/main.conf:


[UdpEndpoint groundstation]
Mode = Eavesdropping
Address =
Port = 14550

[UartEndpoint pixhawk]
Device = /dev/ttyAMA0
Baud = 115200

Wireless Access Point

I bought one (actually 3 because I broke 2) of the RT5370 2.4GHz wireless adapters that had built-in Linux drivers (e.g. on Amazon here or here).


Update and install dependencies:

sudo apt update
sudo apt install dnsmasq hostapd


Edit /etc/dhcpcd.conf (assuming your wifi adapter shows up as wlan1 in ip link):

interface wlan1
    static ip_address=
    nohook wpa_supplicant

Edit /etc/dnsmasq.conf:

interface=wlan1      # Use the require wireless interface

Edit /etc/hostapd/hostapd.conf:

channel=1 # whatever channel you want
country_code=US # so it always has the right max power, channels, etc.
wpa_passphrase=password # be sure to set this to something else

Edit /etc/default/hostapd:

#if debugging: DAEMON_OPTS="-dd -t -f /home/pi/hostapd.log"

Edit /etc/network/interfaces.d/wlan1:

allow-hotplug wlan1

Enable and Run

Then, enable everything:

sudo systemctl disable wpa_supplicant@wlan1
sudo systemctl stop wpa_supplicant@wlan1
sudo systemctl restart dhcpcd dnsmasq hostapd
sudo systemctl enable dhcpcd dnsmasq hostapd

Update Firmware

And, if you wish to reduce dmesg errors (src), then download "Mac" version (actually Linux and the Linux is a Mac .dmg file) from get the rt2870.bin file from common/.

mv /lib/firmware/rt2870.bin{,.bak}
mv rt2870.bin /lib/firmware

Force 40 MHz Mode

And, if you wish to force hostapd to use 40 MHz mode:

tar xavf hostapd-2.7.tar.gz
cd hostapd-2.7

sudo apt remove hostapd
sudo apt install build-essential libnl-3-dev iw crda libssl-dev libnl-genl-3-200 libnl-3-200 libnl-genl-3-dev

Then, edit src/ap/hw_features.c and search for search for wpa_scan_results_free or oper40 and set oper40=1 right below the checks (src. Then:

cd hostapd
cp defconfig .config # in .config uncomment CONFIG_IEEE80211N=y and CONFIG_IEEE80211AC=y and CONFIG_ACS=y
sudo make install # installs /usr/local/bin/hostapd{,_cli}

Then, create /etc/systemd/system/hostapd-custom.service:

Description=Hostapd IEEE 802.11 AP, IEEE 802.1X/WPA/WPA2/EAP/RADIUS Authenticator

ExecStart=/usr/local/bin/hostapd /etc/hostapd/hostapd.conf -d
# -d -t -f /home/pi/hostapd.log
ExecReload=/bin/kill -HUP $MAINPID


And switch to using the custom hostapd:

sudo systemctl disable hostapd
sudo systemctl enable hostapd-custom
sudo systemctl daemon-reload
sudo systemctl restart dhcpcd dnsmasq hostapd-custom

Pixhawk Setup

Set these options (yaw reference):

PLND_ENABLED=1 # enable always land
PLND_TYPE=1 # companion computer
PLND_EST_TYPE=1 # for EKF or raw, see which works best for you

# Note: this would be ideal, but it's actually cdeg, so we'd have to set
# this to 27000, but it won't let me set values outside of [0,360]
# thus, we rotate in code
#PLND_YAW_ALIGN=270 # depending on how you mounted the camera on the drone

Then map a switch on your R/C controller to channel 6. For low PPM value it'll do nothing, for higher it'll stream, and for even higher it'll shut down the Raspberry Pi (for exact values, see script).

Optionally, if you wish to enable precision loiter on some R/C channel:



On Pi:

cd ./vision-landing
./ # --device=tcp: or --device=/dev/ttyAMA0 etc.

Or, if you wish to always run on boot (running /home/pi/vision-landing/ as user pi and group dialout for access to /dev/ttyAMA0):

sudo cp control.service /etc/systemd/system/
sudo systemctl enable control mavlink-router
sudo systemctl start control mavlink-router

On laptop (and outputting debug info, saving images to record/, displaying with GStreamer):

sudo pacman -S gst-python
cd ./vision-landing
./ --record record