Add RPI Ubuntu scripts

[jmachuca77]

This PR adds scripts to create an APSync image using the official Ubuntu LTS image for RPI. The image created contains AP Streaming instead of the old video streaming, Also includes two scripts one to install the master branch of mavlink-router and one for the old async version. However the master branch has advantages and is more stable, but does not support multiple clients in the same port.
Instructions are a work in progress.
Testing is in progress.
Full image file can be found here:

https://s3-us-west-1.amazonaws.com/aero.element.shared/apsync-rpi-20200225025419.img.xz

[davidbuzz]

Folder is called RPI2 , Doc appears to to refer to RPI3 and RPI4, and title makes no attempt to clarify. If this is going to work across all of "RPI2-3-4", then maybe that's the better folder name? Also, there appears to be quite a bit of overlap with the 'raspbian' scripts and 'common/ubuntu', wondering out loud if theres a way to reduce duplicarion.

[rmackay9]

I think it's probably the folder name that is the issue. So we should probably rename RPI2 to just RPI. I don't think we should hold up this PR for the rename though.

@jmachuca77 has at least one additional change to add based on testing that we did today.

[davidbuzz]

i have no strong opinion either way, I was just noting the inconsistency, and that I read most of it. :-)

[jmachuca77]

About the folder name. I thought about that, but the issue is changing the RPI2 folder name would probably affect the old scripts and we would have to update them. I do think its confusing though. I updated the scripts and instructions, I need to add the instructions to create the final image file and then we should be good to go. In the mean time if anyone would like to test the scripts and instructions please do so!

[jmachuca77]

I moved the Realsense build scripts into common, they should work perfectly fine for the Nvidia boards as that's what they were written for. Only change to use them with those boards is to add the CUDA support when calling the scripts from the nVidia specific scripts.
I also added the 5th automated script which automates the installation of the Realsense drivers.
Still need to add instructions for creating the distributable image file. And Instructions to disable the console input to uBoot as if this is not disabled and a flight controller is connected to the serial port at boot, the boot process is interrupted.

@peterbarker do you know how to get the fw_setenv and fw_printenv package to work? Its provided by u-boot-tools, but after installing and trying to run it it complains with

apsync@apsync:~$ fw_printenv
Cannot parse config file '/etc/fw_env.config': No such file or directory

I googled it and it seems to need some sort of setup but I can't find exactly what needs to be done. If we can get that working we can automate the disabling of the console. For now it needs to be done manually by interrupting uboot and changing the env variable or modifying the uboot binary directly and change the bootdelay env variable to -2, but I don't know if that is a good idea:

apsync@apsync:/boot/firmware$ grep -rnw  . -e bootdelay
Binary file ./uboot_rpi_4.bin matches
Binary file ./uboot_rpi_3.bin matches
apsync@apsync:/boot/firmware$ cat uboot_rpi_4.bin | grep bootdelay -a
   Loading Device Tree to %p, end %p ... fdt move failedbootdelaykernel-offsetkernaddrrootdisk-offsetrootaddrBad Linux ARM64 Image magic!
simple-framebufferNo closing ']' bracketNo closing bracketUnbalanced brackets|.^$*+?()[\%.*sInsufficient RAM for page table: 0x%lx > 0x%lx. Please increase the size in get_page_table_size()bootcmd=run distro_bootcmdbootdelay=2baudrate=115200preboot=usb start; if test ! "mmc dev 0"; then fatload usb 0:1 0x3000000 uboot.env; env import -b 0x3000000; fi;loadaddr=0x00200000arch=armcpu=armv8board=rpiboard_name=rpivendor=raspberrypisoc=bcm283xdhcpuboot=usb start; dhcp u-boot.uimg; bootmstdin=serial,usbkbdstdout=serial,vidconsolestderr=serial,vidconsolefdt_high=ffffffffffffffffinitrd_high=ffffffffffffffffkernel_addr_r=0x00080000scriptaddr=0x02400000pxefile_addr_r=0x02500000fdt_addr_r=0x02600000ramdisk_addr_r=0x02700000mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fiboot_net_usb_start=usb startusb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fiboot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fiload_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}efi_dtb_prefixes=/ /dtb/ /dtb/current/scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfileboot_prefixes=/ /boot/boot_scripts=boot.scr.uimg boot.scrboot_script_dhcp=boot.scr.uimgboot_targets=mmc0 mmc1 usb0 pxe dhcp boot_syslinux_conf=extlinux/extlinux.confboot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fiboot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; donescan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplistbootcmd_mmc0=devnum=0; run mmc_bootbootcmd_mmc1=devnum=1; run mmc_bootbootcmd_usb0=devnum=0; run usb_bootbootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fibootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done%s: null device
apsync@apsync:/boot/firmware$ cat uboot_rpi_3.bin | grep bootdelay -a
   Loading Device Tree to %p, end %p ... fdt move failedbootdelaykernel-offsetkernaddrrootdisk-offsetrootaddrBad Linux ARM64 Image magic!
simple-framebufferNo closing ']' bracketNo closing bracketUnbalanced brackets|.^$*+?()[\%.*sInsufficient RAM for page table: 0x%lx > 0x%lx. Please increase the size in get_page_table_size()bootcmd=run distro_bootcmdbootdelay=2baudrate=115200preboot=usb start; if test ! "mmc dev 0"; then fatload usb 0:1 0x3000000 uboot.env; env import -b 0x3000000; fi;loadaddr=0x00200000arch=armcpu=armv8board=rpiboard_name=rpivendor=raspberrypisoc=bcm283xdhcpuboot=usb start; dhcp u-boot.uimg; bootmstdin=serial,usbkbdstdout=serial,vidconsolestderr=serial,vidconsolefdt_high=ffffffffffffffffinitrd_high=ffffffffffffffffkernel_addr_r=0x00080000scriptaddr=0x02400000pxefile_addr_r=0x02500000fdt_addr_r=0x02600000ramdisk_addr_r=0x02700000mmc_boot=if mmc dev ${devnum}; then devtype=mmc; run scan_dev_for_boot_part; fiboot_net_usb_start=usb startusb_boot=usb start; if usb dev ${devnum}; then devtype=usb; run scan_dev_for_boot_part; fiboot_efi_binary=if fdt addr ${fdt_addr_r}; then bootefi bootmgr ${fdt_addr_r};else bootefi bootmgr ${fdtcontroladdr};fi;load ${devtype} ${devnum}:${distro_bootpart} ${kernel_addr_r} efi/boot/bootaa64.efi; if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r};else bootefi ${kernel_addr_r} ${fdtcontroladdr};fiload_efi_dtb=load ${devtype} ${devnum}:${distro_bootpart} ${fdt_addr_r} ${prefix}${efi_fdtfile}efi_dtb_prefixes=/ /dtb/ /dtb/current/scan_dev_for_efi=setenv efi_fdtfile ${fdtfile}; for prefix in ${efi_dtb_prefixes}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${efi_fdtfile}; then run load_efi_dtb; fi;done;if test -e ${devtype} ${devnum}:${distro_bootpart} efi/boot/bootaa64.efi; then echo Found EFI removable media binary efi/boot/bootaa64.efi; run boot_efi_binary; echo EFI LOAD FAILED: continuing...; fi; setenv efi_fdtfileboot_prefixes=/ /boot/boot_scripts=boot.scr.uimg boot.scrboot_script_dhcp=boot.scr.uimgboot_targets=mmc0 mmc1 usb0 pxe dhcp boot_syslinux_conf=extlinux/extlinux.confboot_extlinux=sysboot ${devtype} ${devnum}:${distro_bootpart} any ${scriptaddr} ${prefix}${boot_syslinux_conf}scan_dev_for_extlinux=if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${boot_syslinux_conf}; then echo Found ${prefix}${boot_syslinux_conf}; run boot_extlinux; echo SCRIPT FAILED: continuing...; fiboot_a_script=load ${devtype} ${devnum}:${distro_bootpart} ${scriptaddr} ${prefix}${script}; source ${scriptaddr}scan_dev_for_scripts=for script in ${boot_scripts}; do if test -e ${devtype} ${devnum}:${distro_bootpart} ${prefix}${script}; then echo Found U-Boot script ${prefix}${script}; run boot_a_script; echo SCRIPT FAILED: continuing...; fi; donescan_dev_for_boot=echo Scanning ${devtype} ${devnum}:${distro_bootpart}...; for prefix in ${boot_prefixes}; do run scan_dev_for_extlinux; run scan_dev_for_scripts; done;run scan_dev_for_efi;scan_dev_for_boot_part=part list ${devtype} ${devnum} -bootable devplist; env exists devplist || setenv devplist 1; for distro_bootpart in ${devplist}; do if fstype ${devtype} ${devnum}:${distro_bootpart} bootfstype; then run scan_dev_for_boot; fi; done; setenv devplistbootcmd_mmc0=devnum=0; run mmc_bootbootcmd_mmc1=devnum=1; run mmc_bootbootcmd_usb0=devnum=0; run usb_bootbootcmd_pxe=run boot_net_usb_start; dhcp; if pxe get; then pxe boot; fibootcmd_dhcp=run boot_net_usb_start; if dhcp ${scriptaddr} ${boot_script_dhcp}; then source ${scriptaddr}; fi;setenv efi_fdtfile ${fdtfile}; setenv efi_old_vci ${bootp_vci};setenv efi_old_arch ${bootp_arch};setenv bootp_vci PXEClient:Arch:00011:UNDI:003000;setenv bootp_arch 0xb;if dhcp ${kernel_addr_r}; then tftpboot ${fdt_addr_r} dtb/${efi_fdtfile};if fdt addr ${fdt_addr_r}; then bootefi ${kernel_addr_r} ${fdt_addr_r}; else bootefi ${kernel_addr_r} ${fdtcontroladdr};fi;fi;setenv bootp_vci ${efi_old_vci};setenv bootp_arch ${efi_old_arch};setenv efi_fdtfile;setenv efi_old_arch;setenv efi_old_vci;distro_bootcmd=for target in ${boot_targets}; do run bootcmd_${target}; done%s: null device
apsync@apsync:/boot/firmware$ 

[peterbarker]

I've checked by trees and reflog here and haven't found anything relevant.

[jmachuca77]

Automated the disabling of the uBoot wait for key, turns out all that has to be done is add the uboot.env file to the /boot/firmware folder, updated the disable console script and squashed the commits.

[rmackay9]

@peterbarker says we should test using this list of steps: https://github.com/ArduPilot/companion/blob/master/Common/Ubuntu/testing.txt

[rmackay9]

The access point name appears to be coming up as "ardupilot" but Peter Barker says it should be "ArduPilot".

[jmachuca77]

Those testing steps are outdated they will fail several steps as the mechanisms to start for example mavlink router where changed to system services instead of screen and rc.local. I'll update the steps and add them to the repo.

[rmackay9]

Whether they are out-of-date or not I tried to go through the testing.txt file found here and had these results:

• Connect to AP -- OK although I think the AP name should be "ArduPilot" instead of "ardupilot"
• screen -list -- FAIL - listedDataFlashLogger but cherrypy and mavlink-router were missing
• sudo screen -list -- OK. apweb was visible
• start mavproxy using "mavproxy.py --master :14550" -- FAIL - mavproxy displayed "link 1 down". Also displayed the message, "WARNING: You should uninstall ModemManager as it conflicts with APM and Pixhawk"
• Connect using a GCS using the broadcast packets on port 14550 -- OK. I was able to connect with MP through the access point
• "ensure you can connect multiple GCSs at the same time!" -- FAIL. Could only connect a single instance of MP. Also was not able to shutdown MP and start QGC and have it connect.
• Video Streaming testing -- FAIL. It wasn't clear at all how to test the camera. Connecting a web browser to 10.0.1.128:8000 didn't work
• web interface testing -- OK. AP Web (http://10.0.1.128/index.html) works except for the System Status / Transmitter page.

[jmachuca77]

• screen -list -- FAIL - listedDataFlashLogger but cherrypy and mavlink-router were missing
  For mavlink router:
  This is expected, mavlink-router is started as a service so to check if its running the standard system service commands apply

sudo systemctl status mavlink-router

For cherrypy, this is not installed as this image is using AP_streamline so look here for details on that:
https://github.com/shortstheory/adaptive-streaming
We might be missing the APWeb part of that since I think I installed only the normal APSync version.

• start mavproxy using "mavproxy.py --master :14550" -- FAIL - mavproxy displayed "link 1 down". Also displayed the message, "WARNING: You should uninstall ModemManager as it conflicts with APM and Pixhawk"

The local endpoint to mavproxy is port 14655, so that command should change on the testing instructions. That is the case on all apsync images since the configuration for mavlink-router is standard for all:

[UdpEndpoint to_mavproxy]
Mode = Normal
Address = 127.0.0.1
Port = 14655

You can use the script ~/start_mavproxy/start_mavproxy.sh to start mavproxy more conveniently.

modem manager does need to be removed, I'll check the scripts to see why its not being removed.

• "ensure you can connect multiple GCSs at the same time!" -- FAIL. Could only connect a single instance of MP. Also was not able to shutdown MP and start QGC and have it connect.

This is expected and is a change in behavior from previous images. Previous images included a special version of mavlink-router that was modified by @peterbarker, the mods he made permitted the use of the portlock feature which in turn allows multiple connections to the same port. However it has been my experience that this causes a very big amount of packet loss and an intermittent connection in certain cases. This can be most noticeable while downloading parameters from the vehicle as it takes an extremely long time to do so. This has been verified in multiple vehicles with multiple companion computers ranging from the RPi (W, 3, 4), Nvidia Jetsons (TX2, Xavier and Nano). That version of mavlink-router has also not been updated since Apr 23, 2018 and is now quite behind master. This image is running master which is the reason for the behavior you see.

When disconnecting one GCS and connecting another one the previous connection needs to timeout before it will allow connection from another GCS. This was not the case on Peter's version of mavlink-router, if a GCS connects to a port that only allows one connection, then that port is locked until a reboot occurs.

The scripts contain two versions of the mavlink-router installation script one to install master and one to install Peters version. The automated scripts install master by using

setup_master_mavlink-router.sh

if the old version is desired then you just have to run the normal script

setup_mavlink-router

If you want to connect multiple GCS to one vehicle then additional ports need to be added to the mavlink-router configuration.

• Video Streaming testing -- FAIL. It wasn't clear at all how to test the camera. Connecting a web browser to 10.0.1.128:8000 didn't work

See above comment about ap_streamline. Additional information on how this works can be found here:
https://github.com/shortstheory/adaptive-streaming

[rmackay9]

I've merged these after making some non-functional changes. The changes I made were pretty minor:

• split one of the commits into two
• prepended the directory name (not really necessary because we don't do a lot of cross branch merges but it's consistent with the commits up until now)
• removed some whitespace errors from some files

Thanks!