From 225463764fb044627b2513017be8334e7c8769c1 Mon Sep 17 00:00:00 2001 From: Cliff Brake Date: Mon, 6 Apr 2020 16:00:51 -0400 Subject: [PATCH 1/2] format docs with prettier --- docs/conf-files.md | 25 ++-- docs/docker.md | 37 ++--- docs/gcc.md | 27 ++-- docs/git-submodules.md | 24 ++-- docs/layer-management.md | 33 +++-- docs/packages.md | 34 ++--- docs/raspberrypi.md | 22 +-- docs/sama5d27.md | 70 +++++----- docs/superproject-management.md | 231 ++++++++++++++++---------------- 9 files changed, 275 insertions(+), 228 deletions(-) diff --git a/docs/conf-files.md b/docs/conf-files.md index 4914bb298..c2c322b6d 100644 --- a/docs/conf-files.md +++ b/docs/conf-files.md @@ -4,21 +4,24 @@ OpenEmbedded is configured through "conf" files. These exist at various [places](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#user-configuration) -in the sources. At the top level of the project are several files that -contain high level, or build specific configuration options. For the Yoe -project, these files include: +in the sources. At the top level of the project are several files that contain +high level, or build specific configuration options. For the Yoe project, these +files include: -- **conf/auto.conf**: This file is automatically generated by the the build system - and should not be modified. +- **conf/auto.conf**: This file is automatically generated by the the build + system and should not be modified. - [**conf/site.conf**](../conf/site.conf): This file includes Yoe defaults and - project specific changes that you want to store in git. - This [file](../conf/site.conf.sample) includes samples of things you might want to add. -- **conf/local.conf**: This file includes changes that are specific to a user or build - machine, and are not checked into git. This [file](../conf/local.conf.sample) includes - samples of things you might want to add. + project specific changes that you want to store in git. This + [file](../conf/site.conf.sample) includes samples of things you might want to + add. +- **conf/local.conf**: This file includes changes that are specific to a user or + build machine, and are not checked into git. This + [file](../conf/local.conf.sample) includes samples of things you might want to + add. The above configuration files are included into the build by the -[bitbake.conf](https://github.com/YoeDistro/openembedded-core/blob/master/meta/conf/bitbake.conf#L744) file in the OpenEmbedded Core layer. +[bitbake.conf](https://github.com/YoeDistro/openembedded-core/blob/master/meta/conf/bitbake.conf#L744) +file in the OpenEmbedded Core layer. For a full list of variables available in the OpenEmbedded build system, see the [Variables Glossary](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-variables-glos). diff --git a/docs/docker.md b/docs/docker.md index c2d1eee73..5a42e8b00 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -2,25 +2,28 @@ [up](README.md) -The Yoe project provides a [docker container](https://hub.docker.com/r/yoedistro/yoe-build/) -that has all the dependencies needed to build Yoe. This container is mostly transparent -as it simply wraps the bitbake command. Every time you run bitbake, the container is started -and all bitbake operations happen in the container. The build files still exist in the host -file system so they can be accessed using any of your host tools. +The Yoe project provides a +[docker container](https://hub.docker.com/r/yoedistro/yoe-build/) that has all +the dependencies needed to build Yoe. This container is mostly transparent as it +simply wraps the bitbake command. Every time you run bitbake, the container is +started and all bitbake operations happen in the container. The build files +still exist in the host file system so they can be accessed using any of your +host tools. -Using this container has the following -benefits: +Using this container has the following benefits: - you don't have to worry about installing host dependencies -- you can build on any Linux host distro, even those that are not compatible with Yoe -- as host distributions are updated, then eventually become incompatible with older - versions of OpenEmbedded. Using a container allows you to build old version of Yoe - over long product life cycles. +- you can build on any Linux host distro, even those that are not compatible + with Yoe +- as host distributions are updated, then eventually become incompatible with + older versions of OpenEmbedded. Using a container allows you to build old + version of Yoe over long product life cycles. -Yoe uses docker by default, but this can be disabled by setting `DOCKER_REPO=none` in your -_\-envsetup.sh_ file. You can also set `DOCKER_REPO` to a custom container if -you need something different. +Yoe uses docker by default, but this can be disabled by setting +`DOCKER_REPO=none` in your _\-envsetup.sh_ file. You can also set +`DOCKER_REPO` to a custom container if you need something different. -Any tool (such as `devtool`) that uses bitbake must also be run in the container. As these -tools are not yet wrapped, you must enter the docker container first, and then run the tool. -This can be done by running `dkr` first, which drops you into a container shell. +Any tool (such as `devtool`) that uses bitbake must also be run in the +container. As these tools are not yet wrapped, you must enter the docker +container first, and then run the tool. This can be done by running `dkr` first, +which drops you into a container shell. diff --git a/docs/gcc.md b/docs/gcc.md index cac8f7296..851267396 100644 --- a/docs/gcc.md +++ b/docs/gcc.md @@ -2,17 +2,20 @@ [up](README.md) -The primary use for OE is to cross-compile software on a fast workstation, but there are -times during development or evaluating technology where it may be useful to compile -software on the target device. There is also some software that is simply too nasty for -most of us to figure out how to cross-compile so one workaround is to build a binary on -the target system, and then package that binary in subsequent image builds. -This _hack_ should only be used as a _last resort_ in production builds. +The primary use for OE is to cross-compile software on a fast workstation, but +there are times during development or evaluating technology where it may be +useful to compile software on the target device. There is also some software +that is simply too nasty for most of us to figure out how to cross-compile so +one workaround is to build a binary on the target system, and then package that +binary in subsequent image builds. This _hack_ should only be used as a _last +resort_ in production builds. -The following packages can be added to the image, or installed at runtime using a package -manager such as opkg. +The following packages can be added to the image, or installed at runtime using +a package manager such as opkg. -- `packagegroup-core-buildessential` - this is most often what you need and is very similiar - to the build-essential package on Debian/Ubuntu. [recipe](https://github.com/YoeDistro/openembedded-core/blob/master/meta/recipes-core/packagegroups/packagegroup-core-buildessential.bb) -- `packagegroup-self-hosted` - a much more extensive list of packages that includes what is - in buildessential, plus a lot more. [recipe](https://github.com/YoeDistro/openembedded-core/blob/master/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb) +- `packagegroup-core-buildessential` - this is most often what you need and is + very similiar to the build-essential package on Debian/Ubuntu. + [recipe](https://github.com/YoeDistro/openembedded-core/blob/master/meta/recipes-core/packagegroups/packagegroup-core-buildessential.bb) +- `packagegroup-self-hosted` - a much more extensive list of packages that + includes what is in buildessential, plus a lot more. + [recipe](https://github.com/YoeDistro/openembedded-core/blob/master/meta/recipes-core/packagegroups/packagegroup-self-hosted.bb) diff --git a/docs/git-submodules.md b/docs/git-submodules.md index e4fce0483..08c41c876 100644 --- a/docs/git-submodules.md +++ b/docs/git-submodules.md @@ -2,19 +2,20 @@ [up](README.md) -Git submodules are used in this project to manage the various OE layers for reasons -described in [this document](superproject-management.md). There are a few issues -that are commonly encountered. +Git submodules are used in this project to manage the various OE layers for +reasons described in [this document](superproject-management.md). There are a +few issues that are commonly encountered. ## Modifying Submodules -Git submodules are not checked out as branch, but simply a git version. The first -thing you should do when working on a submodule is check out the appropriate branch, -documented in the [.gitmodules](../.gitmodules) file. If you forget to do this, you -may get an error when pushing and then if you run `git branch`, you'll discover you -are on the _(no branch)_ branch. To recover from this situation, check out the -branch you want to work on and then run `git reflog`. This command will list recent -commits which you can then cherry-pick to your working branch. +Git submodules are not checked out as branch, but simply a git version. The +first thing you should do when working on a submodule is check out the +appropriate branch, documented in the [.gitmodules](../.gitmodules) file. If you +forget to do this, you may get an error when pushing and then if you run +`git branch`, you'll discover you are on the _(no branch)_ branch. To recover +from this situation, check out the branch you want to work on and then run +`git reflog`. This command will list recent commits which you can then +cherry-pick to your working branch. ## Submodule merge conflicts @@ -24,4 +25,5 @@ If you get a submodule merge conflict, run `git reset \` If the remote repository for a submodule changes, run `git submodule sync` to synchronize the new remote information into your working directory. This problem -is most often noticed when the `git submodule update` command cannot find a version. +is most often noticed when the `git submodule update` command cannot find a +version. diff --git a/docs/layer-management.md b/docs/layer-management.md index e1ad0a5c2..d1bdeac2b 100644 --- a/docs/layer-management.md +++ b/docs/layer-management.md @@ -2,16 +2,33 @@ [up](README.md) -All OE layers that are being used in Yoe project are mirrored in the YoeDistro space on github. These layers are synced every two hours with upstream since the global namespace is kept intact, there are no conflicts. We reserve namespaces for local branches +All OE layers that are being used in Yoe project are mirrored in the YoeDistro +space on github. These layers are synced every two hours with upstream since the +global namespace is kept intact, there are no conflicts. We reserve namespaces +for local branches -* ```yoe/``` - Distro Specific branch, needed when we have to do distro speicific fixes which are not upstreamable, this is rare situation, secondly, we create additional layers e.g. yoe/mut ( mut stands for "Master-Under-Test) branches which are used for regular CI builds. +- `yoe/` - Distro Specific branch, needed when we have to do distro + speicific fixes which are not upstreamable, this is rare situation, secondly, + we create additional layers e.g. yoe/mut ( mut stands for "Master-Under-Test) + branches which are used for regular CI builds. + +- `/` - This namespace is reserved for developers, these are also + short lived branches primarily used for development which are deleted after + pull request is accepted in a public branch -* ```/``` - This namespace is reserved for developers, these are also short lived branches primarily used for development which are deleted after pull request is accepted in a public branch - ## Why do we mirror the layers ? -* This provides needed redundancy and development space for Yoe Distro developers. Since we create pure mirrors for public branches and offer namespaces for development, this makes it a good combination for hosting changes when they are in flight. It also provides a mechanism for running a CI loop on the changes to be submitted upstream for specific layers. +- This provides needed redundancy and development space for Yoe Distro + developers. Since we create pure mirrors for public branches and offer + namespaces for development, this makes it a good combination for hosting + changes when they are in flight. It also provides a mechanism for running a CI + loop on the changes to be submitted upstream for specific layers. + +- If a product is to be released one can easily mirror the yoe mirrors + internally and keep a local copy, or one can use same mirroring policy and + local branching policy to create internal branches. Just keep / namespace + safe. -* If a product is to be released one can easily mirror the yoe mirrors internally and keep a local copy, or one can use same mirroring policy and local branching policy to create internal branches. Just keep / namespace safe. - -* For some reason if upstream layers disappear and we still need to maintain the layers, which can happen over a long life of products, then we offer a redundancy +- For some reason if upstream layers disappear and we still need to maintain the + layers, which can happen over a long life of products, then we offer a + redundancy diff --git a/docs/packages.md b/docs/packages.md index f832d662d..0d6d0f331 100644 --- a/docs/packages.md +++ b/docs/packages.md @@ -2,26 +2,30 @@ [up](README.md) -During an image build, OpenEmbedded builds all software into packages, which are then -combined into a rootfs. One benefit of this is you can install packages later into a -running system, much like you would a desktop distribution like Debian, Ubuntu, Arch, -etc. OE supports the RPM, DEB, and IPK package formats, and the format can be set using the +During an image build, OpenEmbedded builds all software into packages, which are +then combined into a rootfs. One benefit of this is you can install packages +later into a running system, much like you would a desktop distribution like +Debian, Ubuntu, Arch, etc. OE supports the RPM, DEB, and IPK package formats, +and the format can be set using the [PACKAGE_CLASSES](https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#var-PACKAGE_CLASSES) variable. ## Installing packages from a local build -You can install packages directly from your local OE build onto a target system with the -following procedure: +You can install packages directly from your local OE build onto a target system +with the following procedure: - packaging information must be included in the target image: - `IMAGE_FEATURES += "packages-management"` - a ssh server must be enabled in the target image: - `IMAGE_FEATURES += "ssh-server-dropbear"` -- see [yoe-simple-image](https://github.com/YoeDistro/meta-yoe/blob/master/recipes-core/images/yoe-simple-image.bb) for an example. +- see + [yoe-simple-image](https://github.com/YoeDistro/meta-yoe/blob/master/recipes-core/images/yoe-simple-image.bb) + for an example. - target system must be on the same network as your build computer. -- run `yoe_setup_feed_server `. This configures the target to fetch packages - from your build computer. Alternatively you can set the MACHINE_IP environment variable. +- run `yoe_setup_feed_server `. This configures the target to + fetch packages from your build computer. Alternatively you can set the + MACHINE_IP environment variable. - run `yoe_feed_server`. This starts a web server that serves packages. - on target system, run: - `opkg update` @@ -36,13 +40,13 @@ If the package is not built yet, simply: - `opkg update` - `opkg install ` -This technique is very useful for testing new packages, or for installing packages that -are only needed for development that you don't want to include in production images -(strace, gdb, testing tools, etc). Dependencies are automatically handled by the package -manager. +This technique is very useful for testing new packages, or for installing +packages that are only needed for development that you don't want to include in +production images (strace, gdb, testing tools, etc). Dependencies are +automatically handled by the package manager. -For larger development teams or field deployments, you might want to set up a feed server -to serve packages to any connected device. +For larger development teams or field deployments, you might want to set up a +feed server to serve packages to any connected device. See also: diff --git a/docs/raspberrypi.md b/docs/raspberrypi.md index 0c088568f..b895deacb 100644 --- a/docs/raspberrypi.md +++ b/docs/raspberrypi.md @@ -18,8 +18,8 @@ 1. `sudo eject /dev/sdX` 1. Install SD card in a Raspberry PI and enjoy your new image -Other Raspberry Pi variants can be built by sourcing the appropriate -envsetup file. +Other Raspberry Pi variants can be built by sourcing the appropriate envsetup +file. ## Enable serial console @@ -30,9 +30,10 @@ in the BSP layer ## Connecting to rPI serial console -The Raspberry PI serial console is avaiable on the expansion header. A USB->serial -cable with flying leads is a convenient way to connect to this. [FTDI](https://www.ftdichip.com/Products/Cables/RPi.htm) (as well as many -other companies supply these cables). The below image shows how the FTDI cable is +The Raspberry PI serial console is avaiable on the expansion header. A +USB->serial cable with flying leads is a convenient way to connect to this. +[FTDI](https://www.ftdichip.com/Products/Cables/RPi.htm) (as well as many other +companies supply these cables). The below image shows how the FTDI cable is connected: ![rPI serial console](raspberry-pi-serial-console.jpg) @@ -43,10 +44,13 @@ The relevant signals are: - FTDI Yellow (RXD) <- rPI Pin8 (TXD) - FTDI Orange (TXD) -> rPI Pin10 (RXD) -See the [schematics](https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/README.md) for more information. +See the +[schematics](https://www.raspberrypi.org/documentation/hardware/raspberrypi/schematics/README.md) +for more information. ## rPI Power Supply -Some of the Raspberry PI products seem to be sensitive to power quality. It is recommended -to use a supply that outputs 5.1V such as the [official supply](https://www.raspberrypi.org/products/raspberry-pi-universal-power-supply/). An inadequate supply may result in lockups or -SD card file system corruption. +Some of the Raspberry PI products seem to be sensitive to power quality. It is +recommended to use a supply that outputs 5.1V such as the +[official supply](https://www.raspberrypi.org/products/raspberry-pi-universal-power-supply/). +An inadequate supply may result in lockups or SD card file system corruption. diff --git a/docs/sama5d27.md b/docs/sama5d27.md index 444cf4774..879a10896 100644 --- a/docs/sama5d27.md +++ b/docs/sama5d27.md @@ -1,12 +1,14 @@ # ATSAMA5D27 -The ATSAMA5D27 from Microchip is a ARM Cortex-A5 CPU available in 3 different packages: +The ATSAMA5D27 from Microchip is a ARM Cortex-A5 CPU available in 3 different +packages: - [IC](https://www.microchip.com/wwwproducts/en/ATSAMA5D27) - [SIP package with embedded SDRAM](https://www.microchip.com/wwwproducts/en/ATSAMA5D27C-D1G) - [SOM package](https://www.microchip.com/wwwproducts/en/ATSAMA5D27-SOM1) -Microchip also provides a [development board](https://www.microchip.com/DevelopmentTools/ProductDetails/ATSAMA5D27-SOM1-EK1). +Microchip also provides a +[development board](https://www.microchip.com/DevelopmentTools/ProductDetails/ATSAMA5D27-SOM1-EK1). ## Building/installing an image @@ -19,60 +21,68 @@ Microchip also provides a [development board](https://www.microchip.com/Developm 1. `lsblk` (note sd card device, and substitute for /dev/sdX below) 1. `yoe_install_image /dev/sdX yoe-simple-image` 1. `sudo eject /dev/sdX` -1. Install SD card in development board and apply power via one of the micro-USB connectors. +1. Install SD card in development board and apply power via one of the micro-USB + connectors. ## Connecting a serial console -The serial console for the development board is provided via the micro USB connector near -the Ethernet connector. The serial device will show up as /dev/ttyACMx on a Linux system. +The serial console for the development board is provided via the micro USB +connector near the Ethernet connector. The serial device will show up as +/dev/ttyACMx on a Linux system. ![Serial Console connector](ATSAMA5D27-SOM1-EK1.png) ## USB Host -USB Host is provided by connecting a port expander to the USB Type-C connector, as shown -in the photo above. +USB Host is provided by connecting a port expander to the USB Type-C connector, +as shown in the photo above. ## Bootstrapping the system using SAM-BA -[SAM-BA](https://github.com/atmelcorp/sam-ba) is a boot assistant that can be used -to bootstrap a system over a UART or USB Client port and program fuses. It is possible -to bootstrap a system completely over SAM-BA with no flash/eMMC/SD memory -programmed. This is useful for systems that perhaps have a eMMC flash installed, but do -not have a SD card slot. To bootstrap such a system, we can: +[SAM-BA](https://github.com/atmelcorp/sam-ba) is a boot assistant that can be +used to bootstrap a system over a UART or USB Client port and program fuses. It +is possible to bootstrap a system completely over SAM-BA with no flash/eMMC/SD +memory programmed. This is useful for systems that perhaps have a eMMC flash +installed, but do not have a SD card slot. To bootstrap such a system, we can: - Load both the AT91Bootstrap and u-boot over a UART or USBClient port, and then boot into u-boot using SAM-BA. -- From u-boot, we can load a kernel from a USB flash device, or from a TFTP server - over the network. -- The kernel can mount a root file system on a USB disk, or use the Yoe updater to - initialize and program a SD/eMMC device. +- From u-boot, we can load a kernel from a USB flash device, or from a TFTP + server over the network. +- The kernel can mount a root file system on a USB disk, or use the Yoe updater + to initialize and program a SD/eMMC device. This process has been automated and described in more detail below: 1. build the target image first as described above -1. `yoe_sam_build_bootstrap_tools` (this builds SAM-BA and a special version of the - at91bootstrap that can be used to load the bootloaders. +1. `yoe_sam_build_bootstrap_tools` (this builds SAM-BA and a special version of + the at91bootstrap that can be used to load the bootloaders. 1. connect the development board USB (J10) to your workstation -1. make sure your linux user has access to serial ports (add to serial, uucp, or whatever group is used for serial ports) +1. make sure your linux user has access to serial ports (add to serial, uucp, or + whatever group is used for serial ports) 1. make sure screen is installed (used for a serial terminal program) -1. power on the dev board and verify it is outputting ROMboot on the serial terminal: `screen /dev/ttyACM0 115200` -1. stop the serial terminal program (SAM-BA now needs to use the serial port): `Ctrl-A k` -1. `yoe_sam_load_uboot_via_samba` (this loads at91bootstrap and u-boot over a UART. Note, - you must stop u-boot from launching a kernel, because there is no kernel to launch.) +1. power on the dev board and verify it is outputting ROMboot on the serial + terminal: `screen /dev/ttyACM0 115200` +1. stop the serial terminal program (SAM-BA now needs to use the serial port): + `Ctrl-A k` +1. `yoe_sam_load_uboot_via_samba` (this loads at91bootstrap and u-boot over a + UART. Note, you must stop u-boot from launching a kernel, because there is no + kernel to launch.) 1. you should now be sitting at a u-boot prompt 1. insert a USB flash disk into your workstation -1. `yoe_sam_install_bootstrap_files ` (this script copies the kernel - and update file to a USB disk for installation) +1. `yoe_sam_install_bootstrap_files ` (this script copies + the kernel and update file to a USB disk for installation) 1. insert USB flash disk into the development board host USB port 1. insert a blank SD card into the full size SD slot -1. from u-boot, load a kernel from USB: `usb reset;fatls usb 0:1;fatload usb 0:1 0x21000000 at91-sama5d27_som1_ek.dtb;fatload usb 0:1 0x22000000 zImage;bootz 0x22000000 - 0x21000000` +1. from u-boot, load a kernel from USB: + `usb reset;fatls usb 0:1;fatload usb 0:1 0x21000000 at91-sama5d27_som1_ek.dtb;fatload usb 0:1 0x22000000 zImage;bootz 0x22000000 - 0x21000000` -At this point the kernel will boot, run the updater from an initramfs bundled with the kernel. -The updater will inialize a SD/eMMC device, and program the system from the \*.upd file on the -USB disk. +At this point the kernel will boot, run the updater from an initramfs bundled +with the kernel. The updater will inialize a SD/eMMC device, and program the +system from the \*.upd file on the USB disk. -See the [Yoe Updater Documentation](updater.md) for more information on how the updater works. +See the [Yoe Updater Documentation](updater.md) for more information on how the +updater works. You can also load a kernel from a TFTP server: diff --git a/docs/superproject-management.md b/docs/superproject-management.md index fb838f92b..73d4583df 100644 --- a/docs/superproject-management.md +++ b/docs/superproject-management.md @@ -2,169 +2,170 @@ [up](README.md) -One controversial choice in Yoe is the use of -Git Submodules vs Repo. Because we need to manage multiple -Git repositories, something is needed. There are a number of -solutions including: +One controversial choice in Yoe is the use of Git Submodules vs Repo. Because we +need to manage multiple Git repositories, something is needed. There are a +number of solutions including: -* Git submodules (what we use now) -* Google repo -* Git subtree -* custom scripts -* Poky style combo repo +- Git submodules (what we use now) +- Google repo +- Git subtree +- custom scripts +- Poky style combo repo In this document, we will use the OE term "layers" to refer to git -subrepositories (submodules). Superproject is the term that describes -the project that manages all the subprojects (layers). +subrepositories (submodules). Superproject is the term that describes the +project that manages all the subprojects (layers). ## Requirements -Yoe is designed foremost for product development, or using Yocto/OE to -build software that will be deployed in real products (vs a technology demo -build). It is also designed to be used by people who may -not be embedded Linux/Git experts, but yet want to create builds and make -incremental improvements over long product lifecycles (can be 5 years or -more). From this, we derive Yoe's requirements (somewhat ordered by priority). +Yoe is designed foremost for product development, or using Yocto/OE to build +software that will be deployed in real products (vs a technology demo build). It +is also designed to be used by people who may not be embedded Linux/Git experts, +but yet want to create builds and make incremental improvements over long +product lifecycles (can be 5 years or more). From this, we derive Yoe's +requirements (somewhat ordered by priority). 1. know exactly what version of all software (layers) were used in a particular build, and be able to rebuild any version on any machine. 1. all build and configuration files (local.conf, etc) must be stored in git. -1. project structure is obvious and easy to understand. Obscure steps such - as copying on soft-linking setup scripts that may be modified is minimized. +1. project structure is obvious and easy to understand. Obscure steps such as + copying on soft-linking setup scripts that may be modified is minimized. 1. hard to make mistakes such as changing a layer version or config file, and then forgetting to record this change. 1. easy to add/remove layers to the build. 1. easy to make changes to layers and submit changes back upstream. -1. easy to fork layers if tweaks need to be made that are not -available upstream. +1. easy to fork layers if tweaks need to be made that are not available + upstream. -Every solution has tradeoffs. To accomplish one goal, you -often have to accept compromises on another goal, so it is likely no -one tools is best to accomplish all of these goals. +Every solution has tradeoffs. To accomplish one goal, you often have to accept +compromises on another goal, so it is likely no one tools is best to accomplish +all of these goals. ## Git Submodules -* pros - * super/sub project organization is obvious -- top level git repo is the +- pros + - super/sub project organization is obvious -- top level git repo is the superproject, and subprojects live under it in the directory tree - * git status gives you status of submodules and files in one command - * easy to store layer versions and conf file changes in the superproject - * difficult to forget to record layer versions (git status shows you) - * well documented in git documentation - * .gitmodules file is simple and easy to read - * easy to track development of layers by specifying branch in .gitmodules, and + - git status gives you status of submodules and files in one command + - easy to store layer versions and conf file changes in the superproject + - difficult to forget to record layer versions (git status shows you) + - well documented in git documentation + - .gitmodules file is simple and easy to read + - easy to track development of layers by specifying branch in .gitmodules, and updating to latest on branch with "git submodule update --remote" - * a number of application developers are already using git submodules in various - projects, where repo is less common except in large build systems (like Android) - * easy to do recursive changelogs into submodules (again, useful for production) - * optimized for development when subprojects are not changing a lot -* cons - * no-branch issue when making changes to a submodule: + - a number of application developers are already using git submodules in + various projects, where repo is less common except in large build systems + (like Android) + - easy to do recursive changelogs into submodules (again, useful for + production) + - optimized for development when subprojects are not changing a lot +- cons + - no-branch issue when making changes to a submodule: http://bec-systems.com/site/696/git-submodules-what-to-do-when-you-commit-to-no-branch - * removing submodules can be a little confusing - * updating a submodule URL requires a "sync" to propagate changes to working + - removing submodules can be a little confusing + - updating a submodule URL requires a "sync" to propagate changes to working submodule. ## Google Repo -* pros - * repo manifest is flexible/powerful - * repo manifest can include other manifests - * easy to switch/sync git urls - * a lot of users due to use in Android build system - * good at managing large numbers of subprojects where you own all the subprojects - (example, repo can apply a tag to all the subprojects). However, with Yocto, you are typically - using 3rd party layers where you don't own the repository and can't push tags to these - repositories. - * optmized for development when subprojects are changing a lot -* cons - * To lock down layers, you have to specify the git hash in the manifest: - https://github.com/Angstrom-distribution/angstrom-manifest/blob/angstrom-v2018.06-sumo/default.xml#L22. This requires additional tooling to be - practical for product development. - * Repo is a separate tool from git that needs to be installed and learned - * official documentation is sparse - * manifest file is more complex and harder to read at a glance than +- pros + - repo manifest is flexible/powerful + - repo manifest can include other manifests + - easy to switch/sync git urls + - a lot of users due to use in Android build system + - good at managing large numbers of subprojects where you own all the + subprojects (example, repo can apply a tag to all the subprojects). However, + with Yocto, you are typically using 3rd party layers where you don't own the + repository and can't push tags to these repositories. + - optmized for development when subprojects are changing a lot +- cons + - To lock down layers, you have to specify the git hash in the manifest: + https://github.com/Angstrom-distribution/angstrom-manifest/blob/angstrom-v2018.06-sumo/default.xml#L22. + This requires additional tooling to be practical for product development. + - Repo is a separate tool from git that needs to be installed and learned + - official documentation is sparse + - manifest file is more complex and harder to read at a glance than .gitmodules - * location of manifest file is somewhat hidden and less obvious how to manage - * difficult to store both subproject info and other files in the - superproject. + - location of manifest file is somewhat hidden and less obvious how to manage + - difficult to store both subproject info and other files in the superproject. ## Git Subtree -At this point git subtree is not activily being considered as it mixes -all the subprojects into one repo, and likely not optimal for managing layers, -most of which will rarely change during a product lifecycle. It seems to be -a tool for more advanced users. +At this point git subtree is not activily being considered as it mixes all the +subprojects into one repo, and likely not optimal for managing layers, most of +which will rarely change during a product lifecycle. It seems to be a tool for +more advanced users. ## Poky style combo repo One popular trend these days are [monorepos](http://danluu.com/monorepo/) which are popular in various large companies (like Google and Facebook). [Poky](http://git.yoctoproject.org/cgit/cgit.cgi/poky) is somewhat of a monorepo -in that it contains bitbake, oe-core, and some other things in one git -repo. This makes it easy to checkout one repository and build a demo image if -you want to test Yocto. However, once you want to add other layers, you still -have to deal with multiple repositories or figure out the +in that it contains bitbake, oe-core, and some other things in one git repo. +This makes it easy to checkout one repository and build a demo image if you want +to test Yocto. However, once you want to add other layers, you still have to +deal with multiple repositories or figure out the [combo-layer](http://git.yoctoproject.org/cgit/cgit.cgi/poky/tree/scripts/combo-layer) -tool. Additionally, if you make changes to one of the pieces of Poky, -extracting these changes to submit back upstream is not as easy as if the -repositories were simply split out in their original form. Also, it is less obvious -you might update just one port of the combo repo, but not others. +tool. Additionally, if you make changes to one of the pieces of Poky, extracting +these changes to submit back upstream is not as easy as if the repositories were +simply split out in their original form. Also, it is less obvious you might +update just one port of the combo repo, but not others. It seems monorepos are best for projects where a company owns everything in the repo. With a Yocto build, most of the layers come from third parties. In this scenario, it seems a superproject type setup is a better fit. We favor the -transparency of showing clearly where each layer came from so it is obvious where -to go for support, etc. +transparency of showing clearly where each layer came from so it is obvious +where to go for support, etc. ## Summary As mentioned before, each solution has trade-offs. Superproject management is messy at best, because there are conflicting goals. Changing layer urls and -tracking floating layers is easier in Repo, but locking down layer versions -is easier in git submodules. With any solution, some amount of tooling will -likely be required to smooth out, or at least document, the pain points. - -For the typical Yoe user who -is building a product, by far the most common use of Yoe is simply building -iterative releases for a product where very little changes from release to -release except application code. These product cycles can be 5 years or more -with 100's of releases. During this time, the layers may change from time to -time to add new packages, tweak the system files, or update to a new version -of Yocto, but these layer changes are relatively few compared to the application -changes. We want to optimize the experience for this use case where a developer -using Yoe is not a Yocto expert, but knows enough to tweak recipes, run builds, etc. - -The ability to manage both configuration files, documentation, -scripts/tools, and subprojects -in a single superproject with git submodules seems like a much simpler model -than repo, where the only thing you can really manage in a repo manifest is the -subproject. This is a simpler setup because everything the user typically needs to -interact with is near the top of the directory tree and easy to find. Things that -change from one iterative build to the next are located in the top level superproject, -so all you need to tag a build is a simple git add/tag/commit/push of the superproject. -Convenience tooling and conf file changes can easily be made, and it is obvious where they are stored. -With repo, this is often a layer of indirection where local.conf and other files -are copied (or generated) to a temporary build directory and it is not obvious how -this happens, or how to capture changes in git. Additionally, if you want to lock down -layers to a git version, typically tooling is written to add these git hashes to the -manifest file for a release build, and then revert this back to floating branches for -development. This tooling is unnecessary with git submodules. +tracking floating layers is easier in Repo, but locking down layer versions is +easier in git submodules. With any solution, some amount of tooling will likely +be required to smooth out, or at least document, the pain points. + +For the typical Yoe user who is building a product, by far the most common use +of Yoe is simply building iterative releases for a product where very little +changes from release to release except application code. These product cycles +can be 5 years or more with 100's of releases. During this time, the layers may +change from time to time to add new packages, tweak the system files, or update +to a new version of Yocto, but these layer changes are relatively few compared +to the application changes. We want to optimize the experience for this use case +where a developer using Yoe is not a Yocto expert, but knows enough to tweak +recipes, run builds, etc. + +The ability to manage both configuration files, documentation, scripts/tools, +and subprojects in a single superproject with git submodules seems like a much +simpler model than repo, where the only thing you can really manage in a repo +manifest is the subproject. This is a simpler setup because everything the user +typically needs to interact with is near the top of the directory tree and easy +to find. Things that change from one iterative build to the next are located in +the top level superproject, so all you need to tag a build is a simple git +add/tag/commit/push of the superproject. Convenience tooling and conf file +changes can easily be made, and it is obvious where they are stored. With repo, +this is often a layer of indirection where local.conf and other files are copied +(or generated) to a temporary build directory and it is not obvious how this +happens, or how to capture changes in git. Additionally, if you want to lock +down layers to a git version, typically tooling is written to add these git +hashes to the manifest file for a release build, and then revert this back to +floating branches for development. This tooling is unnecessary with git +submodules. There is a cost to complexity. We may be temped to reach for the most powerful tool at our disposal, but there is a cost if a simpler tool would work. With -tooling, there is no one-size-fits all. We don't pick up a sledge hammer to drive a tack, -nor do we use a tack hammer to bust up concrete, nor do we buy a robot if we only need -to drive a few nails. The best solutions are the simplest one that does the job well. -"Clear" and "obvious" are the best solutions in a build system. "Neat" and "clever" usually -are not. +tooling, there is no one-size-fits all. We don't pick up a sledge hammer to +drive a tack, nor do we use a tack hammer to bust up concrete, nor do we buy a +robot if we only need to drive a few nails. The best solutions are the simplest +one that does the job well. "Clear" and "obvious" are the best solutions in a +build system. "Neat" and "clever" usually are not. (Note, I don't claim Yoe meets this lofty goal, but this is the vision.) -It seems git submodules are still a better fit for the use case Yoe was designed for, -however this is not set in stone, but simply where we are at this point. It is acknowledged -that repo may make some things easier for OE/Layer core developers where -who are often modifying and updating layers. So, if repo works great for you, by all means keep using -it. That is the beauty of OE layers -- you can put then -together using whatever tools works best for you. Yoe is just an attempt at keeping things as -simple as possible for product developers. +It seems git submodules are still a better fit for the use case Yoe was designed +for, however this is not set in stone, but simply where we are at this point. It +is acknowledged that repo may make some things easier for OE/Layer core +developers where who are often modifying and updating layers. So, if repo works +great for you, by all means keep using it. That is the beauty of OE layers -- +you can put then together using whatever tools works best for you. Yoe is just +an attempt at keeping things as simple as possible for product developers. From b82c82f4a890ac5d55653455185c3043a7fbc2d4 Mon Sep 17 00:00:00 2001 From: Cliff Brake Date: Mon, 6 Apr 2020 16:06:49 -0400 Subject: [PATCH 2/2] add links to git submodule articles --- docs/git-submodules.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/docs/git-submodules.md b/docs/git-submodules.md index 08c41c876..c29789db0 100644 --- a/docs/git-submodules.md +++ b/docs/git-submodules.md @@ -3,8 +3,11 @@ [up](README.md) Git submodules are used in this project to manage the various OE layers for -reasons described in [this document](superproject-management.md). There are a -few issues that are commonly encountered. +reasons described in [this document](superproject-management.md). See this +[article](https://www.atlassian.com/git/tutorials/git-submodule) for an +introduction to Git submodules. + +There are a few issues that are commonly encountered decsribed below. ## Modifying Submodules @@ -15,7 +18,9 @@ forget to do this, you may get an error when pushing and then if you run `git branch`, you'll discover you are on the _(no branch)_ branch. To recover from this situation, check out the branch you want to work on and then run `git reflog`. This command will list recent commits which you can then -cherry-pick to your working branch. +cherry-pick to your working branch. See this +[article](http://bec-systems.com/site/696/git-submodules-what-to-do-when-you-commit-to-no-branch) +for more information. ## Submodule merge conflicts