Skip to content
This repository has been archived by the owner on Sep 28, 2019. It is now read-only.

Feedback: pre-build requirements #21

Closed
TommyMurphyTM1234 opened this issue Aug 31, 2017 · 35 comments
Closed

Feedback: pre-build requirements #21

TommyMurphyTM1234 opened this issue Aug 31, 2017 · 35 comments

Comments

@TommyMurphyTM1234
Copy link

I ran through the build process on a clean Ubuntu 16.04 64 bit installation.
In case it helps to refine the installation instructions at all I found that the script failed a few times until I installed each of the following in turn:

sudo apt install automake
sudo apt install texinfo
sudo apt install makeinfo

Maybe it's too much trouble to enumerate all required packages for all possible build platforms in which case running the script and installing things on a trial and error basis is probably OK?

Curiously the GCC build script did not complain about these tools being missing and did not install them (otherwise they would still be available now for the openocd script) so I don't know if it uses and installs them in the docker container while the openocd script installs and uses them on the host or something?!

One other thing - at the very end of the build process I got this but I don't know if it's significant?

/Host/Work/openocd/scripts/build-helper.sh: line 743: DEPLOY_FOLDER_NAME: unbound variable

@ilg-ul
Copy link
Contributor

ilg-ul commented Aug 31, 2017

the build process on a clean Ubuntu 16.04 64 bit installation. ... the script failed

that's bad. the script has (had?) some tests at the beginning to check the prerequisites, and should not start if there is something missing.

could you list the executables that failed, to extend the initial tests?

installing things on a trial and error basis is probably OK?

in my opinion it is not ok, the script should complain from the beginning.

DEPLOY_FOLDER_NAME

that's bad too, it means something was wrong in the first part of the script, where variables are set.

@ilg-ul
Copy link
Contributor

ilg-ul commented Aug 31, 2017

while the openocd script installs and uses them on the host

good point. I thought most of the interesting actions happen inside the container, where things should already be present.

could you remove those packages from the ubuntu host, retry the script, and take a closer look where things break? if the actions are ok to happen on the host, then the executables need to be tested at the beginning. if the actions are executed on the host by mistake, the script needs to be fixed.

@TommyMurphyTM1234
Copy link
Author

Yes - I will do this later on and post feedback on it.

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 1, 2017

Apologies for the delay on this but I only got to look at it again now.
I double checked with a clean (VM) install of Ubuntu 16.04 64 bit desktop, nothing additional installed other than available updates.
I found that the script would not run successfully until the following were installed on the host:

  • automake
  • texinfo (for makeinfo)
  • libtool

Perhaps it is a mistake that the script looks for these on the host rather than in the docker container(s)?

Also as mentioned above at the end of the build process I got this again:

Changing owner to non-root Linux user...
/Host/Work/openocd/scripts/build-helper.sh: line 743: DEPLOY_FOLDER_NAME: unbound variable

P.S. one other small thing is that the website instructions refer to the working folder as "work" when, in fact, it is "Work" (e.g. ~/Work).

More detailed info if needed:

I started with a clean (VM) install of Ubuntu 16.04 64 bit desktop mode.
I didn't add anything other than installing all available updates.

I followed these instructions for installing/configuring docker:

https://gnu-mcu-eclipse.github.io/developer/build-binaries-prerequisites/

The RISC-V tools build instructions link to this page and also to the docker installation instructions but the OpenOCD build instructions only link to the latter - the former are simpler to follow so maybe the OpenOCD page could also link to them?

I then followed the instructions here for building OpenOCD.
While the page does mention some required packages it also says that the script should check for and install them if necessary so I didn't manually install any packages at this stage.
I simply ran:

bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

This failed because automake was not installed (note that the RISC-V build script did not fail on this):

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:15:14 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...

So I installed automake:

sudo apt-get install automake

and ran the script again but this time it failed on makeinfo:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:15:47 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Checking host makeinfo...
/home/user/Downloads/openocd-build.git/scripts/build.sh: line 444: makeinfo: command not found

So I installed texinfo which provides makeinfo:

sudo apt-get install texinfo

I ran the script again but this time it failed on libtool:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:16:41 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Checking host makeinfo...
texi2any (GNU texinfo) 6.1
/home/user/Downloads/openocd-build.git/scripts/build.sh: Error: libtool is required

So I installed libtool

sudo apt-get install libtool

and ran the script again - this time it worked.

Hope this helps.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 1, 2017

thank you for your detailed message. I'll take a look and try to fix this, but right now I'm very busy with the future project template.

@TommyMurphyTM1234
Copy link
Author

OK - thanks. Just to clarify is it a mistake that the script is trying to find these tools on the host rather than in the docker container? If so and I can spot the difference between the RISC-V GCC build script and the OpenOCD build script I'll see if I can do anything to address this issue (and the unbound variable issue).

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 1, 2017

Just to clarify is it a mistake that the script is trying to find these tools on the host rather than in the docker container?

I guess the problem is a wrong assumption that the script runs on macOS. in this case the script should check the prerequisites on the host, since there are no macOS Docker containers.

probably a pair of if/fi to test the host will solve the problem.

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 2, 2017

I guess the problem is a wrong assumption that the script runs on macOS. in this case the script should check the prerequisites on the host, since there are no macOS Docker containers.

probably a pair of if/fi to test the host will solve the problem.

Yes - that seems to be the problem alright - I commented out the calls to/checks for automake, makeinfo and libtool and did a clean build and it worked fine on Ubuntu. So I guess that these calls/checks are macOS specific. I thought that comparing these to the RISC-V GCC tools build script would make it obvious how to wrap them in a host test but that script doesn't have such a test and yet works fine. I'm also not clear on how to fix the unbound variable issue. So I will leave it to you to make the necessary changes whenever you get time. Thanks for the feedback on this.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 3, 2017

I updated the build script and the helper, be sure you update the git and the submodule.

I also updated the build page, which refers to the global Build Prerequisites page.

the page defines the packages required on Ubuntu. it would be great to try again with a fresh Ubuntu with those packages.

I also added the missing DEPLOY_FOLDER_NAME.

please remove the openocd build folder and try again.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 3, 2017

one small note: the checkout-stable and checkout-dev are not implemented, you need to manually checkout the desired branch.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 25, 2017

is this issue still actual?

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 26, 2017

Hi Liviu

There still seem to be problems as far as I can see.

I started with a clean Ubuntu 16.04 64 bit VM installation with the latest VirtualBox guest extensions and all available Ubuntu updates installed - nothing else manually installed.

I pulled the latest (I presume?) script as follows:

git clone --recurse-submodules https://github.com/gnu-mcu-eclipse/openocd-build.git ~/Downloads/openocd-build.git

I kicked off a build of the win32 and deb32 targets but only got this far before the script terminated:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32 --deb32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Tue Sep 26 10:21:30 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.2-ce, build cec0b72

Checking host automake...

I installed automake on the Ubuntu host and kicked the build off again but I got this (not sure if this is significant?):

Copying openocd-build.git from Downloads...
fatal: Not a git repository (or any of the parent directories): .git

and then it failed with this:

bootstrap...
./bootstrap: Error: libtool is required

I installed libtool and tried again but again it failed with an error that seems to relate to the docker container (/Host/Work/...) so I'm not really sure how to work around that.

Running OpenOCD configure...
bash: /Host/Work/openocd/openocd.git/configure: No such file or directory

I don't think that I am doing anything wrong but if I am please let me know.
Thanks.

Hope this helps
Regards
Tommy

@TommyMurphyTM1234
Copy link
Author

Looking at the contents of ~/Work/openocd/openocd.git the problem looks to be that there is no configure script because bootstrap has not been run first.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 26, 2017

I'll take a look and fix it during the next few days. (I had a similar problem with the gcc script, the changes are small, I need to identify them and apply to this script too).

I might also change the configuration to use git submodules instead of downloding the helper, at least this keeps the scripts in sync.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 26, 2017

bootstrap has not been run first

bootstrap was somewhere in the script, probably something went wrong before trying to run it.

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 26, 2017

bootstrap was somewhere in the script, probably something went wrong before trying to run it.

build.sh does not call bootstrap for openocd.
I modified ~/Downloads/openocd.git/scripts/build.sh to add this:

if [ "${target_name}" == "win" ]
  then

    cd "${build_folder_path}/openocd"

    bash "${work_folder_path}/${OPENOCD_FOLDER_NAME}/bootstrap"

but then it fails with this:

Running OpenOCD configure...
+ aclocal
aclocal: error: 'configure.ac' is required

I'm not sure why because ~/Work/openocd/openocd.git/configure.ac does exist.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 26, 2017

build.sh does not call bootstrap for openocd

it looks like I messed the script recently, I remember before it was there.

@TommyMurphyTM1234
Copy link
Author

Actually I may be wrong.
The script does seem to call openocd's bootstrap but only if build.sh bootstrap is called.
Am I supposed to build as follows?

build.sh bootstrap 
build.sh --win32

I tried that but still get the error with configure.ac.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 26, 2017

I'm a bit busy now, I'll try to fix it tomorrow afternoon.

@TommyMurphyTM1234
Copy link
Author

OK - thanks.

Is the "if false ..." snippet correct here or could this be relevant to the current problems?

# ----- Process "pull|checkout-dev|checkout-stable" actions. -----

do_repo_action() {

  # $1 = action (pull, checkout-dev, checkout-stable)

  # Update current branch and prepare autotools.
  echo
  if [ "${ACTION}" == "pull" ]
  then
    echo "Running git pull..."
  elif [ "${ACTION}" == "checkout-dev" ]
  then
    echo "Running git checkout gnu-mcu-eclipse-dev & pull..."
  elif [ "${ACTION}" == "checkout-stable" ]
  then
    echo "Running git checkout gnu-mcu-eclipse & pull..."
  fi

  if [ -d "${PROJECT_GIT_FOLDER_PATH}" ]
  then
    echo
    if [ "${USER}" == "ilg" ]
    then
      echo "If asked, enter ${USER} GitHub password for git pull"
    fi

    cd "${PROJECT_GIT_FOLDER_PATH}"

    if [ "${ACTION}" == "checkout-dev" ]
    then
      git checkout gnu-mcu-eclipse-dev
    elif [ "${ACTION}" == "checkout-stable" ]
    then
      git checkout gnu-mcu-eclipse
    fi

    if false
    then

      git pull --recurse-submodules
      git submodule update --init --recursive --remote

      git branch

      do_host_bootstrap

      rm -rf "${BUILD_FOLDER_PATH}/${APP_LC_NAME}"

      echo
      if [ "${ACTION}" == "pull" ]
      then
        echo "Pull completed. Proceed with a regular build."
      else
        echo "Checkout completed. Proceed with a regular build."
      fi

    else

      echo "Not implemented."
      exit 1

    fi

    exit 0
  else
	echo "No git folder."
    exit 1
  fi

}

FWIW I tried build.sh checkout-dev and got this:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh checkout-dev

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Tue Sep 26 11:31:21 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
Copying openocd-build.git from Downloads...

Running git checkout gnu-mcu-eclipse-dev & pull...

fatal: Not a git repository (or any of the parent directories): .git

I don't know if this is a different problem or related to the original build problem(s) above.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 26, 2017

fatal: Not a git repository (or any of the parent directories): .git

this is a tricky issue that I'm not very happy with. the problem is that for convenience the initial *-build.git is cloned in Downloads, but later on its content is needed in the Work folder, which might be on a different file system; so I copy it, but doing so is not that easy, since it includes hidden folders. in the gcc build script I fixed this, we'll do the same here.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

the 'Not a git repository (or any of the parent directories): .git' problem was solved:

ilg-deprecated/openocd-build@8b8e0c1

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

bootstrap

the bootstrap script is always executed right after cloning the openocd git, so it should not be required to do it manually. but if bootstrap fails, you need to run it manually before restarting the script. or you can remove everything and start from scratch.

the bootstrap script requires automake, and this is the reason automake is checked to be present on the host. the Prerequisites page also asks for automake.

I suggest you retry with a clean Ubuntu, install the packages listed in the Prerequisites page, and run the script.

if the script still fails, there must be something else missing, and we need to identify the packages, and add them to the Prerequisites page.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

I managed to install a new Ubuntu 16 VM and you were right, libtool was also required; I updated the Prerequisites page.

in my environment the build script was successful for all 4 platforms.

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 27, 2017

Thanks Liviu.
But I started with a clean Ubuntu 16.04 64 bit VM, did all the prerequisite stuff and then ran

bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

but it failed with this:

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
Copying openocd-build.git from Downloads...
Cloning into '/home/user/Work/openocd/openocd-build.git'...
done.
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Just in case I did

rm -rf ~/Work

and tried again but got the same error.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

this looks like a damaged file. define DEBUG=-x before starting the script and you should be able to see all script lines.

@TommyMurphyTM1234
Copy link
Author

Yes - the contents of the libusb-1.0.20.tar.bz2 file are HTML including

We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, and currently requires
the use of javascript to function.  Please check back later.

so it's a problem with SourceForge.
I'll try again later or tomorrow morning.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

hmmm... SourceForge is less and less reliable :-(

probably I should re-post those files on GitHub and update the script to get them from there.

@TommyMurphyTM1234
Copy link
Author

SourceForge is less and less reliable

Is this a common occurrence?
I can download the zips using a browser (presumably because it supports JavaScript as the message above says is required) but getting them with curl or wget results in a HTML file with that error message.

@TommyMurphyTM1234
Copy link
Author

I can download the zips using a browser (

Oh now I can't and I get this in a browser:

The sourceforge.net website is temporarily in static offline mode.
Only a very limited set of project pages are available until the main website returns to service.

I'll leave it until the morning to try again.

Thanks a lot for the work on this - much appreciated.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

I already published the binaries on GitHub:

https://github.com/gnu-mcu-eclipse/files/tree/master/libs

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 27, 2017

Thanks - I tried to hack the script to take them from there but I can't seem to figure out the magic URL that will download the raw file and I just keep getting HTML e.g.:

Downloading "libusb-1.0.20.tar.bz2"...
+ cd /home/user/Work/openocd/download
+ curl -L https://github.com/gnu-mcu-eclipse/files/blob/master/libs/libusb-1.0.20.tar.bz2 --output libusb-1.0.20.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32729    0 32729    0     0  32565      0 --:--:--  0:00:01 --:--:-- 32598
+ '[' '!' -d /home/user/Work/openocd/libusb-1.0.20 ']'
+ cd /home/user/Work/openocd
+ tar -xjvf /home/user/Work/openocd/download/libusb-1.0.20.tar.bz2
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now
user@ubuntu-16:~$ head -20 libusb-1.0.20.tar.bz2 






<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://assets-cdn.github.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">


@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

I already updated the script to use the 'magic' urls:

ilg-deprecated/openocd-build@5a4b4e9

@TommyMurphyTM1234
Copy link
Author

TommyMurphyTM1234 commented Sep 27, 2017

Thanks a lot.
Works great now - win32 build worked fine for me and deb32 churning away now.
Thanks! 👍

Edit: deb32 build completed successfully too. Feel free to close the issue if you see fit to do so. Thanks again.

@ilg-ul
Copy link
Contributor

ilg-ul commented Sep 27, 2017

you're welcome!

I'll probably update the other build script to use GitHub archives too.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants