Can't build swmm5 with conda-build #114

lrntct · 2017-12-06T22:36:22Z

I've followed the instruction in #74 to build the software and got the following error:

$ ~/miniconda3/bin/conda-build conda.recipe/

Cloning into '/home/laurent/miniconda3/conda-bld/libswmm_1512599487959/work'...
done.
checkout: 'HEAD'
Your branch is up-to-date with 'origin/_conda_cache_origin_head'.
==> git log -n1 <==

commit 56fd8b20cdbeaeb7b8aaf82a493ca3e487947cd8
Merge: 79d6c1b 2b30d13
Author: Bryant E. McDonnell <[email protected]>
Date:   Fri Nov 17 10:46:04 2017 -0500

    Merge pull request #105 from bemcdonnell/include
    
    Include Directory

==> git describe --tags --dirty <==

v5.2.0.dev0-122-g56fd8b2

==> git status <==

On branch _conda_cache_origin_head
Your branch is up-to-date with 'origin/_conda_cache_origin_head'.

nothing to commit, working tree clean

Attempting to finalize metadata for libswmm
INFO:conda_build.metadata:Attempting to finalize metadata for libswmm
BUILD START: ['libswmm-5.2.0.dev0-122.tar.bz2']


The following NEW packages will be INSTALLED:

    bzip2:           1.0.6-h6d464ef_2     
    ca-certificates: 2017.08.26-h1d4fec5_0
    cloog:           0.18.0-0             
    cmake:           3.9.4-h142f0e9_0     
    curl:            7.55.1-h78862de_4    
    expat:           2.2.5-he0dffb1_0     
    gcc:             4.8.5-7              
    gmp:             6.1.2-h6c8ec71_1     
    isl:             0.12.2-0             
    libgcc-ng:       7.2.0-h7cc24e2_2     
    libssh2:         1.8.0-h2d05a93_3     
    libstdcxx-ng:    7.2.0-h7a57d05_2     
    libuv:           1.14.0-h56b52c2_0    
    mpc:             1.0.3-hec55b23_5     
    mpfr:            3.1.5-h11a74b3_2     
    ncurses:         6.0-h9df7e31_2       
    openssl:         1.0.2m-h26d622b_1    
    rhash:           1.3.5-hbf7ad62_1     
    xz:              5.2.3-h55aa19d_2     
    zlib:            1.2.11-ha838bed_2    

source tree in: /home/laurent/miniconda3/conda-bld/libswmm_1512599487959/work
cmake: /home/laurent/miniconda3/conda-bld/libswmm_1512599487959/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.20' not found (required by cmake)
cmake: /home/laurent/miniconda3/conda-bld/libswmm_1512599487959/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/bin/../lib/libstdc++.so.6: version `CXXABI_1.3.9' not found (required by cmake)
cmake: /home/laurent/miniconda3/conda-bld/libswmm_1512599487959/_h_env_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_placehold_p/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by cmake)
Traceback (most recent call last):
  File "/home/laurent/miniconda3/bin/conda-build", line 11, in <module>
    sys.exit(main())
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 388, in main
    execute(sys.argv[1:])
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/cli/main_build.py", line 379, in execute
    verify=args.verify)
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/api.py", line 187, in build
    need_source_download=need_source_download, config=config, variants=variants)
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1833, in build_tree
    notest=notest,
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/build.py", line 1148, in build
    utils.check_call_env(cmd, env=env, cwd=src_dir)
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 676, in check_call_env
    return _func_defaulting_env_to_os_environ(subprocess.check_call, *popenargs, **kwargs)
  File "/home/laurent/miniconda3/lib/python3.6/site-packages/conda_build/utils.py", line 672, in _func_defaulting_env_to_os_environ
    return func(_args, **kwargs)
  File "/home/laurent/miniconda3/lib/python3.6/subprocess.py", line 291, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/bin/bash', '-e', '/home/laurent/miniconda3/conda-bld/libswmm_1512599487959/work/conda_build.sh']' returned non-zero exit status 1.

However, if I do:

$ cmake CMakeLists.txt
$ make

The build works properly.
OS: Ubuntu 17.10
swmm version: 56fd8b2

Any idea?

The text was updated successfully, but these errors were encountered:

bemcdonnell · 2017-12-06T22:49:38Z

@lrntct, something was updated when the latest Anaconda came out with removed some dependency libraries required by cmake. This is why both Travis and Circle CI are failing.

have a look at pr #107. Since at this point we’re not so concerned about deploying our builds off, I invite you to simplify the build inside CI to a more typical approach for a C project. You can update the travis CI and extend coverage from OSX and LINUX. I have updated CMAKE in my pr. I like the idea is doing this integrations with the native compilers instead of using conda dependencies since it’s nirmally faster and cleaner.

lrntct · 2017-12-06T23:24:03Z

@bemcdonnell OK, I'll try to update the Travis build by using CMake instead of conda-build.
On a side note, what is the reason of having both TravisCI and CircleCI? It seems to me that Travis can do the same as Circle, even a bit more (OSX). Removing one of them could lower the hassle of having to update 3 CI configurations when changing something.
I'll try to create a multi-OS .travis.yml and see how it goes.

bemcdonnell · 2017-12-07T00:01:58Z

@lrntct, we were spreading different platform builds across CI servers for efficiency. I really like what @goanpeca has put together on Circle. Soon we are going to fully enable the code formatter enforcing (waiting till we finish the regression testing to do this) which are all part of the conda framework. Moreover, Circle CI will also be very helpful to deploy builds to Conda (to support the pyswmm project).

goanpeca · 2017-12-07T00:10:44Z

@lrntct the thing is that using Travis for building OSX stuff takes forever unfortunately, and for sing linux, circle ci is much faster, that is the reason of using circle ci. I am in charge of all the CI infrastructure and I will try to document it more so that it is clear why it is done the way it is.

I know that there is a current plan by @bemcdonnell to switch back to VS instead of conda to reduce build times. This has some pros and cons, but until I have some more time I think it is fine to do it. To be able to exploit the conda exosytem to the full extent, which is what we should still aim for, we need to make a release con conda-forge of both SWMM and pyswmm.

lrntct · 2017-12-07T15:42:18Z

@goanpeca If the ultimate goal is to build the conda packages on conda-forge, then in there current form the CI infrastructure is more to test the conda packaging I guess?

@bemcdonnell @goanpeca Should I work on updating the Travis recipe or not?

goanpeca · 2017-12-07T16:07:30Z

@goanpeca If the ultimate goal is to build the conda packages on conda-forge, then in there current form the CI infrastructure is more to test the conda packaging I guess?

The idea of using conda and conda-build is to have a ready to use Develop version every time a Pull Request is merged during Dev Process, which gets uploaded to https://anaconda.org/owa

So that users can then do

conda install libswmm -c owa

If the ultimate goal is to build the conda packages on conda-forge, then in there current form the CI infrastructure is more to test the conda packaging I guess?

Not really Conda forge will be used to build the official conda packages that will be available to the general public via

conda install libswmm -c conda-forge

Should I work on updating the Travis recipe or not?

I say no, but I don't what is @bemcdonnell 's plan.

Now regarding the error you should probably update to the latest version of conda and cond build and follow ALL The recipe steps:

$ conda update -n root conda -q --no-pin
$ conda install -n root conda-build anaconda-client -q --no-pin
$ conda-build conda.recipe
$ conda create -n swmm libswmm --use-local
$ source activate swmm
$ run-swmm --help

and you needed to not set LD_LIBRARY_PATH.

@bemcdonnell we could remove all the linter CLang format checks from Travis and appveyor, but I do not agree about removing the actual build process with conda.

bemcdonnell · 2017-12-07T19:48:50Z

@goanpeca,

I’ve been thinking about this project independent from pyswmm. Since this project is fundamentally a C Project, I think distilling our CI servers down to clean an simple processes (with emphasis on efficiency) is where we want to go. The regression testing workflow is complete and it’s time that we focus on getting these integrations outfitted with the tests.

The release channel that I most care about for this project is out GitHub page. If it will bring value to the community to have our releases for OWA SWMM living on conda forge, we should still be able to get these integrations working without using the conda framework. I care so much about speed because we are going to want answers fast.

I’m lightly expanding the scope of this thread to encompass pyswmm but just for 1 second. For pyswmm distribution, I want the channels to be in the PyPI and conda forge. As soon as OWA-swmm is outfitted with SWIG support, we can use the python CI servers to fetch the swmm code from this repo, and build the python extension then package it for us for all platforms. That’s where I see conda fitting into the workflow best. Pyswmm is a completely separate project but it is a sibling to this project.

For this project (OWA-swmm), We can do OSX and Linux builds on travis (just as I’m proposing in my new PR with appveyor updates). And we can use CircleCI to do an additional build on either Linux or OSX.. but mainly, code format enforcer checks.

That’s my vision anyways. Can others share theirs?

lrntct · 2017-12-07T20:27:41Z

I’m lightly expanding the scope of this thread to encompass pyswmm but just for 1 second. For pyswmm distribution, I want the channels to be in the PyPI and conda forge. As soon as OWA-swmm is outfitted with SWIG support, we can use the python CI servers to fetch the swmm code from this repo, and build the python extension then package it for us for all platforms. That’s where I see conda fitting into the workflow best. Pyswmm is a completely separate project but it is a sibling to this project.

About the packaging and interaction with pyswmm, I already gave my opinion in a pyswmm issue. I think that when the SWIG interface will be ready, the swmm repo could accommodate a conda and pypi recipes to build and package the code. It will not be a major part of the repo nor require a big knowledge of python. Then, pyswmm could depends on the swmm package.
I think that this way, a stronger separation between the two projects could be achieved. No need for pyswmm to fetch source files from swmm and manage the compilation. Just an import swmm.
I understand the will to have minimal Python code in the swmm repository and minimal C code in the pyswmm repo. I don't think that managing the compilation of swmm from the pyswmm repo is the right way to achieve this separation.

michaeltryby · 2017-12-07T21:23:33Z

Good discussion guys. Just to weigh in here with my two cents. For the record I want to state that I am not opposed to conda packaging. The question I have is, Is it really necessary for Travis, Circle, and AppVeyor to all use conda in order to conda package libswmm? I think there is an opportunity for us to have our cake and eat it too here if we can figure out another way to get packaging done and have a lighter faster CI system for day to day routine development. And I see what @bemcdonnell is describing as a good step in the right direction to accomplishing this.

goanpeca · 2017-12-08T18:48:25Z

We need something like on the README.md, or as a CONTRIBUTING.md or BUILDING.md file.

Suggested content skeleton @michaeltryby @bemcdonnell.

I.a. Development environment (Normal Workflow)

A.) Windows

Install Dependencies

Visual Studio C++ 10. Link to download
Install CMake >= <VERSION> Link to download
Python 3.4. Link to download
Install the nrtest dependencies..... Links to download

Building

To build SWMM then run:

$ cmake -G "Visual Studio <VERSION>" -DCMAKE_INSTALL_PREFIX:PATH=output -DCMAKE_BUILD_TYPE:STRING=Release

Running

$ run-swmm --help

B.) Linux

Install Dependencies

Build dependencies... gcc >= libgcc >=
cmake
Install the nrtest dependencies..... Links to download

On Debian (.deb) based systems you can use

$ sudo apt install ...

On Centos (.rpm) based systems you can use

$ sudo yum install ...

Building

$ cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE:STRING=Release ..
$ make -j $CPU_COUNT
$ make install

Running

$ run-swmm --help

C.) OSX

InstallDependencies

Clang version >= ??? or
gcc version >= ???
cmake
Install python and the nrtest dependencies..... Links to download or commands

If using Homebrew

$ brew install ...

Building

$ cmake -DCMAKE_INSTALL_PREFIX=$PREFIX -DCMAKE_BUILD_TYPE:STRING=Release ..
$ make -j $CPU_COUNT
$ make install

Running

$ run-swmm --help

I.b. Development environment (If using conda)

A.) Windows

Install Dependencies

Visual Studio C++ 8 for python 2.7. Link to download
Visual Studio C++ 10 for python 3.4. Link to download
Visual Studio C++ 14 for Python 3.5 and 3.6. Link to download

Then you can create a development environment with all the dependencies, activate and build from there.

$ conda create -n swmmdev cmake conda-build python=<2.7, 3.4, 3.5, 3.6> nrtest ... <any other thing needed for testing>

Activate the environment:

$ activate swmmdev

Building (with conda-build)

(swmmdev) $ conda-build conda.recipe
(swmmdev) $ conda install libswmm --use-local

Running

(swmmdev) $ run-swmm --help

B.) OSX and Linux

Install Dependencies

Gcc... (provided as conda packages...)
cmake (conda package)
python (conda package ...)

Create environment:

$ conda create -n swmmdev cmake conda-build gcc libgcc python=<ANY VERSION> nrtest ... <any other thing needed for testing>

Activate the environment:

$ source activate swmmdev

Building (with conda-build)

(swmmdev) $ conda-build conda.recipe
(swmmdev) $ conda install libswmm --use-local

Running

(swmmdev) $ run-swmm --help

II. If you are developing for PySwmm

If you are developing for PySwmm you need to install these versions to work with how python is compiled on different platforms:

A.) Windows

Python 2.7

Visual Studio C++ 9 for Python 2.7. Link to download
Python 2.7. Link to download

Python 3.4

Visual Studio C++ 10 for Python 3.4. Link to download
Python 3.4. Link to download.

Python 3.5 and 3.6

Visual Studio C++ 14 for Python 3.5 and 3.6. Link to download
Python 3.5. Link to download
Python 3.6. Link to download

B.) Linux

....

C.) OSX

....

III. Regression Testing

To execute regression tests run:

$ python <script...>

IV. Unit tests

To execute unit tests run:

$ python <script...>

goanpeca · 2017-12-08T19:23:07Z

@bemcdonnell I can start working on this PR to update all this but I would like to know what you think. For the moment I updated the wiki

https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/wiki/Build-Instructions-for-Development

goanpeca · 2017-12-08T22:09:33Z

@lrntct you can update travis, but before we do that we first need to define what is the base system we will use. Using whatever is on travis its ok to have a generic testing infrastructure, but if we plan to build binaries and upload these as artifacts somewhere we need to define system specs and enforce them on travis. If we don't define this then its no different from "doing it on my home computer and sharing it with the rest of the world".

For SWMM since we only use a handful of extra libraries, its not so critical, but still, we should define the oldest compiler we can use to maximize compatibility (Like using Centos 6 with some specific gcc version as a base system or something like that).

This base system could be a docker image which could be used on travis/circle etc.

goanpeca · 2017-12-09T14:48:28Z

My apologies for being so pushy on this (conda) subject. I understand at the moment is not a good choice for CI or development workflow if someone has not used it before. I will be in charge of making the conda packages once we can sort out the nrtest regression tests on CI based on the work by @michaeltryby. Cheers.

See issues #133, #117, #116, #115, #114, #113, #112, #110

bemcdonnell closed this as completed Mar 11, 2018

karosc pushed a commit that referenced this issue Sep 1, 2023

Bug fixes for release 5.2.3

33889ee

See issues #133, #117, #116, #115, #114, #113, #112, #110

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Can't build swmm5 with conda-build #114

Can't build swmm5 with conda-build #114

lrntct commented Dec 6, 2017

bemcdonnell commented Dec 6, 2017

lrntct commented Dec 6, 2017

bemcdonnell commented Dec 7, 2017

goanpeca commented Dec 7, 2017 •

edited

Loading

lrntct commented Dec 7, 2017

goanpeca commented Dec 7, 2017 •

edited

Loading

bemcdonnell commented Dec 7, 2017

lrntct commented Dec 7, 2017

michaeltryby commented Dec 7, 2017

goanpeca commented Dec 8, 2017 •

edited

Loading

goanpeca commented Dec 8, 2017 •

edited

Loading

goanpeca commented Dec 8, 2017 •

edited

Loading

goanpeca commented Dec 9, 2017

Can't build swmm5 with conda-build #114

Can't build swmm5 with conda-build #114

Comments

lrntct commented Dec 6, 2017

bemcdonnell commented Dec 6, 2017

lrntct commented Dec 6, 2017

bemcdonnell commented Dec 7, 2017

goanpeca commented Dec 7, 2017 • edited Loading

lrntct commented Dec 7, 2017

goanpeca commented Dec 7, 2017 • edited Loading

bemcdonnell commented Dec 7, 2017

lrntct commented Dec 7, 2017

michaeltryby commented Dec 7, 2017

goanpeca commented Dec 8, 2017 • edited Loading

I.a. Development environment (Normal Workflow)

A.) Windows

Install Dependencies

Building

Running

B.) Linux

Install Dependencies

On Debian (.deb) based systems you can use

On Centos (.rpm) based systems you can use

Building

Running

C.) OSX

InstallDependencies

If using Homebrew

Building

Running

I.b. Development environment (If using conda)

A.) Windows

Install Dependencies

Building (with conda-build)

Running

B.) OSX and Linux

Install Dependencies

Building (with conda-build)

Running

II. If you are developing for PySwmm

A.) Windows

Python 2.7

Python 3.4

Python 3.5 and 3.6

B.) Linux

C.) OSX

III. Regression Testing

IV. Unit tests

goanpeca commented Dec 8, 2017 • edited Loading

goanpeca commented Dec 8, 2017 • edited Loading

goanpeca commented Dec 9, 2017

goanpeca commented Dec 7, 2017 •

edited

Loading

goanpeca commented Dec 7, 2017 •

edited

Loading

goanpeca commented Dec 8, 2017 •

edited

Loading

goanpeca commented Dec 8, 2017 •

edited

Loading

goanpeca commented Dec 8, 2017 •

edited

Loading