Provide cbc for Conda on Windows

[slel]

• I read the conda-forge documentation and could not find the solution for my problem there.

Issue:

Currently conda-forge provides coin-or-cbc for Linux and macOS.
Could it be provided for Windows too? If not, ideally explain why
(e.g. are some dependencies not available for Windows?).

Requested at:

• Stack Overflow question 58868054: Install coincbc using Conda in Windows

[tkralphs]

Yes, there is a reason that Cbc isn't currently provided for Windows and it's a rather silly one. In short, the easiest fix is to release the new version 3.0, which should address the issue, but this is still in process (see coin-or/Cbc#374).

The slightly longer version is that the latest release 2.10.5 won't build under Conda because our configure script does not properly detect the Fortran compiler flang and a working Fortran compiler is required. In master, we have updated our autotools setup so that I believe the configure script will either detect flang correctly or possibly not even require it. The effort involved in fixing 2.10.5 wasn't justified, given that we hope to retire it soon anyway. There are also binaries on bintray for Windows that seems to be working fine.

[h-vetinari]

So the flang-detection should be something that can be fixed, either on conda-forge side or through a direct patch on the feedstock. I may give it a shot later, now that conda-forge/staged-recipes#14962 is becoming unstuck.

[tkralphs]

This is probably not that easy unless you want to actually patch the configure script directly, which I guess could be done.

[El-Don-Quijote]

Any updates on this one? Would be very convenient to have it work again in Windows :)

[h-vetinari]

PRs are always welcome. I'll try to support if possible, but I cannot lead that effort ATM.

[jpfeuffer]

@tkralphs Are there any updates to this? Is patching configure still the only way to go? Can we maybe create packages from the master branch?

If patching configure is the only way to go, can you link the lines that were changed in your big Refactoring PR that might need to be included into the patch such that flang is found? I cannot find any changes regarding flang in your repository.

[tkralphs]

Unfortunately, the situation hasn't changed a lot. I am still hopeful about locating a volunteer to finish the refactoring or maybe finding time to do it myself sometime over summer. In the meantime, if it's acceptable to make packages from the mater branch, then I would be willing to try to help with that, but it seems like a can of worms. If nothing else, we would need packages for all dependencies as well.

In master, we shouldn't need any patching of anything for flang. In fact, it's probably not even needed at all. I think we have a test for flang in 2.10 that was fatal if it failed, even in cases where flang was not needed. I'm traveling for the next 10 days and can then take another look at this.

[h-vetinari]

We don't have a working flang for windows in conda-forge yet (well, we do have a cobbled-together one based on classic-flang, but it's essentially unused). The new LLVM flang is still WIP though it's something I'm pushing for with quite high priority. It should be usable towards the end of the year, I hope.

In the meantime, if it's acceptable to make packages from the mater branch, then I would be willing to try to help with that, but it seems like a can of worms.

We could publish something into a coinor_dev label, where things are not discoverable by default, and need a specific opt-in like

conda install -c conda-forge/label/coinor_dev coin-or-cbc

[brynpickering]

@h-vetinari after the merging of conda-forge/flang-feedstock#28 is it correct to say there is a working flang for windows in conda-forge? Looks like moving things forward requires starting with a successful windows build on host dependencies, in (I think) this order:

1. https://github.com/conda-forge/coin-or-utils-feedstock
2. https://github.com/conda-forge/coin-or-osi-feedstock
3. https://github.com/conda-forge/coin-or-clp-feedstock
4. https://github.com/conda-forge/coin-or-cgl-feedstock

Happy to contribute to move this forward, if some guidance can be given!

[h-vetinari]

@h-vetinari after the merging of conda-forge/flang-feedstock#28 is it correct to say there is a working flang for windows in conda-forge?

Flang 17 should still be considered pretty experimental. Flang 18 is looking to be much more suitable, and as soon as we have that built, I'll advocate for rolling this out as our default fortran compiler on windows. At around the same time, we can start trying to build the coin-or stack as well. Of course you can experiment with Flang 17 already, but there are still many rough edges.

[El-Don-Quijote]

@h-vetinari are you closer to providing cbc for conda on Windows machine after the merge of conda-forge/flang-feedstock#38? Sorry to have no means to contribute, I am just an occasional user of cbc for some optimization problems.
All the best and thank you!

[jpfeuffer]

There seems to have been another failed PR that tried to add Windows support in the coin-utils recipe.
conda-forge/coin-or-utils-feedstock#15

It still seems to be not working (although the CI logs are deleted by now). Apparently it is not only the missing/incomplete flang support, but also the fact that coinor 2.10 has a very complex build system that makes a lot of assumptions on filetypes, compiler flags etc.
It would be much better with coinor 3 or the master branch but unfortunately there is no release of it in sight.

[tkralphs]

I need to dig back into this to see what's really going on. As I said back at the beginning, I can build the stable version of CoinUtils very smoothly and easily on my own Windows machine with the Visual Studio compiler and no Fortran compiler present. CoinUtils is also built successfully with the Visual Studio compiler on Github Actions, also with no Fotran compiler present. So this is still something specific to conda. I've looked over the comment from the initial debugging of the Windows builds (see conda-forge/coin-or-coinutils-feedstock#1 and it's not very clear to me exactly why the Fortran compiler was originally needed. It appears to be because of name-mangling associated with linking the BLAS and LAPACK libraries. The fortran compiler is needed just to see how to link these libraries, since there are some different name-mangling conventions (I am not particularly an expert on this).

I saw a mention of CBLAS when I was browsing changes and linking to this as the BLAS library could already fix the issue. But with all that said, the failure at the moment has nothing to do with CoinUtils. It is here

D:\a\1\s\recipe\build.sh: line 5: mkdir: command not found

This is a bash script, so it is rather odd that mkdir is not an available command. I don't have the bandwidth to try to replicate this locally right now, but getting past this should have nothing to do with CoinUtils. If someone can dig up another random project somewhere that uses bash on Windows, maybe we can see a difference.

[jpfeuffer]

@tkralphs I fixed the bash errors in conda-forge/coin-or-utils-feedstock#16

But we get all kinds of weird errors in your configure script.

First the unfortunately expected problems with fortran

checking whether we are using the GNU Fortran 77 compiler... no
checking whether flang accepts -g... no
configure: WARNING: The flags FFLAGS="FFLAGS" do not work.  I will now just try '-O', but you might want to set FFLAGS manually.
configure: WARNING: The flags FFLAGS=-O do not work. I will continue with empty FFLAGS, but you might want to set FFLAGS manually.
configure: Fortran compiler options are: 

Then this:

./configure: eval: line 4959: unexpected EOF while looking for matching `"'
./configure: eval: line 4960: syntax error: unexpected end of file

Can you check this and compare the flags and variables with your GitHub actions setup?

[tkralphs]

From the output, it seems that a Fortran compiler (flang) is still present and is detected and I see that this is because we're still requesting it to be installed. I don't think there is an easy way to make this work, but as I indicated above, it's probably not needed. My suggestion was to try not having any Fortran compiler present, which works fine when building on Github Actions. I guess the issue back when we were debugging was determining the name-mangling needed to link to BLAS and LAPACK, which wouldn't be an issue if we use CBLAS and I see CBLAS installed.

The errors in the configure script might be a line-ending problem, but I don't see where those are arising. A direct link to the output where you copied these from would be helpful. I don't know what the setup is on the build machines being used here is, but these same configure scripts are building on a wide range of platforms on Github Actions without problems, so it's doubtful it is the scripts themselves.

[jpfeuffer]

Okay! I will try without Flang.

I fixed the line ending problem by using msys make instead of the regular conda make.

[h-vetinari]

I'm not sure if autotools (or your configure scripts) correctly recognises the new flang. There used to be an old flang that had gotten some build system support, but it's not the same (well, except written by overlapping groups of people).

[julian-belina]

Is there any update on this? A windows installation would be verry handy for me

[jpfeuffer]

Hi!

We recently updated the coinutils package to support windows. I think the plan was to start on cbc next since IIRC it is the only other one depending directly on blas/lapack and therefore needs special care.

The next 3 weeks are however a bit uncertain regarding my time for open source.
It is definitely something that I still plan to tackle since one of our libraries also depends on CoinOR.

[julian-belina]

Thanks for the update, I am looking forward to it.

[tkralphs]

I think the plan was to start on cbc next

Actually, we need to proceed in dependency order CoinUtils->Osi->Clp->Cgl->Cbc. But I believe that the remaining projects should be straightforward, since I believe CoinUtils is actually the only project with a direct dependency on BLAS/LAPACK.

Just taking a quick look, I see BLAS/LAPACK mentioned in the recipes of coin-or-osi, coin-or-clp, and coin-or-cgl, but I think this is not needed, since they all depend on coin-or-utils, which is the only one with a direct dependency.

[tkralphs]

Just wanted to see if we could revisit this. It would be really nice to get it done! The next project in line is Osi.

[h-vetinari]

The upcoming flang 19 is looking much better (aside from SciPy which was working before, it also works for openblas, mumps and various other libraries). You can test it by adding the channel conda-forge/label/llvm_rc. An example of how to do this is conda-forge/scipy-feedstock#269

Looking at the comments further up, you might have to hack the autoconf scripts (depending on the choices they make), if setting FC=flang-new is not enough

[tkralphs]

We eventually came up with a solution in conda-forge/coin-or-utils-feedstock/pull/18 that doesn't require a Fortran compiler. This is working and what is needed now is to replicate it in Osi, Clp, Cgl, and Cbc.