[OpenCV] Update to v4.1.1 #5169

cenit · 2019-01-16T18:08:32Z

Any feedback is welcome!

fixes: #2065
fixes: #2749
fixes: #2799
fixes: #3095
fixes: #3418
fixes: #4160
fixes: #4601
fixes: #4725
fixes: #4758
fixes: #5101
fixes: #5557
fixes: #5876
fixes: #6728
fixes: #6994

cenit · 2019-02-10T17:53:27Z

Waiting on my other open PRs before asking this merge...

cenit · 2019-02-27T12:34:36Z

Thanks to #5417 finally completed, I think that we can start discussing this PR @Codiferous. Please tell me if you see any regression, so that I can start fixing bugs and problems. It could be possible that some ports must be made dependent on opencv3 instead of opencv, so that they keep working as before without big fixups, waiting for the upstream projects to be updated.

cenit · 2019-02-27T12:37:22Z

btw, in parallel of the opencv3 rename, what's your opinion on #3360? Shall I close it and forget it? I think it could be interesting in some edge cases... and not dangerous in any way, am i right?

Rastaban · 2019-02-27T22:06:57Z

In the PR test this is causing failures in zxing-cpp on x86-windows and x64-windows, which has a dependancy on openvc.

cenit · 2019-02-28T14:23:32Z

@Rastaban thanks for noticing. I added a patch and zxing-cpp is building ok locally. Is it fixed here? I see a red cross anyway

Rastaban · 2019-03-05T21:36:02Z

Here are the test results. The left results column is the latest commit to this PR, the right results column is test results from master: Blank space means the port did not exist and Skip typically means somthing it has a dependancy on failed so it was not built.

Processing arm-uwp                        412 vs 411
    ade                                                         Pass vs 
    opencv3                                                     Skip vs 
Processing arm64-windows                  467 vs 466
    ade                                                         Pass vs 
    opencv3                                                     Skip vs 
Processing x64-linux                      603 vs 605
    ade                                                         Fail vs 
    opencv                                                      Skip vs Pass
    opencv3                                                     Fail vs 
    zxing-cpp                                                   Skip vs Pass
Processing x64-osx                        594 vs 596
    ade                                                         Fail vs 
    opencv                                                      Skip vs Pass
    opencv3                                                     Fail vs 
    zxing-cpp                                                   Skip vs Pass
Processing x64-uwp                        438 vs 439
    ade                                                         Pass vs 
    opencv                                                      Skip vs Pass
    opencv3                                                     Fail vs 
    zxing-cpp                                                   Skip vs Pass
Processing x64-windows                    858 vs 857
    ade                                                         Pass vs 
    opencv3                                                     Fail vs 
Processing x64-windows-static             772 vs 773
    ade                                                         Pass vs 
    opencv                                                      Skip vs Pass
    opencv3                                                     Fail vs 
    zxing-cpp                                                   Skip vs Pass
Processing x86-windows                    838 vs 837
    ade                                                         Pass vs 
    opencv3                                                     Fail vs

Here are all of the failures logs:
failureLogs.zip

As a sidenote: We are working on getting the CI system out from behind the wall so you can see the results and poke around yourself. I'm not sure when that will happen though. Until then let us know if there are any logs that would be useful and we should be able to get them to you.

cenit · 2019-03-06T17:57:49Z

I pushed some patches here and there, on the OpenCV dependencies.
I can now build successfully (locally)
on macOS/linux: ./vcpkg install opencv[eigen,jpeg,opengl,openmp,png,tiff,webp,nonfree,contrib,eigen,ffmpeg,gdcm,ipp,jasper,jpeg,openexr,opengl,openmp,png,sfm,tbb,tiff,webp]
on windows: .\vcpkg.exe install opencv[eigen,jpeg,opengl,openmp,png,tiff,webp,nonfree,contrib,eigen,ffmpeg,gdcm,ipp,jasper,jpeg,openexr,opengl,openmp,png,sfm,tbb,tiff,webp,cuda,ovis]

ovis is a little bit problematic to fix on non-windows, there is a strange ninja error I've never seen related to a non-expanded variable inside FLAGS.
and for cuda I don't have any linux system to test it

cenit · 2019-03-06T17:58:48Z

it was a really tedious work. There are so many broken libraries... 😢 😭

cenit · 2019-03-06T18:01:36Z

again, this PR has become a huge PR with many different packages involved. First time something similar was closed and rebuilt in many different parts, copying my work and without even cherry picking my commits (no trace left of it, at the beginning I accepted it but for me it is important to have trace of the work done), last time another admin here merged all of them (luckily picking my commits). I hope that now is ok in this way, also to have a positive feedback by the CI

cenit · 2019-03-06T22:13:35Z

@Rastaban can you please give me an update of failures/pass, since the green mark is still a mirage?

cenit · 2019-03-07T12:43:46Z

i also tested builds on windows with static library linkage (but dynamic crt) and with the latest commit now also this config is fully working (at least here, at least for me)

cenit · 2019-03-07T12:46:33Z

I also added extension for opencv3 from #5123 (a little bit different because it does not work otherwise in static builds...), from #4802 and from #3635, so that in case this PR is accepted there's no regression in functionalities for the backward-compatible opencv3 port

cenit · 2019-03-07T13:04:11Z

While waiting for the logs, in order to understand what to fix next, I also have a problem to submit: due to the official FindLibLZMA.cmake not supporting debug/release configs, OpenCV does not propagate correctly this dependency inside its module file (it is built correctly, but to consume it I have to manually fix a cmake module). Let me explain it better:

Inside the file OpenCVModules.cmake, which is installed in vcpkg/installed/x64-windows-static/share/opencv, we can find a string which is similar to

C:/Users/username/vcpkg/installed/x64-windows-static/debug/lib/lzma.lib;

it must be changed with

\$<\$<NOT:\$<CONFIG:DEBUG>>:C:/Users/username/vcpkg/installed/x64-windows-static/lib/lzma.lib>;\$<\$<CONFIG:DEBUG>:C:/Users/username/vcpkg/installed/x64-windows-static/debug/lib/lzma.lib>;

Shall we produce a regex in the portfile to accomplish this? Is anyone able to help me? (regex are not really part of my strength, at least for now :D )
Maybe @ras0219-msft can you help me?

cenit · 2019-03-12T12:36:11Z

@Rastaban Sorry for the ping, but do you have any news on the CI logs?

cenit · 2019-03-12T12:57:13Z

I was already managing "nonfree" in opencv3/4, so the latest merge is only to resolve conflicts, but nothing should have been changed from before.

Cheney-W · 2019-03-14T05:52:26Z

Here are the latest test results:

Processing x64-linux                      623 vs 618 
     opencv                        **regression**                Fail vs Pass
Processing x64-windows                    876 vs 888
     gdcm2                         **regression**                Fail vs Pass
     libwebp                       **regression**                Fail vs Pass
Processing x86-windows                    855 vs 867
     gdcm2                         **regression**                Fail vs Pass
     libmupdf                      **regression**                Fail vs Pass
     libwebp                       **regression**                Fail vs Pass

And the log as below:
failureLogs.zip

cenit · 2019-03-15T07:44:46Z

I can't reproduce any libmupdf problem locally. I wonder what's the problem because I can clearly see the openjpeg.lib in the command line at linking stage and those missing symbols are exported by that lib.

I have merged with master in the meantime, during the weekend I should have more free time to check remaining issues: I hope to see this PR closed soon, because it is becoming too huge to maintain

cenit · 2019-03-15T07:49:28Z

is it because I didn't update the CONTROL files that maybe some ports are not rebuilt?

cenit · 2019-03-16T13:35:48Z

#5625 might be the proper fix for the problem described in #5169 (comment). Which is not a problem discoverable with CI at the moment, due to absence of any test on built libraries, am I right?

cenit · 2019-03-19T10:13:43Z

I imported also a patch for libmupdf which should fix also problems for x86 builds.
Can I have an update at the end of what's still broken and what not? I doubt the green mark will be here at the end of the CI tests... 😢
To be honest, let me repeat myself. I think this PR is exploding, I cannot fix all the packages here only because my dependencies were broken from the beginning and they were a dependency for other broken packages...

vicroms · 2019-03-19T18:57:54Z

While waiting for the logs, in order to understand what to fix next, I also have a problem to submit: due to the official FindLibLZMA.cmake not supporting debug/release configs, OpenCV does not propagate correctly this dependency inside its module file (it is built correctly, but to consume it I have to manually fix a cmake module). Let me explain it better:
Inside the file OpenCVModules.cmake, which is installed in vcpkg/installed/x64-windows-static/share/opencv, we can find a string which is similar to
C:/Users/username/vcpkg/installed/x64-windows-static/debug/lib/lzma.lib;

it must be changed with
$<$<NOT:$CONFIG:DEBUG>:C:/Users/username/vcpkg/installed/x64-windows-static/lib/lzma.lib>;$<$CONFIG:DEBUG:C:/Users/username/vcpkg/installed/x64-windows-static/debug/lib/lzma.lib>;

Shall we produce a regex in the portfile to accomplish this? Is anyone able to help me? (regex are not really part of my strength, at least for now :D )
Maybe @ras0219-msft can you help me?

I have fixed a similar problem in the past by doing a STRING(REGEX REPLACE ..) in the portfile.
If this is still an issue I can take a look at it.

…cenit/opencv4

cenit · 2019-08-02T20:55:23Z

Thanks for merging. I was doing the same thing :)

cenit · 2019-08-02T20:58:31Z

@vicroms vcpkg_configure_cmake does not contain the "WITH_HALIDE" new feature

edit: OTOH, is it now possible to remove all the other WITH_xxx because vcpkg_check_features takes care of them?

vicroms · 2019-08-02T22:27:34Z

Oops, forgot to add it.

vcpkg_check_features() does not pass the WITH_xxx values to vcpkg_configure_cmake() (but it would be really could if it did), so we still have to pass them manually.

vicroms · 2019-08-05T17:19:34Z

Thanks for investigating that ffmpeg patch!

I'll bring this issue to our team's offline discussion.

There is a new problem we should keep an eye on, for future vcpkg development: a staging area. This is not a problem now, since the feature works and I still don’t know what we are losing.

ffmpeg would like a basic opencv lib to enable some features (the patch was there to make it possible, but the feature is not even exposed). In turn, opencv would like ffmpeg ready to enable some other features. This would require a staging area, so that if i’d like to install opencv[ffmpeg] this would trigger:
1. build `opencv[core]` in a staging area
2. build ffmpeg pointing to `opencv[core]` 
3. build `opencv[ffmpeg]`

I'm hoping to merge this PR soon, are there any issues that you are aware that should be addressed first?

cenit · 2019-08-05T20:34:21Z

Not any that I am aware. Please proceed with merging!
I am also on vacation this week (after such a long time without you'd never imagine 🤣), which could mean two different things:
a) I have more time to help in case of big problems
b) or I might just turn off pc for a while...

Finally, what about versioning in the library filenames? Shall we keep it or remove? Since there can only be one version in Vcpkg installed foldertree, even if anyone uses overlays, I'd push for unversioned library file names. This would ease work for devs on projects that use explicit link inside VS solutions, because starting from now I'd hope for more aggressive updates for opencv 😄

vicroms · 2019-08-07T17:16:21Z

I am also on vacation this week

Enjoy your vacations!

Finally, what about versioning in the library filenames? Shall we keep it or remove?

We think it's OK to remove the version from the library filenames.

cenit · 2019-08-08T17:13:25Z

Since it wouldn't be bad to find OpenCV 4 when I will be back at work, I did the removal of versioning from windows library names

vicroms · 2019-08-09T21:34:41Z

/azp run

azure-pipelines · 2019-08-09T21:34:50Z

Azure Pipelines successfully started running 1 pipeline(s).

vicroms · 2019-08-12T16:30:03Z

🎉🎉🎉🎉🎉

Finally after months of work this PR has been merged!

Thanks to everyone that contributed with their feedback and specially to @cenit for having the willpower and patience to see this through! 😄

The plan in the future is to handle updates to OpenCV much faster, and I think all the improvements made to the port in this PR will enable us to achieve that goal.

cenit · 2019-08-12T18:05:11Z

Thank you too, very much

cenit · 2019-08-12T18:06:29Z

It has been an extremely long journey, with so many parallel PRs necessary to make this one see the light! Thanks to everybody. It has been difficult, but very satisfying

cenit · 2019-08-12T22:53:37Z

@vicroms not sure we handled properly the meta-package thing you recently pushed.
This is the message I have when trying to update

Warning: could not reinstall feature opencv[eigen]:x64-linux
Warning: could not reinstall feature opencv[flann]:x64-linux
Warning: could not reinstall feature opencv[jpeg]:x64-linux
Warning: could not reinstall feature opencv[opengl]:x64-linux
Warning: could not reinstall feature opencv[png]:x64-linux
Warning: could not reinstall feature opencv[tiff]:x64-linux
Warning: could not reinstall feature opencv[eigen]:x64-windows
Warning: could not reinstall feature opencv[flann]:x64-windows
Warning: could not reinstall feature opencv[ipp]:x64-windows
Warning: could not reinstall feature opencv[jpeg]:x64-windows
Warning: could not reinstall feature opencv[opengl]:x64-windows
Warning: could not reinstall feature opencv[png]:x64-windows
Warning: could not reinstall feature opencv[tiff]:x64-windows
Warning: could not reinstall feature opencv[webp]:x64-windows

not the nicest welcome to the latest OpenCV release. Without proper exposure in the metapackage, many features are hidden behind the opencv4 package...

cenit · 2019-08-12T22:54:45Z

I will open soon another PR, trying to fix these things and also update OpenCV 3 to a proper status. Having OpenCV3 still around in a broken state is something I cannot stand, sorry

vicroms · 2019-08-13T00:33:49Z

@vicroms not sure we handled properly the meta-package thing you recently pushed.
This is the message I have when trying to update

Warning: could not reinstall feature opencv[eigen]:x64-linux
Warning: could not reinstall feature opencv[flann]:x64-linux
Warning: could not reinstall feature opencv[jpeg]:x64-linux
Warning: could not reinstall feature opencv[opengl]:x64-linux
Warning: could not reinstall feature opencv[png]:x64-linux
Warning: could not reinstall feature opencv[tiff]:x64-linux
Warning: could not reinstall feature opencv[eigen]:x64-windows
Warning: could not reinstall feature opencv[flann]:x64-windows
Warning: could not reinstall feature opencv[ipp]:x64-windows
Warning: could not reinstall feature opencv[jpeg]:x64-windows
Warning: could not reinstall feature opencv[opengl]:x64-windows
Warning: could not reinstall feature opencv[png]:x64-windows
Warning: could not reinstall feature opencv[tiff]:x64-windows
Warning: could not reinstall feature opencv[webp]:x64-windows

not the nicest welcome to the latest OpenCV release. Without proper exposure in the metapackage, many features are hidden behind the opencv4 package...

That was a HUGE oversight on my part, the right thing to do is for the meta-package to expose all the features in opencv4.

cenit · 2019-08-13T08:01:31Z

Thanks for having fixed this quickly in #7659

luzpaz mentioned this pull request Jan 18, 2019

[opencv] Update to OpenCV 3.4.4 #4955

Closed

3 tasks

cenit mentioned this pull request Jan 22, 2019

[libwebp] update to v1.0.2 #5210

Closed

3 tasks

cenit mentioned this pull request Jan 30, 2019

[OpenEXR] Update to v2.3.0 #5227

Closed

5 tasks

Codiferous added the wip label Feb 8, 2019

cenit changed the title ~~[WIP] [OpenCV 4] Add a port of this new release~~ [OpenCV 4] Update to v4, keep v3 as another port Feb 27, 2019

cenit changed the title ~~[OpenCV 4] Update to v4, keep v3 as another port~~ [OpenCV] Update to v4, keep v3 as another port Feb 28, 2019

cenit changed the title ~~[OpenCV] Update to v4, keep v3 as another port~~ [OpenCV] Update to v4, keep v3 as another port. Fix many dependencies along the way Mar 7, 2019

cenit mentioned this pull request Mar 19, 2019

[wip] Fix CMAKE_SYSTEM_PROCESSOR (Resubmission of #4688) #5697

Closed

vicroms self-assigned this Mar 19, 2019

Merge branch 'master' of https://github.com/Microsoft/vcpkg into dev/…

cf227f5

…cenit/opencv4

[opencv] Feature halide

986aaff

vicroms and others added 2 commits August 2, 2019 16:07

Fix regression in UWP

41d250e

[ffmpeg] remove unnecessary patch pointing to old OpenCV version

50af0c6

[opencv] remove versioning from windows dll filenames

0cd7053

vicroms added 4 commits August 8, 2019 11:30

[opencv] Move port to opencv4

c956f05

[opencv,opencv4] Make meta-package install OpenCV 4.1

96800f6

[opencv3] Rename old port opencv->opencv3

8e2ba63

Add failure messages when another OpenCV is already installed

62ca7b8

vicroms merged commit 4fb5152 into microsoft:master Aug 12, 2019

traversaro mentioned this pull request Aug 13, 2019

YARP does not compile with OpenCV 4 robotology/yarp#1672

Closed

vicroms mentioned this pull request Aug 15, 2019

[opencv] Expose all features from opencv4 in meta-package #7659

Merged

cenit deleted the dev/cenit/opencv4 branch September 25, 2019 18:02

cenit mentioned this pull request Jul 5, 2020

[many ports] Remove _find_package guards that break *_FOUND #12157

Merged

[OpenCV] Update to v4.1.1 #5169

[OpenCV] Update to v4.1.1 #5169

Conversation

cenit commented Jan 16, 2019 • edited Loading

cenit commented Feb 10, 2019

cenit commented Feb 27, 2019 • edited Loading

cenit commented Feb 27, 2019

Rastaban commented Feb 27, 2019

cenit commented Feb 28, 2019

Rastaban commented Mar 5, 2019

cenit commented Mar 6, 2019

cenit commented Mar 6, 2019

cenit commented Mar 6, 2019

cenit commented Mar 6, 2019

cenit commented Mar 7, 2019

cenit commented Mar 7, 2019

cenit commented Mar 7, 2019

cenit commented Mar 12, 2019

cenit commented Mar 12, 2019

Cheney-W commented Mar 14, 2019

cenit commented Mar 15, 2019

cenit commented Mar 15, 2019

cenit commented Mar 16, 2019

cenit commented Mar 19, 2019

vicroms commented Mar 19, 2019

cenit commented Aug 2, 2019

cenit commented Aug 2, 2019 • edited Loading

vicroms commented Aug 2, 2019 • edited Loading

vicroms commented Aug 5, 2019

cenit commented Aug 5, 2019 • edited Loading

vicroms commented Aug 7, 2019 • edited Loading

cenit commented Aug 8, 2019

vicroms commented Aug 9, 2019

azure-pipelines bot commented Aug 9, 2019

vicroms commented Aug 12, 2019

cenit commented Aug 12, 2019

cenit commented Aug 12, 2019

cenit commented Aug 12, 2019

cenit commented Aug 12, 2019

vicroms commented Aug 13, 2019

cenit commented Aug 13, 2019

cenit commented Jan 16, 2019 •

edited

Loading

cenit commented Feb 27, 2019 •

edited

Loading

cenit commented Aug 2, 2019 •

edited

Loading

vicroms commented Aug 2, 2019 •

edited

Loading

cenit commented Aug 5, 2019 •

edited

Loading

vicroms commented Aug 7, 2019 •

edited

Loading