Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OpenMVG/OpenMVS] fix tools #12229

Merged
merged 76 commits into from
Aug 28, 2020
Merged

Conversation

cenit
Copy link
Contributor

@cenit cenit commented Jul 2, 2020

Upgrade SuiteSparse, add new vlfeat port, restore the custom FindLAPACK (for clapack) I wrote one year ago that went lost (and with it each and every port depending on clapack was broken again on linux), remove the unnecessary FindBLAS module, and finally

  • add openMVG tools
  • fix openMVG build error
  • enable cuda on openMVS
  • handle openMVS tools properly

along the way, it fixes: #12387 (a working qt5 was necessary to build tools cited above) and fixes many problems encountered along the way towards a working solution

@JackBoosY JackBoosY added category:port-bug The issue is with a library, which is something the port should already support category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist labels Jul 3, 2020
@NancyLi1013
Copy link
Contributor

The failures for openmvg

CMake Error at scripts/cmake/vcpkg_copy_tools.cmake:47 (message):
  Couldn't find this tool:
  D:/packages/openmvg_x86-windows/bin/openMVG_main_AlternativeVO.exe.
Call Stack (most recent call first):
  ports/openmvg/portfile.cmake:89 (vcpkg_copy_tools)
  scripts/ports.cmake:76 (include)

@cenit
Copy link
Contributor Author

cenit commented Jul 3, 2020

@NancyLi1013 thanks for the review. I was copying tools also when feature was not enabled...

@cenit cenit marked this pull request as draft July 3, 2020 11:22
@cenit cenit marked this pull request as ready for review July 3, 2020 13:14
@cenit cenit force-pushed the dev/cenit/openmvs_openmvg branch from e53fdfb to 4ed4059 Compare July 3, 2020 13:28
@cenit
Copy link
Contributor Author

cenit commented Jul 5, 2020

I cannot reproduce CI build failures locally. Both x64-windows-static and Linux build fine here.
I tried also macOS and x64-windows and they work ok (at least those are confirmed by CI)
@NancyLi1013 @BillyONeal Any help is appreciated understanding what's going on

@cenit
Copy link
Contributor Author

cenit commented Jul 5, 2020

I was expecting a "openmvs now passes on CI in x64-linux, please remove it from ci.baseline.txt", this is the opposite 🤣 (also openmvg now fails for CI)

@cenit cenit marked this pull request as draft July 6, 2020 15:51
@cenit
Copy link
Contributor Author

cenit commented Jul 6, 2020

Converted to draft due to some problems highlighted. Will re-open it as soon as they are fixed

@cenit
Copy link
Contributor Author

cenit commented Jul 7, 2020

Errors highlighted that I am fixing are due to problems within the libraries themselves, not due to CI errors.

Error in x64-windows-static is unrelated to this PR but due to dependency problems:

openblas.lib(memory.c.obj) : error LNK2005: DllMain already defined in vlsift.lib(generic.c.obj)

Simply, this is the first port using both libraries and highlighting that they are build incorrectly.

x64-linux too is due to an error in building SuiteSparse with OpenBLAS.

The bigger problem is that I don't have those problems locally, both in x64-Linux and in x64-windows-static, so it might be due to binarycaching libraries with problems and not hashing modifications in external files that might have fixed them in the meantime...
What should I do? Shall we try triggering a rebuild of only those problematic libs with a CONTROL bump? @NancyLi1013 @BillyONeal

@cenit cenit marked this pull request as ready for review July 7, 2020 12:17
@cenit
Copy link
Contributor Author

cenit commented Jul 7, 2020

ok both libraries now work when used in a downstream project. Before they were maybe building here on vcpkg, but for sure they were broken when imported downstream...

@cenit
Copy link
Contributor Author

cenit commented Jul 7, 2020

@cdcseacave are you interested in accepting most of the fixes for OpenMVS upstream?

@cdcseacave
Copy link
Contributor

Thank you for making this patches. I understand most of the OpenMVS build changes and I can apply them directly in the main original repo as long as they maintain the compatibility with Linux and MacOS (from your tests it seems to do). Some changes I do not understand though, like those at the end of the main CMakeLists.txt and in MvgMvsPipeline.py. Can you pls explain why are those needed and if I should make them in the main repo as well?

@cenit
Copy link
Contributor Author

cenit commented Jul 7, 2020

MvgMvsPipeline.py modifications are characteristic for vcpkg. Since we know perfectly where the tools from openmvg are located, I applied those info to fix relative paths provided and use correct ones. Those shall not be upstreamed, since they might be broken if the user does not adopt a folder strategy identical to vcpkg.
Maybe only fixes related to ComputeFeatures and ComputeMatches calls can be applied (original ones were triggering errors as invalid, are the new ones ok in your opinion?)

I don’t understand exactly otoh which modifications are you referring to when you were asking for details regarding cmake toolchain.

@Neumann-A
Copy link
Contributor

@NancyLi1013: He should do a git checkout master -- ports/qt5-base/portfile.cmake to reset the changes to the portfile of qt5-base since that are only unnecessary ws changes which changes the hash of that port unnecessarily

@NancyLi1013
Copy link
Contributor

@Neumann-A
Thanks for your reply. I checked the changes again, there are something about spaces and without any logical changes.
So it is unnecessary to update the version.

@Neumann-A
Copy link
Contributor

Neumann-A commented Aug 25, 2020

@cenit: You need to of course update your master first before you try to restore some changes ;)

or use :

git remote add upstream https://github.com/microsoft/vcpkg.git
git fetch upstream
git checkout upstream/master -- ports/qt5-base/portfile.cmake

you currently restored an old version of qt5-base/portfile.cmake

@cenit cenit force-pushed the dev/cenit/openmvs_openmvg branch from 6ee4b30 to d29561c Compare August 25, 2020 10:11
@cenit
Copy link
Contributor Author

cenit commented Aug 25, 2020

I removed the unnecessary whitespaces-only changes in qt5-base portfile, which remained after the fix for #12387 was applied externally from this PR and merged with master.

Please let me know if you have any other question or review for this PR

@JackBoosY
Copy link
Contributor

The only thing we need to do is test this PR.

@NancyLi1013
Copy link
Contributor

All features have finished testing. The test results are as follows:

  • dlib[cuda,fftw3,sqlite3] x86-windows x64-windows-static pass

  • geogram[graphics] x86-windows x64-windows-static pass

  • lapack-reference[noblas,blas-select] x86-windows pass

  • lapack-reference[noblas,blas-select] x64-windows-static failed (Not support static build on Windows)

  • lapack-reference[cblas] x86-windows x64-windows-static failed (Seems to be fixed in PR [lapack-reference] fix cblas feature #12929)

  • mlpack[tools] x86-windows pass

  • mlpack[tools] x86-windows x64-windows-static failed

    if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
        file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/bin" "${CURRENT_PACKAGES_DIR}/debug/bin")
    endif())
  • openmvg[opencv,software] x86-windows pass

  • openmvg[opencv] x64-windows-static pass

Note: openmp is currently broken on Windows. Has been disabled on Windows.

  • openmvg[software] x64-windows-static failed
[311/355] cmd.exe /C "cd . && F:\12229\vcpkg\downloads\tools\cmake-3.17.2-windows\cmake-3.17.2-win32-x86\bin\cmake.exe -E vs_link_exe --intdir=software\SfM\CMakeFiles\openMVG_main_ComputeFeatures.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\link.exe  @CMakeFiles\openMVG_main_ComputeFeatures.rsp  /out:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.exe /implib:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.lib /pdb:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.pdb /version:0.0  /machine:x64 /nologo  /debug /INCREMENTAL /subsystem:console  && cd ."
FAILED: Windows-AMD64-Debug/openMVG_main_ComputeFeatures.exe 
cmd.exe /C "cd . && F:\12229\vcpkg\downloads\tools\cmake-3.17.2-windows\cmake-3.17.2-win32-x86\bin\cmake.exe -E vs_link_exe --intdir=software\SfM\CMakeFiles\openMVG_main_ComputeFeatures.dir --rc=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\rc.exe --mt=C:\PROGRA~2\WI3CF2~1\10\bin\100177~1.0\x64\mt.exe --manifests  -- C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\link.exe  @CMakeFiles\openMVG_main_ComputeFeatures.rsp  /out:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.exe /implib:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.lib /pdb:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.pdb /version:0.0  /machine:x64 /nologo  /debug /INCREMENTAL /subsystem:console  && cd ."
LINK Pass 1: command "C:\PROGRA~2\MICROS~1\2017\ENTERP~1\VC\Tools\MSVC\1416~1.270\bin\Hostx64\x64\link.exe @CMakeFiles\openMVG_main_ComputeFeatures.rsp /out:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.exe /implib:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.lib /pdb:Windows-AMD64-Debug\openMVG_main_ComputeFeatures.pdb /version:0.0 /machine:x64 /nologo /debug /INCREMENTAL /subsystem:console /MANIFEST /MANIFESTFILE:software\SfM\CMakeFiles\openMVG_main_ComputeFeatures.dir/intermediate.manifest software\SfM\CMakeFiles\openMVG_main_ComputeFeatures.dir/manifest.res" failed (exit code 1120) with the following output:
   Creating library Windows-AMD64-Debug\openMVG_main_ComputeFeatures.lib and object Windows-AMD64-Debug\openMVG_main_ComputeFeatures.exp
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_constructor referenced in function "public: __cdecl openMVG::features::SIFT_Image_describer::SIFT_Image_describer(struct openMVG::features::SIFT_Image_describer::Params const &,bool)" (??0SIFT_Image_describer@features@openMVG@@QEAA@AEBUParams@012@_N@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_destructor referenced in function "public: virtual __cdecl openMVG::features::SIFT_Image_describer::~SIFT_Image_describer(void)" (??1SIFT_Image_describer@features@openMVG@@UEAA@XZ)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_new referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_delete referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_process_first_octave referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_process_next_octave referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_detect referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_calc_keypoint_orientations referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_vl_sift_calc_keypoint_descriptor referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
main_ComputeFeatures.cpp.obj : error LNK2019: unresolved external symbol __imp_update_gradient referenced in function "public: class std::unique_ptr<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128>,struct std::default_delete<class openMVG::features::Scalar_Regions<class openMVG::features::SIOPointFeature,unsigned char,128> > > __cdecl openMVG::features::SIFT_Image_describer::DescribeSIFT(class openMVG::image::Image<unsigned char> const &,class openMVG::image::Image<unsigned char> const *)" (?DescribeSIFT@SIFT_Image_describer@features@openMVG@@QEAA?AV?$unique_ptr@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@U?$default_delete@V?$Scalar_Regions@VSIOPointFeature@features@openMVG@@E$0IA@@features@openMVG@@@std@@@std@@AEBV?$Image@E@image@3@PEBV673@@Z)
Windows-AMD64-Debug\openMVG_main_ComputeFeatures.exe : fatal error LNK1120: 10 unresolved externals
  • openmvs[cuda,openmp] x86-windows x64-windows-static pass

  • ceres[lapack,suitesparse,cxsparse,eigensparse,tools] x86-windows x64-windows-static pass

@NancyLi1013
Copy link
Contributor

@cenit
It seems that openmvg[software] cannot work on static build. Please see the above error info.

mlpack[tools] failed on x64-windows-static, which seems to add the following codes:

 if(VCPKG_LIBRARY_LINKAGE STREQUAL "static")
        file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/bin" "${CURRENT_PACKAGES_DIR}/debug/bin")
 endif())

Besides these two features, others work fine now.

@NancyLi1013 NancyLi1013 removed the requires:testing Needs tests added before merging label Aug 26, 2020
@cenit
Copy link
Contributor Author

cenit commented Aug 26, 2020

OpenMVG software does not work on static CRT build, I know that and fixes are going on to make it work in the future (not for this PR).
I will explicitly disable them if required by the user, so that the error you encountered does not happen anymore.
I will also fix the mlpack regression.
Thanks for all the hard work in tests!!

@JackBoosY
Copy link
Contributor

@cenit If this feature is not support static build currentlly, please add an error message to portfile.cmake.

@cenit
Copy link
Contributor Author

cenit commented Aug 26, 2020

@JackBoosY @NancyLi1013 I added an error message in OpenMVG in case "software" feature is requested in a CRT-static triplet and I should have fixed the mlpack "tools" feature regression

Please let me know, as always, if you have any other review or question

ports/suitesparse/CONTROL Outdated Show resolved Hide resolved
@NancyLi1013
Copy link
Contributor

LGTM now.

mlpack[tools] can work fine and openmvg[software] is as expected.

This PR can be tagged as reviewed once all CI checks have passed.

@NancyLi1013 NancyLi1013 added the info:reviewed Pull Request changes follow basic guidelines label Aug 27, 2020
@BillyONeal BillyONeal merged commit 0a506f7 into microsoft:master Aug 28, 2020
@BillyONeal
Copy link
Member

@NancyLi1013 You don't need to wait for CI to pass before marking reviewed; we aren't going to merge it red :)

@BillyONeal
Copy link
Member

@cenit Thanks for your EXTENSIVE contribution!

I don't claim to truly understand everything in here but I don't see any red flags and @strega-nil already signed off. Merged!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
category:port-bug The issue is with a library, which is something the port should already support category:port-feature The issue is with a library, which is requesting new capabilities that didn’t exist info:reviewed Pull Request changes follow basic guidelines
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[qt5-base:x64-osx] build failure
8 participants