Compiling using native windows Visual Studio compiler

[Tripphippie]

I'm attempting to build Essentia using Visual studio and native windows tools. I'm not a linux guy and I don't want to setup cygwin and all that nonsense.
Would someone be so kind as to send me a copy of essentia_algorithms_reg.cpp that I could use?
I'm going to replace the python with something else and I need a template to know what to output. If anyone is interested in native windows builds I might be able to upstream my changes. Also what is up with the RogueVector class. The methods that access private members of std::vector won't compile but I think with swap semantics and the c++11 compiler features you shouldn't have to use the rogueVector at all.

[dbogdanov]

Sure, here is the file, and your contributions are more than welcome.

[Tripphippie]

Thanks so much.
Any ideas on what to do with RogueVector? It's basically a hack and it won't compile in this version of VCC. I'm almost convinced I can remove it completely but I need to run some tests to be sure.

[dbogdanov]

Try changing back to the the old code of RogueVector for WIN32 which is now commented.

[q-depot]

I've been trying to compile with VS for months, still can't get it to work, for the RogueVector I've replace these functions:

template
void RogueVector::setData(T* data)
{
T* this_data = this->data();
this_data = data;

// this->_M_impl._M_start = data;
}

template
void RogueVector::setSize(size_t size)
{
this->resize( size );
this->shrink_to_fit();

// this->_M_impl._M_finish = this->_M_impl._M_start + size;
// this->_M_impl._M_end_of_storage = this->_M_impl._M_start + size;
}

@dbogdanov is there any chance somebody can look at a proper VS solution? seems many people would like to use it on windows and to be honest I don't know anybody using codeblocks.

thanks

[Tripphippie]

@q-depot Does that work? I may give your implementation a go.
@dbogdanov That was the first thing I tried to do. The implementation has changed and that code doesn't compile anymore in VS.
If I understood what the RogueVector was trying to accomplish then I think I can come up with a good implementation for Visual Studio. I think the swap constructor does what RogueVector is trying to do which is avoid a copy.

[q-depot]

I've managed to compile everything in VS and build a static library, but the library itself didn't work so I couldn't test that implementation.
Besides the roguevector there are many other issues you need to go through, namespace, includes etc.
I'm about to have another go so I'll keep you posted, but I think I'm giving up with the static library, the one I've compiled was about 1gb( osx static lib is 40mb), not sure why it gets so big.

[Tripphippie]

I have a static lib working right now and I've managed to link it to a test application but I don't have roguevector implemented. I ran my test app and it stopped on RogueVector because I put a _DebugBreak() stub in there to remind me to fix it. I'm not sure why the lib would be so big. mine is also 1.2gb. I had to pull a fresh build of ffmpeg to get stuff to link. I'm going to try your version of RogueVector next as I've spent most of today resolving libav dependencies.

[q-depot]

IMO the 1.2gb lib is not usable, I also reckon it should be compile without the dependencies.

[Tripphippie]

I think it might have something to do with the dependencies why it is so large. I'm going to try to make it into a DLL after I get this static lib working. Then I might be able to see why it is so large. I think we might have a situation where the entire cygwin c-runtime is linked into this static lib. All the dependencies would need to be recompiled under windows to correct the problem. Keep in mind this is under a debug build so it may shrink when all the debug junk is removed.

[q-depot]

the static library in release was about 700-800Mb so still huge.

Today I've made some progress, as I said I gave up(for now) with the static library, instead I've compiled the source straight in my app, I've stripped out most of the library and all the dependencies.
I had to fix a few issues mostly with the cmath library, and I uncommented the roguevector implementation which seems to be working.
The current implementation is not fully working but at least I can build and compute(badly) the mfcc.

[MGarvinNYC]

I'm glad to hear that there's an effort to build a Windows/VS version!

Re the size (800MB?!): There is supposedly a way to cross-compile a Windows lib from Linux. I don't think that the library will do much good in itself, as it's not VS-compatible, and supposedly has no Python bindings. But perhaps it would give an indication of size to expect in a reasonable VS build.

[dbogdanov]

800mb is so huge! 😮 The cross-compiled library using mingw-w64 on Linux should be no more than 20mb.

[dbogdanov]

@q-depot Your fix for RogueVector using data(), resize() and shrink_to_fit() looks dangerous. If the size of the vector was smaller before setSize, I would expect resize() to mess up with the data in the data() pointer.

[q-depot]

I'm using the commented code you left there. It seems fine

[q-depot]

I've found another issue while building in debug, the static free method in algorithfactory.h seems in conflict with the reserved one, I had to rename it to something like this:

static void freeAlgo(BaseAlgorithm* algo) {
delete algo;
}

it would be good to fix it in the master branch.

[Tripphippie]

Ok so a little update on this. I have a static lib working mostly. I have a dll that is much much smaller (7mb) but is missing a bunch of ESSENTIA_API export tags, it will grow as I add missing things. I can't get taglib to work at all since the dll in the package folder is for gcc or something. So MetadataReader is missing from the lib. I'm not actually sure at this time how I got taglib working in the static version of the library, strange. I'm currently decorating missing functions but it is impossible to know which ones are missing until I link with an test exe so I may not find them all.

EDIT:
Oh man! I think I just stumbled across some dark hole of dllimport and headers and inlining and dlls. I may have to refactor this whole thing a little. I get link errors on global variables and statics.

[q-depot]

what errors are you getting? I had similar problems with kissfft

[Tripphippie]

Taglib won't link because the lib file is missing and I'm too lazy to compile the whole thing. Even if it existed linker would be looking for the wrong name mangling scheme as the DLL is full of gcc names. I didn't have problems with kiss as the source is there. I just pulled the source into my project. I'm having problems getting certain symbols to export in the DLL. I'm going to have to get all the test programs to compile and link and then maybe I have a chance of getting a full complete DLL. Problem is that gcc exports most things by default while MSVC requires templates to be instantiated. I had to add some template instance declarations to force the template instance to be created and exported. Now I'm stuck on getting statics and globals to export properly. Here's my current error list.

2>BasicExtractor.obj : error LNK2001: unresolved external symbol "int essentia::activatedDebugLevels" (?activatedDebugLevels@essentia@@3HA)
2>BasicExtractor.obj : error LNK2001: unresolved external symbol "int essentia::debugIndentLevel" (?debugIndentLevel@essentia@@3HA)
2>BasicExtractor.obj : error LNK2001: unresolved external symbol "class essentia::Logger essentia::loggerInstance" (?loggerInstance@essentia@@3VLogger@1@A)
2>BasicExtractor.obj : error LNK2001: unresolved external symbol "public: static class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const essentia::standard::Algorithm::processingMode" (?processingMode@Algorithm@standard@essentia@@2V?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@B)
2>BasicExtractor.obj : error LNK2001: unresolved external symbol "private: static class essentia::EssentiaFactory<class essentia::standard::Algorithm> * essentia::EssentiaFactory<class essentia::standard::Algorithm>::_instance" (?_instance@?$EssentiaFactory@VAlgorithm@standard@essentia@@@essentia@@0PAV12@A)

[MGarvinNYC]

I'm curious about how this is going. Stalled for the holidays?

[Essentia]

Hello,

Somehow my company email and name (ESSENTIA) got linked to this post ....
As far as I could read on the post, it's related music files?

---

Gabor Simonsich

Desarrollador de Negocios
Business Developer
Essentia

Cel: +598 95 120021
[email protected]
www.essentia.com.uy

On Wed, Dec 30, 2015 at 4:22 AM, MGarvinNYC [email protected]
wrote:

I'm curious about how this is going. Stalled for the holidays?

—
Reply to this email directly or view it on GitHub
#330 (comment).

[dbogdanov]

@Essentia Yes it is. You got referred accidentally, sorry for that.

[dbogdanov]

I've found this for building taglib for Visual Studio: http://stackoverflow.com/a/5365962

[Tripphippie]

This has come to a halt for the time being. Holidays is over and the skunk-works time I had at work has expired with it. If I disable the debug macros I can get it down to 1 variable not linking properly. I'm not sure how to fix this problem.I think this link is relevant. http://stackoverflow.com/questions/17614172/c-template-singletons-in-a-dll
So I think I need to move some stuff around in the headers and have templates in cpp files which is weird.
last error....
2>BasicExtractor.obj : error LNK2001: unresolved external symbol "private: static class essentia::EssentiaFactory * essentia::EssentiaFactory::_instance" (?_instance@?$EssentiaFactory@VAlgorithm@standard@essentia@@@Essentia@@0PAV12@A)

[MGarvinNYC]

Sorry to hear that the real world is intruding! :-) It sounds like you're so close, too.

Perhaps Dmitri or others could shed some light on that last problem?

[Tripphippie]

I cracked some sort of solution but I don't like it. None of my other solutions have worked except this. If I compile the DLL with #include "algorithmfactory_impl.h" and then comment that line out when I include it in my executable it works. #ifdef'ing the line out doesn't work for some reason. neither does moving the code from algorithmfactory_impl.h to a cpp file and only including it in the dll project. I think I might try moving the code to a "algorithmfactory_private.h" and making a copy to use when importing the DLL. Basically the templated function is expanding in both compilation units and then failing to link because it is in a different compilation unit, it needs to only include the basic interface and not any of the template function bodies. Windows DLLs differ from SOs on linux in that they don't invoke the linker when binding. you can end up with the same templated classes in both objs and statics become a problem.

edit: nevermind. I'm still not sure why this works but I moved the template instantiation to the header with the extern keyword and it works without commenting out lines selectively. Now to actually try to use some algorithms and probably sort out some more issues since this is like a basic basic link test that is working.

double edit:
Seems all the win32 3rd party dlls don't match the libs that are right next to them. Need to get win32 libs for ffmpeg.

[mathieu-b]

Hi
I am also trying to have Essentia up and running under Visual Studio 2015.
Any updates about how to possibly implement RogueVector (or an equivalent) under VisualStudio?
@wackou / @Tripphippie / @dbogdanov ?

Are there any resources detailing the RogueVector specifications and usage?

Thanks

[wackou]

the RogueVector initially was a hack in C++98 to support non-owning vector and allow a nicer api than passing (float* array, int size) as arguments to all functions.

A better solution (since C++11) would be to use a std::array (http://en.cppreference.com/w/cpp/container/array) which has the efficiency of passing raw pointers around but has a nice object-oriented api. That involves changing a bit of code, but it probably would be worth it in the long run. Alternatively, one could define an essentia::Array type, use it everywhere in the api, and typedef it to a given type (std::array for instance)

For a quick solution, though, I maybe looking at the source code of roguevector (https://github.com/MTG/essentia/blob/master/src/essentia/roguevector.h#L77) and see how it could be adapted to the (I imagine) new libstdc++ from the latest Visual Studio would be faster.

[mathieu-b]

Ok, thanks for the clarification and suggestions @wackou :)

[maxgraf96]

@moebiussurfing hey! Did you get it running with OF? I'm currently trying to integrate essentia into JUCE with Visual Studio, but I'm also getting linker errors.

[moebiussurfing]

hey @maxgraf96, I do not have tried yet to compile by myself.
Maybe I'll try someday... But in my case it could be hard, bc I should need to update the OF addon too...and I don't know if I have enough skills or too much time.

BTW if somebody can share the dll/lib would be nice!

[maxgraf96]

Alright, if you ever do let me know, I'll post some updates here in case I can get it to work! But for now it looks like it's pretty much impossible with VS...

[q-depot]

to build in VS you need to:

• fix some errors in the c++ code, it should be just a bunch of exports and a few minor changes
• update waf
• update the wscript

all these changes should be in my fork: https://github.com/q-depot/essentia/commits/python_win

The build instructions mentioned by @dbogdanov are a good starting point

@moebiussurfing some of your errors are missing dependencies, I compiled my library without that stuff, just use the minimum dependencies, there should be an option for "lightweight" build.

@dbogdanov would be great if you could check at those changes in my repo and merge

[hosaka]

@q-depot Any hits as to what should be done with the RogueVector?
Neither the OS_WIN32 elif nor the commented out OS_LINUX version compile. I am using the MTG/essentia repo and the python_win branch, which looks to be the as your fork.

Also, were you able to reduce the static lib size in the end? 800mb is not useable for the hardware I am planning to run it on.

[KyrillosL]

Hi @q-depot & @hosaka
I'm also struggling at compiling an Essentia.lib (or a .dll) that could be used later with Visual Studio (2017) under Windows 10.

When trying to compile with only one algorithm (MonoLoader), it still compiles 38 algos, including the streaming and freesound algorithms. Here is my command in the x64 command prompt :
python3 waf configure --mode=release --build-static --lightweight=libav,libsamplerate --fft=KISS --msvc_targets="x64" --include-algos=MonoLoader --static-dependencies

It doesn't find my 3rd parties dependencies, even if they are in the folder.

Thanks!

Edit : Installing pkgconfiglite through chocolatey solved my problem.

[KyrillosL]

Hi,
I managed to get a .dll and a .lib, but I can't link to my project under visual studio.

After setting ESSENTIA_EXPORTS to 0 in the config.h file as recommended previously, I get this unresolved external symbol error :
LNK2001 unresolved external symbol "__declspec(dllimport) public: static class essentia::standard::Algorithm * __cdecl essentia::EssentiaFactory<class essentia::standard::Algorithm>::create(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class essentia::Parameter const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class essentia::Parameter const &,class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &,class essentia::Parameter const &)" (__imp_?create@?$EssentiaFactory@VAlgorithm@standard@essentia@@@essentia@@SAPEAVAlgorithm@standard@2@AEBV?$basic_string@DU?$char_traits@D@std@@V?$allocator@D@2@@std@@0AEBVParameter@2@0101@Z)

I've tried with different flags (debug/release/reduced dependencies), but I can't manage to get it linked properly.
Does anyone found a workaround ?
Thanks !

[VitaMusic]

I'm interested in well working *.lib and *.dll of Essentia compiled for Visual Studio 2019, Windows 7, x64
is it possible to use MKL when compiled ?

[moebiussurfing]

yes, should be nice that some successful compiled files can be shared to download...

[imiskolee]

@k

Hi @q-depot & @hosaka I'm also struggling at compiling an Essentia.lib (or a .dll) that could be used later with Visual Studio (2017) under Windows 10.

When trying to compile with only one algorithm (MonoLoader), it still compiles 38 algos, including the streaming and freesound algorithms. Here is my command in the x64 command prompt : python3 waf configure --mode=release --build-static --lightweight=libav,libsamplerate --fft=KISS --msvc_targets="x64" --include-algos=MonoLoader --static-dependencies

It doesn't find my 3rd parties dependencies, even if they are in the folder.

Thanks!

Edit : Installing pkgconfiglite through chocolatey solved my problem.

did you successful compile it on windows use waf?

[goloskokovic]

@imiskolee

#330 (comment)

[dbogdanov]

The python_win branch has been updated further, cherry-picking from @q-depot and with other modifications.

[imiskolee]

@imiskolee

#330 (comment)

thanks, hte prebuilding lib includes all third party depencies ?

[yyf]

@dbogdanov do you know if anyone gets Essentia running in streaming mode on Windows 10 using VisualStudio-built lib or dll? Not sure what's the latest with #204 ?

Thanks.

[goloskokovic]

@yyf
I can confirm running streaming MFCC example: https://github.com/goloskokovic/essentiaMSVC

[yyf]

@goloskokovic thanks for your example! My python waf configures successfully but python waf fails at
[260/260] Linking build\src\essentia.dll
There are warning messages like
cl: Command line warning D9025: overriding /w wutg /W3
and with verbose output (waf -v) it complains about
... warning C4661: ... no suitable definition provided for explicit template instaniation request
and lots of messages in red, for example,
... \algorithm.cpp.1.o is created more than once... The task generators are..., If you think that this is an error, set no-errcheck_out on the task instance
Build failed
-> task in _essentia failed with exit status 2: {task ...: cxx essentia.cpp -> essentia.cpp.1.o}
Lastly with more verbose (waf -v -v) the last red message complains about
File "waf", line 165 in <module>... Scripting.waf_entry_point(cwd, VERSION, wafdir)...
Did you see this before when you first built the MFCC example? My python waf config step passes except there are three python3 items not found: python-config, library python3.7in $prefix/libs, and library python3.7m in $prefix/libs.

Thanks.

[goloskokovic]

@yyf
Just use prebilt essentia.lib provided in that repo with headers

[yyf]

@goloskokovic I tried to build the MFCC example with your prebuilt essentia.lib but got lots of linker errors (716):
LNK2038 mismatch detected for '_ITERATOR_DEBUG_LEVEL': value '0' doesn't match value '2' in ...obj
LNK2005 ... already defined in essentia.lib...
...
LNK1120 90 unresolved externals
Maybe I miss something?

[goloskokovic]

Be sure to link 3rdparty libs
#330 (comment)

https://stackoverflow.com/questions/7668200/error-lnk2038-mismatch-detected-for-iterator-debug-level-value-0-doesnt

In short you are probably mixing release and debug

[yyf]

@goloskokovic Finally got it compiled. My MFCC (and other examples) execute fine without error, but they don't output any result/file for some reason. It doesn't behave like the MFCC example in the prebuilt extractor binaries, where it logs "...start processing... writing results to file..." Ever seen this before?

[goloskokovic]

@yyf
Try running it from the Command Prompt: MFCC.exe mfcc.wav mfcc.yaml

[yyf]

@goloskokovic Works now. Had some typo in my debug setting. Thanks again.

[WindowsNT]

So, do we have some static library for x64 and x86 for Visual Studio?
Or (better), a DLL.

[moebiussurfing]

So, do we have some static library for x64 and x86 for Visual Studio? Or (better), a DLL.

maybe could use this one?
#330 (comment)

[andreaagostini]

Hi, I'm trying to build a minimalistic version of Essentia, with no dependencies, on Visual Studio 2022. I can't use dbogdanov's build because I need it to be built with /MD (Multithreaded DLL) rather than /MT (Multithreaded static library). I'm stumbilng onto the RogueVector thing, too. I understand that some of you (@dbogdanov, @q-depot) have found a working fix for this, but I must be missing something as I'm not finding it in your repos... Any hint about this?

[moebiussurfing]

Hi, I'm trying to build a minimalistic version of Essentia, with no dependencies, on Visual Studio 2022. I can't use dbogdanov's build because I need it to be built with /MD (Multithreaded DLL) rather than /MT (Multithreaded static library). I'm stumbilng onto the RogueVector thing, too. I understand that some of you (@dbogdanov, @q-depot) have found a working fix for this, but I must be missing something as I'm not finding it in your repos... Any hint about this?

there where some python projects to compile if I not remember bad.
There's some big file (700mb) shared around here too, a lib to download or something.
I do not tried again, but i'm still interested on this.
if you get success post it here please.

[jamiebullock]

Hi, I'm trying to build a minimalistic version of Essentia, with no dependencies, on Visual Studio 2022. I can't use dbogdanov's build because I need it to be built with /MD (Multithreaded DLL) rather than /MT (Multithreaded static library). I'm stumbilng onto the RogueVector thing, too. I understand that some of you (@dbogdanov, @q-depot) have found a working fix for this, but I must be missing something as I'm not finding it in your repos... Any hint about this?

@andreaagostini Out of interest, how are you setting the CRT config? The only way I can see to do it is via CXXFLAGS, something like set CXXFLAGS="/MD" && waf configure