VST/(i) Debug message.

[mikobuntu]

Hi ,

I always get a debug ( non fatal and has been happening since the 0.4.x series ) message whenever I load a VST instrument or effect into LMMS :- " failed getting shared memory "

The debug message originates from include/RemotePlugin.h:1161: and include/RemotePlugin.h:1180:

What I would like answered:- Is shmem actually being allocated to the VST's?

Is there an understandable way to show what ( if any ) shared memory is being allocated?

I have had a brief look at '''icps''' and also searching the pid of the RemoteVstPlugin.exe process through cat /proc/*/maps, but I'm a bit unsure of what I'm looking for.

thanks Mikobuntu :

[tresf]

IIRC this is the software's way of telling you that it can't use QT's implementation of shared memory blocks between processes.

The offending code is here: RemotePlugin.h#L1160

@tobydox is our resident expert at this stuff, perhaps he can explain more and explain if this is ever even possible with a VST plugin or not.

[mikobuntu]

@tresf 👍 thanks !

[tresf]

Probably not related, but some upstream convo at least...

https://bugreports.qt-project.org/browse/QTBUG-27765
https://bugreports.qt-project.org/browse/QTBUG-5123

[curlymorphic]

Ive been looking into this.

What I would like answered:- Is shmem actually being allocated to the VST's?

yes. I knew it was, because the audio buffer is being passed successfully (vsts work). but i wanted to dig deeper to see what is happening and why we are getting the error message.

Is there an understandable way to show what ( if any ) shared memory is being allocated?

I have taken the offending method above and put some debugMessage calls in.

void RemotePluginClient::setShmKey( key_t _key, int _size )
{
    debugMessage( "setShmKey\n " );

#ifdef USE_QT_SHMEM
    debugMessage( "Qt shareed mem\n" );
    m_shmObj.setKey( QString::number( _key ) );
    if( m_shmObj.attach() || m_shmObj.error() == QSharedMemory::NoError )
    {
        m_shm = (float *) m_shmObj.data();
    }
    else
    {
        char buf[64];
        sprintf( buf, "bbbb failed getting shared memory: %d\n", m_shmObj.error() );
        debugMessage( buf );
    }
#else
    debugMessage( "not Qt shmem\n" );
    if( m_shm != NULL )
    {

        char buf[64];
        sprintf( buf, "Detatching %p \n ", m_shm );
        debugMessage( buf );
        shmdt( m_shm );
        m_shm = NULL;
    }

    // only called for detaching SHM?
    if( _key == 0 )
    {
        return;
    }

            char buf[64];
            sprintf( buf, "requesting memory  : %d\n ",_size );
            debugMessage( buf );
    int shm_id = shmget( _key, _size, 0 );
    if( shm_id == -1 )
    {
        debugMessage( "aaa failed getting shared memory\n" );
        char buf[64];
        sprintf( buf, "size : %d\n",_size );
        debugMessage( buf );

    }
    else
    {
        m_shm = (float *) shmat( shm_id, 0, 0 );
        char buf[64];
        sprintf( buf, "shmem addr : %p\n", m_shm );
        debugMessage( buf );
    }
#endif
}

When loading Classic delay, a 2 in 2 out effect
http://www.vst4free.com/free_vst.php?id=1874

I get the following in my terminal

RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: requesting memory  : 4096
 RemotePlugin::DebugMessage: shmem addr : 0xf759e000
unique ID: 6f75
RemotePlugin::DebugMessage: inputs: 2  output: 2
RemotePlugin::DebugMessage: creating editor
RemotePlugin::DebugMessage: editor successfully created
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: Detatching 0xf759e000 
 RemotePlugin::DebugMessage: requesting memory  : 4096
 RemotePlugin::DebugMessage: aaa failed getting shared memory
RemotePlugin::DebugMessage: size : 4096
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: requesting memory  : 4096
 RemotePlugin::DebugMessage: shmem addr : 0xf759e000

This is showinging setShmKey being called 3 times. once on creation, once to dealloc and a thrid time to reallocate.

The important bit being that shmem is being allocated with a size of 4096 and an address of 0xf759e000.

I tried this with many pluging all with simular results. they were all 2 in, 2 out

I tried http://www.voxengo.com/product/span/ with 8in 8 out

RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: requesting memory  : 4096
 RemotePlugin::DebugMessage: shmem addr : 0xf759e000
unique ID: 4vo6
RemotePlugin::DebugMessage: inputs: 8  output: 8
RemotePlugin::DebugMessage: creating editor
RemotePlugin::DebugMessage: editor successfully created
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: Detatching 0xf759e000 
 RemotePlugin::DebugMessage: requesting memory  : 10240
 RemotePlugin::DebugMessage: aaa failed getting shared memory
RemotePlugin::DebugMessage: size : 10240
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: not Qt shmem
RemotePlugin::DebugMessage: requesting memory  : 16384
 RemotePlugin::DebugMessage: shmem addr : 0xf7581000

with more channels , 4 times as many, we request an area of memory 4 times larger.
also the final pointer for the larger buffer is different to the initial buffer pointer. one point of not is that adjusting my period size made no difference to the allocated buffer size.

In conclusion memory of the correct size is being allocated. I feel this is a bug of an incorrect error message. any thoughts?

[tresf]

Terrific research. This may explain a bit why VST preset previews were so quickly crashing the software (thus leading to the disabling of previewing VST presets).

I'll look at your findings a bit more in detail this week. Thanks for the investigation.

[tresf]

So I've re-read your analysis on this and you believe this is fine then?

I tend to agree the message should be more clear as to what is "failing" but furthermore, why isn't it ever using QT shared memory to begin with?

[curlymorphic]

After some more research RemotePlugin.h l39

#ifdef LMMS_BUILD_WIN32
#define USE_QT_SEMAPHORES
#define USE_QT_SHMEM
#endif

It would appear that Qt Shared mem is used only on win32 builds. I am not aware of anyway of attaching a debugger to this in a virtual machine.

So I've re-read your analysis on this and you believe this is fine then?

Yes. I feel the shmem is working fine.
The shmem is the audio buffers in/out, if this did not work we would have no audio from any vst. But after stepping thru this code many times I am 100% that the shmem is created and accessed correctly by both the host and the plugin.

[curlymorphic]

I tend to agree the message should be more clear as to what is "failing"

Im sure i can change it to not show the error on memory deallocation, thats that part that i feel is a bug.

I didnt want to implement this until it is agreed that this is the actual bug.

[tresf]

It would appear that Qt Shared mem is used only on win32 builds. I am not aware of anyway of attaching a debugger to this in a virtual machine.

Should we use it for all platforms? It seems rather strange to never enable it and then write an error saying it is not enabled. :)

-Tres

[curlymorphic]

The error message is failed getting shared memory not QtSharedMem.

Should we use it for all platforms?

I dont know is the real answer. My gut feeling is to leave it as it is, My reason for thinking like this is, Why would someone put in all the effort to use different shared memory implementations on different platforms if this was not needed,

[tresf]

Why would someone put in all the effort to use different shared memory implementations on different platforms if this was not needed,

Well, apple for example has very low limits (sorry for the dated server article here). My understanding is that Qt's method improves this but if it adds additional overhead on Linux I suppose it is better without it.
http://support.apple.com/en-us/HT4022

But I really don't know enough to be able to say.

The error message is failed getting shared memory not QtSharedMem.,

Ah... Sorry, I read the wrong part (mistook aaaa for bbbb).

-Tres

[tresf]

That article had limits that are misleading... Here's a better one...

"The default configuration for shared memory on a Mac is best described as "minimal", and is the same on every Mac, regardless of the amount of total memory or the number of processors. These instructions tell you how to alter the shared memory configuation." -2010 Spy Hill Research

http://www.spy-hill.com/~myers/help/apple/SharedMemory.html

[curlymorphic]

Ah... Sorry, I read the wrong part (mistook aaaa for bbbb).

Now that was my fault. originally them changes and debug messages were for my eyes only. :(

[curlymorphic]

http://www.spy-hill.com/~myers/help/apple/SharedMemory.html

After reading them links, i'm still not able to advise anymore on the implications of changing to QtSharedMem

[tresf]

After reading them links, i'm still not able to advise anymore on the implications of changing to QtSharedMem

Well I don't know much... I did most of the shmem research trying to resolve this bug #703 so a lot of it is from memory and I was struggling to find a good writeup on it.

When doing my research I found that services such as PostgreSQL have historically had issues on OSX due to some really low defaults for shared memory and Toby's advice at the time was to force Apple to use QT's implementation. If Linux doesn't need it and is better without out it... no harm no foul. :)

Possibly unrelated, but this was a QtSharedMem bug I found back when doing the research...
https://bugreports.qt.io/browse/QTBUG-27765

[curlymorphic]

Would it be possible for someone to build and test a branch on a mac please ?

It contains some debug output to confirm what shmem we are using

git clone -b i1468 https://github.com/curlymorphic/lmms.git

1. build
   1.5 branch from console
   2 add a VeSTige
   3 add a vst instrument
   4 Look in console for debug output, this may take a few seconds to appear

it should look like

RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: Not Qt SHARED MEM
RemotePlugin::DebugMessage: requesting memory  : 4096
 RemotePlugin::DebugMessage: shmem addr : 0xf7e1e000
fixme:thread:GetThreadPreferredUILanguages 52, 0x33f2ec, 0x33f568 0x33f2f4
RemotePlugin::DebugMessage: inputs: 0  output: 2
RemotePlugin::DebugMessage: creating editor
RemotePlugin::DebugMessage: editor successfully created
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: Not Qt SHARED MEM
RemotePlugin::DebugMessage: Detatching 0xf7e1e000 
 RemotePlugin::DebugMessage: requesting memory  : 2048
 RemotePlugin::DebugMessage: non Qt shmem failed getting shared memory
RemotePlugin::DebugMessage: size : 2048
RemotePlugin::DebugMessage: setShmKey
 RemotePlugin::DebugMessage: Not Qt SHARED MEM
RemotePlugin::DebugMessage: requesting memory  : 2048
 RemotePlugin::DebugMessage: shmem addr : 0xf7e1e000

The important bit is Not Qt SHARED MEM or Qt SHARED MEM

p.s I have no intention of pushing a branch full of debug output.

[tobydox]

I'd also like to have a Qt-only implementation but that would break VST plugins on Linux which are launched using WINE. We need the native implementation for the communication between LMMS and the WINE-based VST host process. The Qt implementation is required/use for implementing RemotePlugins in all other cases on all other platforms. Nevertheless it's still possible that there's a problem somewhere in the implementation.

[curlymorphic]

Thanks @tobydox

[tresf]

Would it be possible for someone to build and test a branch on a mac please ?

I'd be happy to but VeSTige isn't supported on that platform yet due to a bug (AFAIK) in the winegcc compiler on that platform. #698

Can I use another plugin (i.e. Zyn) to test this instead?

[curlymorphic]

Do we support external plugings on mac? None of the native ones will use shared memory.

[tresf]

Do we support external plugings on mac? None of the native ones will use shared memory.

I don't know. What about Zyn? I always thought that was an external plugin (or perhaps just the GUI is?)

[curlymorphic]

Ive just loaded a Zyn, it does not use the code under concern.

[tresf]

Thanks. I suppose switching QtSharedMem on for Apple is moot until we get Wine finally working then.

FYI, Toby had made a branch to help this Mac + Wine effort called stable-1.0-vstbase-fix be7a82b. @lukas-w if we have the time and there's no negative side effect, this might be worth merging in to clean up branches.

-Tres

[badosu]

Are we using wine for VSTs on mac? Maybe we should create an issue for native mac VST support?

[tresf]

Are we using wine for VSTs on mac?

Yes, that is the idea.

Maybe we should create an issue for native mac VST support?

Sure unless it bears similarities in code to #416?

[badosu]

No, I guess it's very different from #416.

Well, the reason I talked about this is because wine does not work very well with OSX.

[tresf]

the reason I talked about this is because wine does not work very well with OSX.

Hmm... Is this statement from experience or research?

This is from the WineHQ project page:

Current Status (x86)
Brief: Wine works well on OSX for Intel based Macs minus a proper distribution package (Mac Application Bundle) to fully integrate with "drag and drop" and "open" or "open with" functionality. Wine also has shortcomings with desktop icon integration and execution of certain types of DOS or blended Windows/DOS programs.

This tends to suggest it would work well, no?

[badosu]

Ok, so I am probably wrong. :-)

[lukas-w]

stable-1.0-vstbase-fix […] be7a82b might be worth merging in to clean up branches.

I can't say much about the changes that were made there but if it doesn't break anything, sure.