Proposal: Adding Soundfont linked modulators

[jjceresa]

1)Overview.
Linked modulators is a SF specification (rev 2.04 - 9.5.4) that extends the modulators capabilities of classic non-linked modulators. A non-linked modulator have 2 MIDI event inputs sources and a generator destination.
Linked modulators allows a soundfont designer to think and design a "complex" modulator tailored to the need of a particular sound articulation during playing performance. This complex modulator can have any MIDI events inputs and a generator destination. Its internal structure is free. This way the soundfont designer can easily substitute a complex modulator by another. Also the musician at performance time can select a sound articulation or another on the fly without express need to select another preset (at least in most situation). In others words "complex" modulator helps to think, design, and play sound articulated instruments in real time.

2)Implementation in actual branch linked-modulators
This brings full SF linked modulators to fluidsynth

2.1) Checking linked modulators at soundfont loading time.

• Identical complex modulators in the same zone are naturally removed (as it is the case for non-linked modulators).
• Any invalid complex modulators are removed.

2.2)At noteon time.
The concept of modulators precedence (instrument zone(local/global) , preset zone(local,global)) and the additive effect of presets modulators works also for complex modulators the same way it works for non-linked modulators. So the behaviour is straightforward in regard to actual know SF specifications.

2.3)Real time modulation on CC change is implemented.

Note that taking account of linked modulators at noteon time (2.2) or during CC change (2.3) is done without lost of performance in regard to non-linked modulators (i.e the overload is very low).

3)Soundfont editor abilities
This is a major issue. Few free soundfont editor have linked modulators capabilities. Polyphone editor actually has partial capabilities (probably due to misunderstanding about SF specifications).
I hope that editor developer would add this capabilities !.

Opinions, comments are welcome.

[derselbst]

I do believe that it's a useful addition to fluidsynth. I am however having trouble to get used to it. Can you provide any soundfont(s) that make use of this? Or do you have a practical example how linked modulators could be useful?

[mawe42]

I am quite excited about this feature, but also have a little trouble to get my head around it. It seems like it would be very useful for real-time playback, especially if Fluidsynth is used with very expressive musical instrument MIDI controllers like breath controllers or also the project I am working on. But I haven't read the spec yet.. will do that in the next few days.

You say there are "few free soundfont editors" with support for linked modulators. Do you actually mean "few" (in which case there would be at least one), or "none"? I know you say that Polyphone has partial support... but is this enough to actually try out the capabilities of linked modulators? And if not.... how do we test and experiment with this feature if there are no editors we can use?

[jjceresa]

Can you provide any soundfont(s) that make use of this?

No, unfortunately. The only soundfont i had made is for code testing purpose and have really no sound interest. Anyway i can provide this ofc.

Or do you have a practical example how linked modulators could be useful?

Following figures is an attempt to show the interest of linked modulators.

I know you say that Polyphone has partial support... but is this enough to actually try out the capabilities of linked modulators?

Polyphone allows all cases illustrated by fig.2 (i.e product of any modulators). Unfortunately Polyphone doesn't support case illustrated by fig.3 (the most interesting imo). Fig. 3 case allows to localize the effect of some modulators on a generator by grouping these modulators into one "complex" modulator. I think that this helps to design effect in a hierarchically way.
For example in fig 3:

• During playing, it is easy to cancel (or weight) the effect of mod 1_2_3 by setting CC5 to 0. At the same time, if CC5 is used to control the effect of another modulator (complex or not) connected to another generator, we have a way to modulate distinct generators using the same CCs palette (CC1,CC2,CC3,CC4 ) but producing different effect selected by CC5 at appropriate time in the song.
  To get this result, at design time we need only to set CC5 on mod 3 src2. To get the same result with unlinked modulators , we need to set CC5 on src2 of each modulator in the group.
• At edition time, it is easy to disable a group of modulator mod (i.e1_2_3) by setting mod 3 amount to 0. With unlinked modulators, we need to set amount value of each modulator to 0. Unlinked modulator is rather a flat model compared to linked modulators.

how do we test and experiment with this feature if there are no editors we can use?

We can do fig. 2 case with Polyphone. (the main developer is recently aware of fig.3 case lack).
At the time of starting writing this feature i wasn't aware of this limitation. I had discovered this later.
I haven't yet take time to search after other soundfont editor.

Note: As providing attached files is no more possible since GitHub is in the hands of Microsoft, /doc/linked_modulators_figures.pdf have been added in last commit on branch linked-modulators.

[derselbst]

@mrbumpy409 FYI

[bengan]

Note: As providing attached files is no more possible since GitHub is in the hands of Microsoft,

The maximum size for files is 25MB and the maximum size for images is 10MB.

https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/

[derselbst]

Note: As providing attached files is no more possible since GitHub is in the hands of Microsoft,

The maximum size for files is 25MB and the maximum size for images is 10MB.

The file in question is 101KB in size. Presumably this is too small in 2019.

[mawe42]

Seems to work for me: linked_modulators_figures.pdf

[jjceresa]

@elementgreen

Linked modulators is a SF specification (rev 2.04 - 9.5.4) that extends the modulators capabilities of classic non-linked modulators. Actually we have a fluidsynth branch that support full linked modulators.
Unfortunately soundfont editor doesn't support this. However, as far a soundfont editor already have modulators editing capabilities, taking account of linked modulator is easy to do with little effort.

I wonder if there is perspective to get swami with linked modulator edition capabilities ?

[derselbst]

Don't think elementgreen is doing any programming in mid-term... at least according to what he said last time I spoke to him.

[jjceresa]

Don't think elementgreen is doing any programming in mid-term...

I see..... Hope that elementgreen is fine. Talking about swami, I would like to get a source version that runs on Windows. I have already successfully compiled it from source (from swami-2.0.0.tar.gz on sourceforge) . At running time a violation access happens. I suspect that swami-2.0.0.tar.gz package is not the latest. I wonder which is the source package that runs successfully on linux. Thanks for any information on this.

[elementgreen]

I haven't done much on Swami in a long time and recently decided that I likely wont be doing anything with it in the future, because of moving on to other projects. The most recent Swami source code is the git repository on github. When I was building for windows, I used the same source code and cross compiled on Linux using MinGW I believe. If anyone wants to take over maintaining Swami, I'd happily pass it along to someone capable.

[derselbst]

Good to hear from you.

If anyone wants to take over maintaining Swami, I'd happily pass it along to someone capable.

@elementgreen I would like to. We also have a PR ready that adds DLS loading to fluidsynth using libinstpatch. Unfortunately the recent git version of libinstpatch is broken. Many assertions fail when loading an arbitrary dls file. I'd like to fix that as well.

** (process:8908): CRITICAL **: ipatch_sf2_voice_set_sample_data: assertion 'IPATCH_IS_SAMPLE_DATA (sample_data)' failed

** (process:8908): CRITICAL **: ipatch_sf2_voice_cache_sample_data: assertion 'voice->sample_data != NULL' failed

[jjceresa]

I haven't done much on Swami in a long time and recently decided that I likely wont be doing anything with it in the future, because of moving on to other projects.

Many thanks for your work and commitment .

[elementgreen]

@derselbst Great to hear you are interested in maintaining Swami! Let's continue the discussion via email.

[elementgreen]

@jjceresa You're very welcome. I'm glad there is the possibility of it continuing on.

[jjceresa]

You're very welcome. I'm glad there is the possibility of it continuing on.

Basically, i think that it is possible to get swami building and running on any OS (linux,windows,mac).
(As fluidsynth is actually for fluidsynth with the help of cmake).
For info, i have already compiled the source on Windows using VS10. Complete build is successful . Apart the long and laborious list of dependencies, compilation is really easy and very minor things need to be done in the source to get it without compilation error.
Actually at running time a critical warning occurs during swami_init()-> g_type_class_ref (SWAMI_TYPE_ROOT):
(swami.exe:1124): Glib-GObject-WARNING **: specified class size for type SwamiContainer ' is smaller than the parent type's IpatchContainer' class size

(swami.exe:1124): Glib-GObject-CRITICAL **: file d:\winprojet\soundfontspec\soundfontedi
tor\swami\sources\glib-2.32.0\glib-2.32.0\gobject\gparamspecs.c: line 2421: assertion
'g_type_is_a (object_type, G_TYPE_OBJECT)' failed

(swami.exe:1124): Glib-GObject-CRITICAL **: file d:\winprojet\soundfontspec\soundfontedi
tor\swami\sources\glib-2.32.0\glib-2.32.0\gobject\gobject.c: line 539: assertion 'G_IS_PARAM_SPEC (pspec)' failed

And later an access violation occurs during swami_init()->swami_control_prop_assign ()->g_param_spec_ref (pspec).

Actually i have no knowledge on gobject and libswami components to debug this !
Anyway it will be more appropriate to continue the discussion in an other place ?

[mawe42]

Polyphone allows all cases illustrated by fig.2 (i.e product of any modulators). Unfortunately Polyphone doesn't support case illustrated by fig.3 (the most interesting imo).

A quick heads-up: it looks like Polyphone now supports all three cases. I haven't actually tried it to use it yet, but adding three modulators in polyphone, linking the last two to the first produces the following Fluidsynth debug output with the current linked-modulators branch:

fluidsynth: debug: "pz:Yamaha Grand Piano/1" "iz:Yamaha Grand Piano/21" voice modulators ---------------------------------
...
mod11 Src: link                    -> dest         52, amount    111.00, flags   0, src2  90, flags2  16
mod12 Src: MIDI CC=      20 + unip -> link-dest    11, amount    222.00, flags  16, src2   0, flags2   0
mod13 Src: MIDI CC=      21 + unip -> link-dest    11, amount    333.00, flags  16, src2   0, flags2   0
fluidsynth: debug: Number of voice modulators: 12,  Total members:14

[jjceresa]

A quick heads-up: it looks like Polyphone now supports all three cases.

Yes, Polyphone author added these on my request. Unfortunately I cannot try it because installation of recent Polyphone version are now refused on Windows XP ! :).

I haven't actually tried it to use it yet, but adding three modulators in polyphone, linking the last two to the first produces the following Fluidsynth debug output with the current linked-modulators branch:

Actually, this is for printing purpose only. This is displayed by fluid_defsfont.c-line 1175 - fluid_zone_print_voice_mod(), for debug build only. This way we have a feedback of all modulators (simple or complex modulators) in the current voice for this preset zone pz:Yamaha Grand Piano/1 and instrument zone iz:Yamaha Grand Piano/21.

Note when editing a complex modulator in Polyphone, index ordering of any complex modulator's member should be free (soundfont spec). In other words it doesn't matter if the ending member (the one connected to a generator (i.e 52) ) is before or after others member.

However, inside fluidsynth, soundfont complex modulator are reordered by the soundfont loader.
fluid_zone_print_voice_mod() displays modulator using this internal ordering in voice's mod table:
-all complex's member are at consecutive index (i.e 11,12,13).
-the 'ending modulator' (the one connected to a generator) is always the first member in the list (mod11).
-other members follows (mod12,mod13).

member count is the number of members inside this complex modulator (i.e 3).
fluidsynth: debug: Number of voice modulators: 12...... shows that there is 12 modulators in this voice (probably 11 simple modulators (mod00 to mod10) and 1 complex modulator (mod11,12,13).
fluidsynth: debug: .... Total members:14 is the total number of modulators in voice.