colugomusic
patreon
Plugin Sandboxing
🕑 Added 2024-08-26 16:07:42 +0000 UTCI have spent the past few days doing a bunch of research into plugin sandboxing. This is where plugins are run in a separate process from the DAW. The main advantage of doing that is that is so that if a plugin crashes or otherwise misbehaves, it doesn't take the entire DAW down with it. Instead, the sandbox process crashes, then the DAW can detect the situation and let the user know what happened and allow them to restart the sandbox if they want.
In Blockhead's case there are a couple more advantages:
Blockhead's UI is implemented using Godot and so it's using OpenGL. This is pretty rare for a DAW and so plugin developers might not be anticipating it. If they are not careful then they might end up doing OpenGL calls which interfere with Blockhead's rendering.
The code for opening up the plugin editor windows currently has to be a bit hacky because I don't have direct control over the main message loop of the application (because that's deep in the guts of Godot and not straightforward to modify.) The Linux version of Blockhead is currently suffering greatly from this problem.
Plugin sandboxing is a really complicated problem and there isn't much specific information about it online so my research has been more generally on the problem of "how do you transfer data between two real-time threads that belong to two separate processes, in a way that works on Windows, macOS and Linux?" The answers I am going with at the moment are "shared memory using the Boost.Interprocess library" and "busy-waiting with a tuned back-off mechanism". I've yet to find out how well my idea works.
It bugs me a bit that this problem is essentially a solved one (many proprietary DAWs already implement plugin sandboxing) but I still have to re-solve it myself, and it also occurs to me that the problem can be solved in a generic enough way that it would work for whatever type of sandboxing a DAW wants to do. For example, many DAWs will create a sandbox process per plugin instance like in this simple example of a project with a single track containing two effect plugins:
Another DAW might instead prefer to create one giant sandbox containing every plugin. The connections between the plugins are managed inside the sandbox. If one plugin crashes then it takes down the entire sandbox (probably with no way of knowing for sure which plugin was the one that crashed.) Then the DAW can present the user with the choice to restart the entire sandbox, or restart individual plugins. In my opinion this isn't that bad because a lot of the time it will probably be obvious to the user which plugin it was that crashed (it's probably the one that they downloaded from the internet 20 minutes ago.)
Some added complexities here are that some plugins have multiple inputs and outputs, and the DAW may want a way to inject additional input/output signals at various points in the signal chain:
This makes the inter-process communication a lot more fiddly but it's doable I think.
In my opinion, if it works then this second design is the way to go because:
It's much more efficient than creating one sandbox per plugin.
Even if you want to create one sandbox per plugin (perhaps giving the user the option), this design can still be used for that anyway, by simply creating multiple sandboxes which each contain a single plugin.
A DAW may want to do some other sandboxing scheme, for example, creating one sandbox per track, and this design is compatible with that too.
My plan at the moment is to create an open-source plugin sandboxing system so that no DAW developer has to figure this crap out ever again. I will probably try to get something working with Blockhead and CLAP first and then put it on GitHub. If I'm lucky then someone may even come along and implement VST support for me so there's my ulterior motive.
The project is going to consist of three main parts, the sandbox process, a client library for interacting with it, and a plugin scanning process. The plugin scanner already exists - the one I use at the moment for Blockhead is clap-info. This is the process which launches automatically when Blockhead starts up and scans the system for installed CLAP plugins. Plugins can even crash when they're being scanned so that's why that happens in another process.
All interaction with the sandboxing system will happen through the client library, which will encapsulate the hairy problem of communicating with the sandbox processes.
So hopefully this all works in practice. I've done IPC in the past but the really tricky part here is that the communication is happening between two real-time audio threads, so it's necessary to avoid syscalls and traditional locking mechanisms. That is actually quite a funny stipulation because in a way it narrows down the list of possible approaches to the problem so much that, as far as I know, you are left with two options for the shared memory synchronization strategy: futexes, or busy-waiting.
Futexes are really only available on Linux. On macOS they are sort-of supported, but not in any official way. On Windows, futexes exist, but they don't work across multiple processes. So that leaves me with busy waiting. And I can do my best to reduce the CPU load by implementing some kind of intelligent back-off strategy.
Additional reading:
https://ardour.org/plugins-in-process.html
Developer of Ardour on why Ardour doesn't implement sandboxing at all. I don't really agree with the arguments here but I will readily admit that I don't know what I'm talking about. Maybe once I have more experience in this area I will better recognize these stated problems. The article is quite interesting and useful regardless.
https://timur.audio/using-locks-in-real-time-audio-processing-safely
Timur Doumler's article on busy-waiting. The situation described here is a bit different from mine (his problem involves one audio thread and one UI thread synchronizing access to some shared resource, rather than two real-time audio threads hopping data back and forth between two processes in the middle of an audio callback.)
https://shift.click/blog/futex-like-apis/
Information about the availability of "Futexes" on various platforms.
Comments
eliobrid
yeah makes sense, i remember there's some difference between how arrays are allowed to be used or smthin
Colugo Music
I did use the compatibility mode to begin with but if I remember correctly I struggled quite a bit to get the waveform rendering working properly without some of the newer shading language features
eliobrid
drat, yeah that's it This is probably a lot to ask but have you considered switching to the compatibility mode / OpenGL 2.1? feel free to ignore this request. Switching would allow older devices/weak devices -that can't do OpenGL 3.3- to use blockhead. idk. Just for compatibility's sake. Let's reuse old technology. Im not knowledgeable about all the differences/incompatibilities, but i have programed with Godot and the different OpenGL options, and just visually, I'm pretty sure blockhead wouldn't need OpenGL 3.3. It could potentially be implemented with 2.1. I could be very wrong though Again, i don't want to send you down a research / debugging hell hole, so please ignore this personal request -because of the time it would take- / -just not a concern- / -not or potentially not possible-.
Colugo Music
it's whatever godot 3 uses . i think it's probably opengl 3.3
eliobrid
unrelated to sandboxing... what version of OpenGl is blockhead using? trying to run blockhead on a machine that can do OpenGL 3.1. im one of those people who's curious why it would be using a specific version :)
Beats
resolved! it was an error on my end. one very nube question, how do i reset the play marker? once it passes through my samples i cant reset it.
Colugo Music
check if there are unicode characters in the path of the files you are trying to open. I think that is an old bug that i haven't fixed
Beats
HELLO, this looks so cool. Unable to import samples though! files show up as empty when i locate them via the daw. what am i doing wrong? much appreciated
Michael West
I'm blown away you're using Godot for this, I can't wait to tell a buddy about this. That's pretty awesome. So I assume you're writing a lot of you're own plugins for it then? Oh wait what we got UPDATES?!?!?! cool! I'm about to go on a trip I'll test linux versions on my laptop.
Colin C
hey i don't have a ton to add here, just wanted to say thank you. I've got autism and this is th first daw i've ever seen that doesn't immediately overwhelm with unnecessary clutter, or just completely nonsensical menus, keybinds, and workflow. I love the clear visual design, and the wysiwyg style editing. Gives me hope i'll actually make my music one day. i'd sell a organ if it meant i could make sure you were able to reach lots of people with this.
Colugo Music
that sounds great thanks i will have a look at that!
kurt korthals
Ahhh, you're here! so you're able to chime in directly, very nice :)
Oli Larkin
Thanks Kurt... I haven't done much of this kind of thing tbh, but @Colugo might be interested to check out the WDL source code which includes some utilities for dealing with shared memory and is what is used in Reaper for plugin sandboxing AFAIK. https://github.com/justinfrankel/WDL/blob/main/WDL/shm_connection.h https://github.com/justinfrankel/WDL/blob/main/WDL/shm_msgreply.h of course that's only a small part of the problem.
kurt korthals
A more general open-source approach to this problem would be amazing, though I'm sure it's very complicated b/c it can all rest on the architecture of the DAW. You should try to connect with Oli Larkin from iPlug2, who nearly single-handedly wrote that multi-format, multi-platform, open sourced audio plugin platform in his free time. He's very nice, in europe, and knows a LOT about how all plugin formats work, so he can probably offer some insights and feedback on your ideas, as I'm sure he's thought about this before. https://github.com/iPlug2/iPlug2 https://iplug2.discourse.group/ https://discord.com/invite/7h9HW8N9Ke
Colugo Music
Ah i was wondering why Bitwig had that option. That makes sense, some related plugins may need to exist in the same running process in order to communicate. Thanks for the info!
George Napier
Fantastic discussion, even for non-developer types as myself. :) Please excuse me if you are already aware of this. Bitwig offers the option to choose the level at which you Sandbox: Individual plugins, by manufacturer, or all plugins in one sandbox. The problem with individual plugins (as I understand and have experienced it) is that some plugins want to talk to each other (e.g., SonicLab's various Bot plugins which operate in a client/server model). Individual sandboxing can cause problems in that case. By manufacturer avoids this problem since all of the plugs from a company can talk to each other, but not to plugs from other companies. (I don't think that matters much yet.) The big sandbox is the big sandbox. I have found sandboxing by manufacturer works very well. I think it's great that you are going to solve this problem and make it open source. Much luck and success to you!
ROPETIGHT
One of the major reasons Bitwig is my main squeeze
