Timothy Baldridge
patreon
Time for some performance tuning
🕑 Added 2019-08-19 22:16:10 +0000 UTCSo I got a rather large modlist installed and while testing the compilation of the modlist with Wabbajack, I realized we have a rather big performance problem.
This problem comes mostly from the way Wabbajack finds matches in the files in your MO2 and the download archives. It does this by creating SHA256 hashes of all the files. So far we only cached the hashes of data inside the archives, but we generate hashes for the rest of the files on-the-fly. This means if you have 50GB of installed mods, we have to hash through that 50GB every time you try to compile a modlist. On my (rather beefy) machine, this process takes about 15 minutes.
In addition the patching process often needs to access files inside the archives, and that extraction process is scattered across several locations of the code. As they say in software engineering: "there is no problem on earth that can't be solved with another layer of abstraction" and so the Wabbajack VFS is born.
Although the name the VFS may sound complicated is very simple: it's a cache of every single file in your MO2 folder an the contents of every archive. It's also a way to extract and access files in a well defined manner. As a programmer I can ask the VFS for files that match a certain criteria, and to allow me to read them. It doesn't matter if those files exist on disk or inside a bsa inside a 7zip the VFS knows how to stage (extract) the file and give me access. By putting all that complexity in a single place it'll help clean up the main Wabbajack code and make my life as the developer a lot simpler.
So status update: the VFS is done (as of today) and now I just need to update the compiler/installer to take advantage of it. Time to refactor some code!
