Timothy Baldridge
patreon
This one weird trick can save you dozens of GB
🕑 Added 2020-08-09 02:06:21 +0000 UTCI couldn't resist the troll title on this one. A few weeks ago a developer in our Discord (thanks Unnoen!) pointed me at a new feature in recent versions of Windows 10. It seems they finally came up with a mostly decent version of disk compression that works quite well. Disk compression has been in windows since the Win 98 days, but it's always been pretty crappy. The compression algorithm, was slow, and it had to re-compress files whenever they were modified.
This new compression system for Windows is a bit better. Firstly, it uses LZX compression which gets good compression rates for minimal decompression CPU overhead. Secondly the compression only happens once, as a batch job. You compress the file, and later if you modify the file it becomes uncompressed. This means it works great for files that will be read only, like most game assets. And finally this compression happens at a pretty low level in the OS driver stack, meaning the OS is free to optimize the code, use multiple cores, and pre-fetch data.
As of the latest version of Wabbajack (2.2.0.0) we offer a config option (disabled by default) that will compress your entire modlist using this feature. While it's true decompression isn't free, in my tests I've seen little impact in load times, most likely because Bethesda games are so single-threaded anyways, that you end up having several cores free to do decompression. And what does this gain us? Install sizes that are about 60% of the original install. So a 100GB modlist will be about 60GB with compression.
We don't enable this feature (seen in the screenshot) by default because it is quite heavy on the system. Every file has to be compressed, so expect your CPU cores to be leveraged heavily during install. But that's the tradeoff, slightly slower install times in exchange for smaller disk sizes, and perhaps faster load times on super slow storage (like HDDs). There's also a slight chance this could cause bugs in the install, so use it at your own risk, but these operations work at such a low level that it's unlikely to be noticed by any games we support.
