sfmlab
patreon
SFMLab Migrations 2020
🕑 Added 2020-03-16 16:19:03 +0000 UTCHi everyone,
It's been a busy month, but we're almost out of the woods. You may have noticed some major changes and severe service disruption over the past few weeks. To those who asked, my answers have been terse and vague. The banner on the homepage explained little, and naturally that had people worried. As such, I owe everyone an explanation regarding the state of SFMLab over the past month, and how we will continue going forward.
The Secret History of SFMLab
But first, I'll provide some historical context. I started developing SFMLab six years ago, in January of 2014. The idea was to create a site where people could safely upload SFM models rather than relying on sites like Mega, MediaFire and FileSmelt. Links to those sites tended to expire after some time, which lead to models getting lost after periods of inactivity. Building a site that mirrors the Steam Workshop but allows for adult content as well? A no-brainer.
But the project also served another purpose. At the time, my Python and devops skills were limited. I wanted to use the site as an opportunity to improve my familiarity with Python. This is where the issues start to come in. Building the first version of the site was easy enough. But when you're learning about effectively using certain frameworks by only reading the documentation and using your own intuition, mistakes start to slip through. Especially early on in 2014, I made a bunch of poor technical design decisions that came back to haunt me every time I would like to make a change to the site. This made it difficult to keep development of the site moving forward.
In 2016, my interests had shifted from SFM towards Blender, and I noticed I wasn't the only one in the community frustrated by SFM's limitations and Valve's apparent disinterest in maintaining project. What kept people from switching to Blender was the hike in learning curve (especially before version 2.8x) and a lack of models to start animating with. This is where SmutBase comes into the picture. SmutBase was intended to be a separate site where people could easily share NSFW models for more professional animation tools.
I started considering the technical aspects of the site. 2 additional years of Python and server management experience under my belt, having learned many lessons from SFMLab's development, I decided to design SmutBase with a more flexible and future-proof backend design. This paid off. SmutBa.se launched in 2016 and was much easier to maintain than SFMLab.
Cut to Winter of 2019. SmutBase now has a sister site, Open3DLab. It runs on the same backend as SmutBase and was built in a single weekend, without causing additional strain for the system. There are many features and improvements to be made to SmutBase still, but because the sites are running the same code and share the same database, any features added to one of them, will immediately be active for the other as well.
SFMLab on the other hand was still a problem child. It's still got a large active user base, new content gets uploaded very frequently. Making changes requires a lot of effort on my part, and the site still grows every day. And there are many features that I would like SFMLab to have as well. SmutBase has this great "chunked" upload form that allows us to display a progress bar for the upload. Implementing that for SFMLab was impossible in its current state. Design decisions I made 6 years ago were still biting me in the ass.
Shoveling myself out of the shit.
In order to be able to maintain SFMLab in the future, it would need to be moved to the SmutBase codebase. I knew this from the day SmutBase launched. But it would be a massive operation, involve service disruption, headaches and frustrations. I would also need to take time off from work in order to make it happen in a reasonable timespan. Remember: maintaining these sites is a side-project made possible by the limited ad revenue and the people on Patreon. Moving to a new place, switching jobs and the subsequent burn-out were contributing factors in the first two years after launching SmutBase. After working through that, I still didn't know where to start. While I did feel urgency, I kept delaying and putting it off.
2019 was a turning point for me. There are many reasons, some best discussed elsewhere, but the main takeaway is that I felt a need to move forward. The later half of 2019 was dedicated to slowly create feature parity between SmutBase and SFMLab, and introduce some changes to the SmutBase code to allow the same code to host multiple sites. Open3DLab served as a trial run. Launch a new site on the same codebase and look for potential issues. This plan went off almost without a hitch. The next step would prove the most difficult. Moving SFMLab's content to the new site.
Moving site content is never a 1 to 1 operation. Because the backends have different data structures, there are difficulties. Each entry needs to be processed individually, and transformed to the new data model. In addition, any assets like downloads and preview pictures need to be imported and moved as well. I set up a task queue to facilitate this. Several worker processes would run through items, images, uploads and comments in parallel. Refining this process would take some time, and running the final import and sync also took longer than expected. Also, once the import was started, no additional changes to the database should be made, as new changes would not be synced over and be lost. This is why the site was set to read-only mode.
I had taken a week off from work to build, deploy and finalize this migration process. In hindsight, too ambitious. Once the first import had started, edge cases manifested, most of them related to matching fields from one database to the other, as well as the task queue getting overloaded. This left the database in a sort of half imported/half failed state, which I spent the tail end of the first week cleaning up. As a result, the migration window had to be extended for some time, since I had to get back to work on my day job the next week, spending only the weekends on the migrations. That's why the read-only mode on SFMLab.com persisted for so long.
DoppelgΓ€nger
You may have noticed that I've left out the migration user accounts from the previous explanation. Because SFMLab and SmutBase were launched as separate sites, they have their own user database. Convenient at the time, but a hurdle to overcome at this point. One way to import the data would be to just copy over all the usernames, email addresses and passwords, right? Wrong! While the databases are separate there's a lot of overlap in the users. Although there are many slight differences. Some people have used different usernames, email addresses and passwords for signing up. Some usernames have been taken by completely different people. These types of collissions are difficult to reconcile.
That's why I have opted for a different approach. All uploads are assigned to a system user. People who have a SmutBase account can log in and people who only had a SFMLab account can sign up for a new "SmutBase" account. The next step is to enter the old SFMLab credentials to connect their SFMLab account to their SmutBase account. All uploads will then be re-assigned to their SmutBase account. A newsletter will be sent out to all SFMLab users, notifying them of the change and provide instructions for what to do next.
I had my doubts about the approach I described above. It is disruptive and inconvenient for users. However, the main benefit would be speed of the entire process, and it would allow for shutting off the old site as soon as the migration is completed. Any resolution of accounts, uploads and comments could happen in the months after, at the convenience of the user. It's a trade-off that I still feel confident about. My only worry is that people will not understand and blindly ask questions on Discord without reading any of the messages I've sprinkled all over the site.
Unfinished business
You may have noticed that some things did not get transfered over, like Patreon perks and Verified Uploader status. With the new codebase, a new system becomes accessible to us. This allows you to connect your account directly to the site. Patreon perks will then be granted automatically, rather than manually. The Patreon tiers will be revised, with new perks to be introduced soon. Please stay tuned for the reveal of these perks, and send in your suggestions.
In addition, we will be making an official application form for Verified Uploaders. The old system was loosely defined, poorly documented and involved some effort to enable for each individual user. With the new system, we'll have an checklist that must follow before Verified status is granted automatically. The benefit of this is that it will also finally introduce Verified users to SmutBase and Open3DLab at the same time. Having the sites on the same codebase means that these features can be rolled out for all sites at the same time, while only requiring development time once. While this is being built, please take some time to get used to the new site layout, upload forms and moderation process. More information is coming soon.
Lastly, you might have noticed that the TOR/Onion links do not work as expected. There's some adjustments to be made to the configuration on the TOR endpoints, as well as the site configuration. These changes are mainly related to the lack of HTTP to HTTPS redirects when visiting the site on the clearnet. The current TOR setup relies on direct access to the unencrypted HTTP version, since accessing via the HTTPS version would result in a certificate mismatch. A solution for this is in the works.
Moving forward
For the next three months, I'll be watching the amount of users that have successfully connected their account, and monitoring whether or not the re-assigning of projects goes off without a hitch. In the mean time, there will be more technical improvements, as well as new features introduced to the site in the coming months. With the sites now fully integrated, maintenance tasks will be easier to perform, and new updates can be added with less effort. To me, this is incredibly exciting, and I hope you will stick with us as we move to the future.
I'll also be looking to hear from everyone if they find missing content or data errors that might have been introduced as a result of the migration. If you have additional questions, feel free to ask in our Discord chat, or send inquiries directly to admin@sfmlab.com.
Kind regards,
Ganonmaster
SFMLab, Open3DLab & SmutBase Administrator
Comments
DarkZeka
It's explains why my user don't work anymore.
Pygo
Whew for a minute i thought i was gonna have to finally buckle down and learn blender. Lol. Thanks for the work ganonmaster.
divebomber
i have to commend your dedication to what you started and still have day job! This is a great site and deserves our support if we use SFM or and Blender..now I usually mess up and the Patreons I support usually get a little grumpy with me and I dont blame them as I usually make dumb mistakes when asking them things..here goes..u will notify us when we can log in to sfmlab and then we can transfter to new site? tks in advance ..gonna check my tier now and will up it if i can..
