@CJ_Clippy
patreon
Futureporn February 2024 Update
🕑 Added 2024-02-15 06:19:01 +0000 UTCBackend Progress
It's amazing just how much WORK goes into making a video site. This is not a complaint (I like this work) but I sometimes have to take a step back and appreciate websites like youtube that do incredible amounts of work behind the scenes.
A lot of work has been going into Futureporn's backend. This is the part of the website visitors never see, but it's smooth function is incredibly important for Futureporn's growth.
We've made a lot of progress towards automation and rock solid reliability, but we're not there yet. I ran into several challenges revolving around IPFS kubo, the software that runs 24/7 providing video to many visitors.
One problem I came across was that kubo primarily uses a PULL system to get the content it needs to serve. That is, when I add a new VOD to the system, kubo needs to know the content ID of that VOD. Then, kubo can reach out to the internet and request the matching content from other IPFS kubo instances.
This is a good system and it works, but there are caveats. Firstly, a computer that is adding content to it's local IPFS kubo instance must have enough RAM to cover the size of the video. A big video file means that lots of RAM is required. Secondly, the computer needs to stay online for as long as it takes for the content to seed to other IPFS kubo provider nodes. This usually means that kubo needs to run as a daemon, 24/7.
None of these are issues for a dedicated server. The issues arose when I added a new backend system (Windmill) meant for ephemeral (short-lived) tasks like generating thumbnails, transcoding video, and creating Mux assets (CDN1).
Another important task I need to automate is adding VODs to IPFS kubo, but because of it's caveats, kubo is not great to run on an ephemeral task runner.
Ideally, my task runner would spawn a worker, download the vod file from backup, add the video to IPFS kubo via http API, update the database with the newly generated Content ID (CID), then it would be all done and the worker would shutdown.
In practice, I tried this, and the results were not good. IPFS kubo's API is very rudimentary, and would often fail due to network inconsistencies or transfer timeouts. It was a rare occurrence to see a large file transfer succeed.
After lots of IPFS forum and issue tracker browsing, I found the recommended approach for adding large files to kubo via API is to first pack the content into a .CAR file-- essentially an IPFS-friendly .zip file. This sounded great at first, until I realized it required kubo to run on the worker machine, and required more RAM than what my worker was configured for.
The solution I came up with was to write a new program called link2cid which essentially gives my worker instances the PUSH system I had hoped for. Instead of telling kubo which IPFS CID to PULL, link2cid downloads the VOD from a URL and pushes it into IPFS kubo.
This is a new superpower which helps kubo grab files that are not yet on IPFS. And because the onboarding happens on a reliable local connection, we avoid the kubo API reliability issues I ran into previously with adding large files via network.
Moving fast and breaking things
I broke my production deployments, again. This means that beyond the automatic updates that happen when I upload new VODs, I can't push new site updates and bugfixes at the moment. This happens from time to time as I make improvements to the site and the tooling that creates it.
It takes time for me to get the system back on track. Myself is included in that system and there's a certain amount of fear that I have to overcome every time I go through a cycle of change.
Learning a new way
Kubernetes is a system for managing all parts of an application. Database, app, worker, API, etcetera can all be expressed as microservices which make up a Kubernetes cluster. Within Kubernetes, individual services can be replicated and scaled up to handle demand. Auto-scaling can be configured to help meet demand that fluctuates on certain days (weekends) or certain times of day.
I think it's essential that Futureporn migrates to Kubernetes because Kubernetes offers structure where I have none. I have this issue right now where there are so many system components and each component has it's own unique way of deploying to production. There is currently no "publish all" button which gets all services updated to the latest version, and Kubernetes/helm fixes this for me.
Kubernetes has been around for many years, but I've been avoiding it for various reasons. Fear of complexity, doubt that microservices are the right choice, etc. Lately, I think Kubernetes is just the tool I need to dial things up to 11, make my systems standardized, predictable, and agile enough for fast iteration.
It's taking me more time than I hoped to learn everything I need to make a production deployment on Kubernetes. I know I've been promising new website features for months and I feel guilty for not delivering them sooner. At the same time, I think I'm doing a good thing in strengthening the base layer which will serve as a foundation for years to come.
Roadmap
Here is my current plan for what to implement and in what order. This list is subject to change based on the results of today's poll.
1. Deployment fixes (Kubernetes migration)
3. Stream Archive (100% speedrun progress indication)
2. 240p encodes
4. Multi-vtubers
5. Funscripts (vibrator integration)
6. Chat playback
7. Pay the moderators
8. VOD uploads
9. Opening up CDN2 to free tier
Minecraft?
I watch VShojo talent a lot. I see them having fun in Minecraft and it makes me want to jump back into that voxel world. Fun fact, I ran a Minecraft server from 2010~2015 and that's one of the things that got me a lot of experience in system administration.
I've been thinking of starting a Futureporn modded Minecraft server. Does that sound interesting?
I discovered that there is a mod which can play video. Imagine watching Futureporn VODs in-game while you build a giant penis for the monthly erection contest!
Stream Archive progress
Here's an update on Stream Archive progress. It's a table with sortable, filterable content to help you find the stream you're looking for.
Let's Make It Happen
Years ago I had a manager who would drop by to see how things were going. "Let's make it happen!" he would say after checking in. I always found it inspiring and I'm reminded of it now. It takes a lot of work to build a thing, but it's worth it. I'm going to get back to it now.
Thank you for your support! Futureporn would not exist without the patrons. Thank you for believing in the dream! Together we can create the Galaxy's best VTuber hentai site.
Poll
Which features are most important to you? Vote for as many as you like. The results will influence the order the features are worked on.
Comments
François d'Aruna
I agree with wat Eyldontknow said, I mostly want to support you so there is an archive of the streams I missed. I personally hate it when video's/streams I want to watch disappear from the internet
@CJ_Clippy
Thank you. It's so easy for me to get lost in details and forget about the big picture. What you mention really is the most important thing.
EyIdontknow
I don't think most of us support you cause we want as fast as possible new features all the time. We're to support you in creating an ever lasting archive which will mature over time. So having fewer new features for more stability, safety and maybe better playback in the future seems like a good trade.
