Skip to main content
Tim-Doerzbacher.comTim-Doerzbacher.com

Personal Cloud Software Pain Points

For the last few weeks, I’ve been evaluating using ownCloud for building my own personal cloud. The install process is more time consuming than it is difficult. Now that I’ve got my server up and running well, I’ve been putting it through the paces more. I’d be lying if I said that I wasn’t hitting a bit of resistance.

I did have second thoughts early on that maybe I should’ve gone with Nextcloud. Before panicking, I decided to read up more on Nextcloud. The issues that were extant in those reviews of Nextcloud’s software were more or less the same deal as ownCloud, but with slightly different caveats.

Same shit, different day.

Feeling trapped in a corner, I realized I may have to donate some money to the cause to get the solution that I desired. I initially scoffed at the Play Store charging more than $0, but times were tough.

Then I realized my error.

For the best experience with both contact and calendar syncing, it now seems obvious to me to use DAVx5. It solves the issues that plagued me. Syncing information while avoiding unnecessary duplicates is actually a difficult problem to solve, and I’m grateful it’s been solved. My initial reluctance was overcome when I found out that I could download said software for free, compile it, and install it on my phone for nothing. I could even do updates to my heart’s content. Or… I could spend a couple bucks and never have to open the Android IDE again.

My first experiences with Linux was through the difficult and cumbersome Slackware distribution. You have to do almost everything yourself, and it takes a lot of time. Hell yeah, I’m willing to give anyone $5 (or less) to free me from that perpetual Sisyphean maintenance cycle.

I thus enthusiastically accepted that donation rather quickly.

I’m hoping to be able to do a tutorial soon to help others through this, but I’ve found the existing documentation and community help more than sufficient.

Well done to both ownCloud and Nextcloud both.

Cannibalizing Servers

In an era inundated with plagues of thin-skinned people with too much free time, I feel it necessary to first state the obvious: I’m referring to removing serviceable parts for use in the repair of other equipment of the same kind. People eating other people is likely a micro-aggression by the time you’ve read this. This has been your trigger warning. #futureproof

— The Editorial Staff

It’s Time

Near the beginning of the year, I realized it was time for an upgrade. I had things largely figured out requirements wise, and was planned on pulling the trigger on purchasing everything in a couple weeks. I had priced out a reasonable video card that would do what I needed (PNY – XLR8 Gaming Single Fan NVIDIA GeForce GTX 1650 SUPER) for around $220. At time of writing, it’s now going for over $550 on Amazon. I saw similar video cards skyrocket during that time to nearly $1,000 dollars.

Oh shit…

My upgrade plans were suddenly in peril.

Don’t Panic

Knowing that panic is best to be avoided, I immediately forgot that advice. I ended up spending the next month or so battling the sort of defeatist ruminations our society has been so thoroughly plagued of as of late. I knew that panic buying expensive video cards was to be avoided, yet I knew not what to do. I waited for inspiration. Time then passed slowly and fast at the same time.

I spent a lot of time lamenting the lost of whole chains of computer and electronic stores such as RadioShack, CompUSA and Circuit City. (Technically RadioShack still exists, but it’s not what it once was.) I still don’t understand how it happened. When did it become so hard to get hardware? My old server reliably running for 10 years left me blindsided by how much the world changed. Compared to 20 years ago, it’s beyond my ability to understand.

This lead me to realize that I was becoming a greybeard. I’ve been programming, building computers, and whatnot for more than 20 years. I even have a bunch of gray hairs now (which I jokingly tell my wife are her fault). I’m a greybeard and I should be better than this. I owe it to my family and my wife to do better. It’s time to go level up past Gandalf.

Around that time I stumbled upon Micro Center. It reminded me much of what proper stores were like. I looked through their catalogue seeing the same thing I’ve seen everywhere else: sold out video cards.

Greybeard Plotkai

That’s when my new greybeard status kicked in. I’m not a victim. There has to be a better way to look at this problem. Do they sell the computers with the components I need inside them? Are the motherboards in them still a standard size? Will this fit in my existing 4U server case?

As it turns out, yes they do, yes they are, and barely yes on that last one.

Hell, even Costco has desktop computers with NVIDIA 1650/1660s in them for reasonable prices. If a 1660 costs $700, that $900 barebones kit with a 1660 is looking mighty nice. There is no law against turning a gaming rig into a proper server. I was already going to need a new motherboard, RAM, CPU, and other things. Why not get 80% of the way there with 20% of the money? Find a good deal, add $100 of RAM, and BAM! You’ve got yourself a server.

It’d have been nice to think of this earlier. Hell, I watched Moneyball. I should’ve seen this coming.

Conclusion

And that, my friends, is how I was able to upgrade my server within budget.

In our next episode, I’ll explore more about where I’m going with cloud software now that I have the metal. It’s down to ownCloud and Nextcloud.

In possibly a future sidebar, I’ll explain how convincing your wife to spend a whole day in the car hunting for computer parts is easier than you might expect. Sell the trip as an opportunity to try out a new restaurant in a new city. Buy tickets to a local art museum. By the time you get to “and let’s stop by Micro Center for 15 minutes” she’ll already be sold.

Personal Cloud Requirements Gathering

To distract myself from excruciating pain while paining for a dental procedure, I figured I’d go over what an ideal end state for my personal cloud would look like. Let’s start by looking at what devices I’ll need to be worried about as head of IT in my household.

Devices to Support

DeviceOperating SystemStorage Space (GB)Notes
Pixel 2 XLAndroid64GBMost of this photos/videos taken w/ the camera app that I need off the phone
Old Samsung tabletAndroid32GB (estimate)Possibly going to be retired soon. Screen isn’t easy to read and it’s not fast enough to keep up with software bloat.
iPhone ?iOS128GB (estimate)Wife’s cat and flower photo collections must be treated with equal importance to financial records
iPadiOS256G (estimate)Used for audio books and researching supplies and materials for remodeling
Macbook AirMacOS256GB (estimate)Floater laptop that shouldn’t have much on it
Macbook ProMacOS512GB (estimate)Important information and video editing
Custom 4U serverLinux8TBThis will be the server handling the cloud. It is the one that needs to be greater than all the other devices attached. Can’t be larger than 8TB at the moment due to that costing too much to backup w/ current hardware.
Dell XPSLinux256GBFloater laptop for serious software engineers who also value battery life
System76 Serval WSLinux1TBUltimate badass laptop with a cooling system louder than jet engine with matching power
This table doesn’t include devices that don’t nee to be synced up with the cloud. All the Raspberry Pis don’t need complicated backup and syncing.

Observations

  1. All systems are *nix systems. That affects nothing, but does make me happy.
  2. I’ll need, at minimum, 6TB to handle everything as is. I’ll therefore want 8TB in the server for allow for growth.
    • Going above 8TB gets expensive, as single external drives larger than that are very expensive. Having two 8TB drives to rotate through is much more affordable.
    • I can always add another 4TB drive to the RAID for a total of 12TB down the road. I’ll have to reconsider my backup plans potentially.
    • This should keep me covered on space for a few years. Will definitely need to upgrade within 5 years, but no sooner than 2 years.
  3. This will create a lot of network traffic
    • I was considering splitting the home network into multiple isolated networks:
      • Private network for me and my wife
      • Guest Network for friends and family
      • IoT, Smart TVs, and other devices with histories of atrocious security
    • Doing this ahead of time would definitely be feature creep for the “build my own cloud” project.
    • Might just put the guests on the same network as the IoT, as neither are to be trusted.
    • I’ve got a Raspberry Pi or two laying around for building a custom router

Conclusion

My goal is to get this new system up and running before 2021Q2. However, hardware is becoming harder to get one’s hands on. That’s why I expect this to take some time. I’m also expecting my tax return won’t be arriving before beginning of March.

Long Awaited Server Upgrade

https://pixabay.com/photos/calculating-machine-calculator-370777/

An Upgrade Is In Order

My current server is running a Intel(R) Core(TM) i3-2100 CPU. Based on archeological evidence and carbon dating, experts estimate that it is around 10 years of age. It’s had minor upgrades over the years with a respectable 4 × 2TB drives setup in RAID5 for 6TB usable. However, it’s simply too long in the tooth. It’s been working flawlessly for so long, I seem to have taken it for granted.

I really only thought about it when I’d occasionally have to reboot it every couple months. Looking at it that way, I almost feel bad putting out such a long running workhorse to pasture.

End of the day, I should feel proud that I built a system that worked near flawlessly for a decade.

Mission accomplished. What’s the next mission?

Requirements

I’ve become a simple man when it comes to servers. All it takes me is half a dozen cores, heaps of memory, and terabytes of space to be happy. I’m wondering how much the hardware game has changed since I last went down this road. I’m hoping I’ll be left with the end result of a reliable workhorse for many years.

Off the top of head, here are requirements, typical workloads, and tasks the server will encounter:

  • Building the personal cloud that spawned this thread
  • Storing over 9,000 new photos and videos daily from my wife’s phone of our 8 month old cat.
  • Video stabilization and other processing of grainy VHS home movies
  • Secure storage of personal documents
  • 3D scanner processing
  • Minecraft server
  • Git repo to avoid relying on Github too much

Here’s what I’m thinking will be sufficient:

Parts List

Total: $1,000 estimated

Conclusion

If I can get 5 years out of this build, I’ll be happy. There’s a leap year in there. That’s a savings of $0.0003 per day. Cost of ownership should be around $0.55/day ($1,000 / (5 × 365 + 1)).

Virtual Homesteading

For a few years now, I’ve been gotten complacent and lazy due to Google’s services. Free space on someone else’s server was too tempting, regardless of privacy concerns. It’s time I do something about it by creating my personal cloud for my household. This is a big enough undertaking that I might as well write down my thoughts in this blog.

Requirements

  • Automatic syncing of photos from our mobile devices
    • Apps will need to be available for both iOS and Android
  • Easily securely share files with third parties
  • Works well with syncing large files
  • Zero external dependencies
  • Back up and restore procedure in place
    • Automatic syncing of all data to an offsite location
    • Offsite backup should be encrypted
    • Incremental backups to external hard drives

Known Unknowns

  • How to test the backup and restore procedure?
  • How to sync the backups to an external location?
    • Probably looking at a Raspberry Pi based solution with an 8TB external hard drive.
    • Looking at around $250 for both.

Evaluating Options

The above requirements hint at me needing two main pieces of software. I can’t only worry about the cloud software. If my server dies, I’ll need a rock solid plan for restoring from a backup.

It’s also been maybe 6 years since I’ve done a major upgrade to my server. At this point, I’m thinking of replacing everything except the case. I like my giant black rock sitting on the shelf, and it’s got plenty of space for loading up drives.

On cloud software front, I’m a bit torn between Nextcloud and ownCloud. I’m leaning towards ownCloud because the documentation seems a bit nicer. There aren’t really any other viable options for open source software that I’ve been able to find.

The backups remain a mystery to me. Up until now, I’ve been using a Raspberry Pi based computer and SyncThing to automate partial backups. That’s not going to work well. I also don’t want to have to use a GUI to backup. Currently I’m leaning towards using BorgBackup. It seems pretty feature rich.

Closing Thoughts

The general plan of attack will be:

  • Upgrade server hardware with bare install of OS (do not immediately start copying files back!)
  • Create backup and restore procedure, and test that it works
  • Install and configure ownCloud
  • Reload all the terabytes of old data into the new cloud
  • Set up my phone to sync with my new cloud to disentangle myself with Google as much as possible

Thinking each of those deserves its own post, so we’ll leave it there for now.