Skip to main content
Tim-Doerzbacher.comTim-Doerzbacher.com

Virtual Homesteading – Part 3

To distract myself from excruciating pain while paining for a dental procedure, I figured I’d go over what an ideal end state for my personal cloud would look like. Let’s start by looking at what devices I’ll need to be worried about as head of IT in my household.

Devices to Support

DeviceOperating SystemStorage Space (GB)Notes
Pixel 2 XLAndroid64GBMost of this photos/videos taken w/ the camera app that I need off the phone
Old Samsung tabletAndroid32GB (estimate)Possibly going to be retired soon. Screen isn’t easy to read and it’s not fast enough to keep up with software bloat.
iPhone ?iOS128GB (estimate)Wife’s cat and flower photo collections must be treated with equal importance to financial records
iPadiOS256G (estimate)Used for audio books and researching supplies and materials for remodeling
Macbook AirMacOS256GB (estimate)Floater laptop that shouldn’t have much on it
Macbook ProMacOS512GB (estimate)Important information and video editing
Custom 4U serverLinux8TBThis will be the server handling the cloud. It is the one that needs to be greater than all the other devices attached. Can’t be larger than 8TB at the moment due to that costing too much to backup w/ current hardware.
Dell XPSLinux256GBFloater laptop for serious software engineers who also value battery life
System76 Serval WSLinux1TBUltimate badass laptop with a cooling system louder than jet engine with matching power
This table doesn’t include devices that don’t nee to be synced up with the cloud. All the Raspberry Pis don’t need complicated backup and syncing.

Observations

  1. All systems are *nix systems. That affects nothing, but does make me happy.
  2. I’ll need, at minimum, 6TB to handle everything as is. I’ll therefore want 8TB in the server for allow for growth.
    • Going above 8TB gets expensive, as single external drives larger than that are very expensive. Having two 8TB drives to rotate through is much more affordable.
    • I can always add another 4TB drive to the RAID for a total of 12TB down the road. I’ll have to reconsider my backup plans potentially.
    • This should keep me covered on space for a few years. Will definitely need to upgrade within 5 years, but no sooner than 2 years.
  3. This will create a lot of network traffic
    • I was considering splitting the home network into multiple isolated networks:
      • Private network for me and my wife
      • Guest Network for friends and family
      • IoT, Smart TVs, and other devices with histories of atrocious security
    • Doing this ahead of time would definitely be feature creep for the “build my own cloud” project.
    • Might just put the guests on the same network as the IoT, as neither are to be trusted.
    • I’ve got a Raspberry Pi or two laying around for building a custom router

Conclusion

My goal is to get this new system up and running before 2021Q2. However, hardware is becoming harder to get one’s hands on. That’s why I expect this to take some time. I’m also expecting my tax return won’t be arriving before beginning of March.

Virtual Homesteading – Part 2

https://pixabay.com/photos/calculating-machine-calculator-370777/

An Upgrade Is In Order

My current server is running a Intel(R) Core(TM) i3-2100 CPU. Based on archeological evidence and carbon dating, experts estimate that it is around 10 years of age. It’s had minor upgrades over the years with a respectable 4 × 2TB drives setup in RAID5 for 6TB usable. However, it’s simply too long in the tooth. It’s been working flawlessly for so long, I seem to have taken it for granted.

I really only thought about it when I’d occasionally have to reboot it every couple months. Looking at it that way, I almost feel bad putting out such a long running workhorse to pasture.

End of the day, I should feel proud that I built a system that worked near flawlessly for a decade.

Mission accomplished. What’s the next mission?

Requirements

I’ve become a simple man when it comes to servers. All it takes me is half a dozen cores, heaps of memory, and terabytes of space to be happy. I’m wondering how much the hardware game has changed since I last went down this road. I’m hoping I’ll be left with the end result of a reliable workhorse for many years.

Off the top of head, here are requirements, typical workloads, and tasks the server will encounter:

  • Building the personal cloud that spawned this thread
  • Storing over 9,000 new photos and videos daily from my wife’s phone of our 8 month old cat.
  • Video stabilization and other processing of grainy VHS home movies
  • Secure storage of personal documents
  • 3D scanner processing
  • Minecraft server
  • Git repo to avoid relying on Github too much

Here’s what I’m thinking will be sufficient:

Parts List

Total: $1,000 estimated

Conclusion

If I can get 5 years out of this build, I’ll be happy. There’s a leap year in there. That’s a savings of $0.0003 per day. Cost of ownership should be around $0.55/day ($1,000 / (5 × 365 + 1)).

Virtual Homesteading – Part 1

For a few years now, I’ve been gotten complacent and lazy due to Google’s services. Free space on someone else’s server was too tempting, regardless of privacy concerns. It’s time I do something about it by creating my personal cloud for my household. This is a big enough undertaking that I might as well write down my thoughts in this blog.

Requirements

  • Automatic syncing of photos from our mobile devices
    • Apps will need to be available for both iOS and Android
  • Easily securely share files with third parties
  • Works well with syncing large files
  • Zero external dependencies
  • Back up and restore procedure in place
    • Automatic syncing of all data to an offsite location
    • Offsite backup should be encrypted
    • Incremental backups to external hard drives

Known Unknowns

  • How to test the backup and restore procedure?
  • How to sync the backups to an external location?
    • Probably looking at a Raspberry Pi based solution with an 8TB external hard drive.
    • Looking at around $250 for both.

Evaluating Options

The above requirements hint at me needing two main pieces of software. I can’t only worry about the cloud software. If my server dies, I’ll need a rock solid plan for restoring from a backup.

It’s also been maybe 6 years since I’ve done a major upgrade to my server. At this point, I’m thinking of replacing everything except the case. I like my giant black rock sitting on the shelf, and it’s got plenty of space for loading up drives.

On cloud software front, I’m a bit torn between Nextcloud and ownCloud. I’m leaning towards ownCloud because the documentation seems a bit nicer. There aren’t really any other viable options for open source software that I’ve been able to find.

The backups remain a mystery to me. Up until now, I’ve been using a Raspberry Pi based computer and SyncThing to automate partial backups. That’s not going to work well. I also don’t want to have to use a GUI to backup. Currently I’m leaning towards using BorgBackup. It seems pretty feature rich.

Closing Thoughts

The general plan of attack will be:

  • Upgrade server hardware with bare install of OS (do not immediately start copying files back!)
  • Create backup and restore procedure, and test that it works
  • Install and configure ownCloud
  • Reload all the terabytes of old data into the new cloud
  • Set up my phone to sync with my new cloud to disentangle myself with Google as much as possible

Thinking each of those deserves its own post, so we’ll leave it there for now.

New Years Goals

It’s very easy to miss the mark if you don’t know where you’re aiming.

To that end, here’s some goals for the year:

  • Create a private cloud for me and my family
    • Google’s notorious for cancelling services without little warning.
    • Facebook and Instagram don’t always act in the best interests of their users.
    • I don’t want everything I take a picture of being uploaded to someone else’s server farm.
  • Create another robot
    • I did a bunch of work on designing different robots over the years.
    • Buying a house right before getting married during a global pandemic was distracting.
    • This is mostly an excuse to purchase a laser engraving and cutting machine.
  • Finish first pass of remodeling my house
    • Living room, main stairwell, and dining room remain.
    • Kitchen, master bedroom, office, and guest bedroom are mostly done already.
    • Link to everything that I used to help others. Utilize affiliate links where possible to profit if deserved.
  • Write a post for each major goal
    • Writing down goals and evaluating keeps me honest and me keep moving forward.
    • One’s ability to write can help or hinder one’s progress through life. Might as well Git Gud™.

New Year / New Website

It was sometime after the initial Sniffle Snafu Panic of 2020 that I realized I’d not updated my website in years. It’s unfortunate, but a cold hard fact of life that microscopic entities can be deadly. Bad times are inevitable if there’s a bad enough pathogen, and there will be death. That we’ve had. We should be reminded that we all should be grateful to be alive every day.

Around that time there was the proposal of 15 days to slow the spread. Expecting to bunker down for only a short period, I resolved myself to make a new website. I knew I’d not be in the office for a couple weeks. Fast forward many moons to the present year, and I’d still not yet accomplished my original goal. I’ve still not been back in the office.

It seems that they should have said 15 months to slow the spread

That said, it’s been more than a few weeks. Time to go on doing what I can. Time to update this website.

So It Begins Again

I’m writing this as I start playing with WordPress again. This is not the first blog I’ve started. It may hopefully be the first blog I’ve not abandoned. Time will tell.

Initial thoughts are that WordPress is still just like riding a bike you don’t like downhill. It’s easy enough to get back up to speed, but it’s not a pleasant journey. This is especially true and something goes wrong. Where are the damn breaks on this thing?

That said, I’ve had to error_log(...) the hell out of code in the past. I’ve been through worse.

Let’s play with the templates and styling again. It’s not glorious; but sure as God’s got sandals, it gets the job done.

First Time Home Owner Regrets

The original regret.

It’s been almost a year since becoming a first time homeowner. It’s been pretty great. However, some mistakes have been made.

I neglected to take photos of rooms before I moved things in. I really regret not having proper before and after photos of past renovations.

My record keeping was also shoddy. It would have been nice to go back and look at what materials were used. Adding affiliate links to the Lowes, Amazon, and other purchases would have been a nice option in the future.

Not all hope has been lost. I’ve only renovated a single bedroom. The rest of the stuff I’ve done has been replacing electrical outlets and adding half a dozen or more circuits. There’s still plenty of projects to do.

CATPOO Progress

It’s been some time since I’ve posted on this blog. Now I’ve actually been working on projects worth mentioning, the long period of neglect is over.

Sometime in fall of 2017, I decided to see what kind of robot I could make. Since going down the rabbit hole, the biggest hurdles faced thus far:

  • Learning how to model in 3D.
  • Electrical engineering in general.
  • Mechanical engineering in general.
  • Explaining why I’m doing this.

I’m no expert in the first three after less than a year of learning. I never expected to be. I only wanted to know enough to accomplish my goal: build a proper robot.

What makes a “proper” robot? I’ve decided upon these criteria:

  • It has to be built entirely from open-source tools and designs.
  • It has to be remotely controllable from a cell phone or tablet.
  • It has to have a high resolution camera and should have night vision.
  • It should have additional sensors to avoid hazards in the environment.
  • It should powered by an Arduino and Raspberry Pi together to allow future software updates for expanded functionality.

Now that I’ve accomplished those goals, I’ve moved on to the polishing phase. Part of that phase includes writing things up in such a way that anyone else in the world could create the same robot.

In the mean time, here are some pictures from along the way:

Late 2017 Prototype

Early 2018 prototype
Early 2018 prototype

Mid 2018 Prototype with its top off on the pool table

Mid 2018 Prototype in the dark
Mid 2018 Prototype in the dark

Mid 2018 Prototype
Mid 2018 Prototype

 

Video Stabilization

Recently I purchased a kayak. I wanted to take photos with my GoPro mounted to the top. Seemed simple enough. It was, and thus was totally unacceptable.

Let’s make this user story more complicated.

The Story

I wanted to be able to create a time lapse video quickly to send to my friends. From there, I’d send the video to anyone who was with me that day. If they wanted an original high resolution image used to create the video, they could give me the time stamp. From there, I could send the dozen or so images that created that second of video easily.

Put another way: I was in essence creating a quick way to preview the thousands of photos GoPro can take in a short time.

Naïve First Try

Seems simple enough:

  1. Hit record on GoPro
  2. Kayak
  3. Download Images
  4. Let GoPro Studio do its thing

That didn’t work.

That shakiness makes it impossible to watch. Ain’t nobody got no time for that.

Second Pass

I came to realizing no filter GoPro could provide would help that. Manually rotating and lining up all frames would take forever. Time to automate.

First I created script stabilize:

#!/bin/bash
# Copyright 2016 Tim Doerzbacher <tim at tim-doerzbacher.com>

MELT=melt
# Stabilization options
MELT_FILTERS="-filter vidstab shakiness=8 smoothing=20 optzoom=0 maxangle=0.05"
# Encoding options
MELT_OPTIONS="vcodec=libx264 b=16000k acodec=aac ab=128k tune=film preset=slow"

if [ -z "$1" ] ; then
  echo "Usage: $0 <infile> [outfile]"
  exit 1434;
fi

SRC="$1"
DEST="`echo $1 | sed -r 's/\.(mp4|mpg|mov)/.stabilized.m4v/'`"
MLT_FILE="`echo $1 | sed -r 's/\.(mp4|mpg|mov)/.stabilized.mlt/'`"

if [ ! -z "$2" ] ; then
  DEST="$2"
fi

if [ "$SRC" = "$DEST" ]; then
  echo "Did not recognize that file type."
  exit 1 # Not random
fi

echo "Stabilizing: ${SRC}"
echo "Destination: ${DEST}"

echo
echo "======== Round #1 ========"
echo
$MELT "$SRC" $MELT_FILTERS -consumer "xml:${MLT_FILE}" all=1

echo
echo "======== Round #2 ========"
echo
$MELT "$MLT_FILE" -audio-track "$SRC" -consumer "avformat:${DEST}" $MELT_OPTIONS

So the new order of things was:

  1. Hit record on GoPro
  2. Kayak
  3. Download Images
  4. Let GoPro Studio do its thing
  5. Process resultant video with stabilize

This method left a bit of weirdness at the edges. I tried using the auto-zoom functionality but didn’t like the result. Since GoPro studio already threw out a lot of the original resolution, zooming in further throws away even more information. The end result is far from optimal and looks blurry:

There Must Be a Better Way

There was a better way. I have a large amount of sequentially named files and a snazzy Linux server. I’m losing a lot of resolution by stabilizing the video after its already been scaled down.

Time to make a new script: encode.

#!/bin/bash
FFMPEG=ffmpeg
$FFMPEG -pattern_type glob -i '*.JPG' -c:v libx264 video.mp4

Now I have an easy way to create one video out of all images taken that day.

$ encode JPG; stabilize video.mp4

At this point, I can take video.stabilized.m4v and import it into GoPro. Now I have full resolution video that I can then crop as necessary. The result works out quite a bit better:

Still, I’m having problems smoothing the video out. At this point, I began I wonder how it would work with better data. The water drops taking up so much of the field of view. It is probably affecting the stabilization.

Better Data

A few days later I had the opportunity to revisit the lake to do some plein air painting.

After that was done, I recorded and processed another time lapse using the scripts above:

Ultimately, I’m pretty happy with the last few attempts at making a kayak trip bearable. Things are still not perfect. I’m not aware of a way to make the GoPro take more than two frames per second. That’s causing the Flux processing in GoPro Studio to really make a mess of things.

With any luck I’ll have a another post soon explaining how to mitigate these issues.

References

Here are some of the pages I referenced while creating this. Thanks, guys!

Holonomic in the Hizzouse

Partially assembled top view
Partially assembled view from above

I’ve spent months planning my next CATPOO and it’s all printed out. All requisite parts are either here or on their way in the mail.

Highlights include:

  • Omnidirectional wheels for holonomic goodness
  • Infrared night vision camera
  • 16x High intensity IR LEDs
  • 7″ LCD with touchscreen
  • External USB and network ports
  • Wireless AP support
  • Rear motion sensor for Spidey Sense
  • Forward and rear proximity sensors

Front view showing IR LEDs and camera
Front view showing IR LEDs and camera

Once I get the few minor parts remaining from Pololu, I’ll be ready done with the building in a day. If you think you might need it, and it’s only a $1 or $2, just get it. Waiting three days to finish the build because I’m short on 1″ #4 screws is a terrible experience.

Once built, I’m nervous when it comes to figuring out how to coordinate all four servos to move in a straight line. Previously, I ran into trouble with getting the forward and reverse speeds to be equal. It could also be related to the servos I’m using. I plan to isolate whether it’s my fault, the motors’ faults, or a combination of both. I’ll have eight servos soon.

If I can’t get four functional servos out of that, I’m smashing the servo controller I have a sledgehammer and switching to stepper motors. I didn’t want to spend the money or effort, but it’s looking like that’s the direction I’ll be going.