Popular Tags:

Homestead Network Upgrades

October 22, 2017 at 1:05 pm

Despite coming from the networking side of IT, I tend to use regular consumer grade equipment at home. It typically just works, and I’m not looking for extreme reliability or features. I’ve been using hardware from Linksys, Netgear, and the other consumer network vendors for at least the last 10 years. Sometimes, though, things happen that make you reevaluate your previous life choices…

For me, that thing was an email that I received from Verizon saying my router was infected with malware. Since I always take basic precautions like changing the default password and locking down external ports, I was a bit surprised. Turns out, there was a vulnerability in the firmware that had gone unpatched for months… In hindsight, I should not have been that surprised. At all. I thought I had purchased a flagship router that would be supported for at least a few years, but it didn’t look like any more patches were coming. Ever. I looked into trusty old DD-WRT figuring that I could flash the router and at least get another year out of it, but apparently the R7000 has some performance issues with DD-WRT.

After having issues like this a few times with generic consumer grade stuff over the years, no matter the vendor, I decided enough was enough. I researched available options in the enterprise hardware space (way too expensive and time consuming to set up), looked at open source alternatives (cheap, but time consuming, and not well integrated), and even looked at the more pro-level offerings from consumer manufacturers (underwhelming). After a few days, I decided on and purchased some Ubiquiti hardware based on the many good reviews and a few personal recommendations from networking folks I respect.

Ubiquiti’s hardware is solid stuff, performance wise, and they have a very good reputation. The hardware is what I would call “Enterprise Lite”, meaning it’s not Cisco, but its perfect for small to medium businesses who just want things to work. Additionally, the Unifi configuration system and dashboard is excellent, taking a significant configuration and support burden off of me.

The initial hardware purchase was:

  • Unifi Secuirty Gateway Pro (Amazon)- I definitely went overkill here. The entry model USG is capable of routing gigabit at near wirespeed. However, I decided that I likes the extra ports for a few future projects, like the barn office.
  • Unifi Switch 8, 60 Watt (Amazon)- Since the new network was not an all-in-one setup, I needed something to power the other devices around the house. This managed switch provided a lot more than just that, though. The VLANs will come in handy when we set up the home office.
  • Unifi AP AC Pro (Amazon)- Another bit of overkill for home use, but this one was easier to justify than the firewall. Simply put, it has more power, and I need that given the 2′ thick stone walls in the farmhouse.
  • Unifi Cloud Key (Amazon)- Though not strictly necessary, the Cloud Key allows you to run your network controller app on dedicated hardware. It can also be linked to the Unifi cloud portal allowing for a very convenient and secure hybrid cloud management platform.

Simple Network DiagramThe hardware wasn’t cheap, but surprisingly, it wasn’t much more than I paid for the R7000 two years ago. If I had chosen the regular USG, the price difference would have been negligible.

As for the setup, it was easier than I thought. I racked the USG Pro, plugged in the switch, then the cloud key. Thankfully I had already run the line to the wireless AP so that was easy. I also threw in a Raspberry Pi server for fun. It took about 10 minutes to patch everything together. But what about the configuration?

Well, thanks to the Unifi software on the Cloud Key, I was able to “adopt” the other devices and have them configured in no time at all. My basic single vlan setup was ready to go out of the box. All totaled, I had the network up and running in 20 minutes. Time vs the R7000? Maybe an extra 10 minutes.

Unifi Dashboard

What has it been like living with “Enterprise Lite” hardware at home? Fantastic. Having a useful dashboard that I can glance at to see the status of the home network is a perk I didn’t think I would care about, but I’ve used it several times already. The speed is true gigabit on wired, the wireless coverage is solid, and we don’t have random drops in connectivity anymore. And as for patches… I’ve already had two patches come through for stack. It’s a simple matter of hitting the upgrade button for the device, or setting up auto-upgrade. As far as I’m concerned, I’m never going back to consumer gear again.

Saving The Jeep: A New Series

July 31, 2017 at 5:41 pm

There has been a 1979 Jeep CJ-7 in my family for nearly 40 years now. My Great-Uncle bought it new from the dealer, and it passed to my father, then my mother, and now to me. The Jeep has many fond memories associated with it. I can still remember the first time I road in it, with my uncle taking dad and me to the cabin. I remember when both my mom and dad were separately teaching me to drive and made me promise not to tell the other. I remember when, on one particular lesson, mom drove the Jeep off a steep embankment and I had to calm her down and get it out. Countless stories are wrapped up in that hunk of metal; precious memories that I wouldn’t trade for the world.

Unfortunatly life gets in the way sometimes. Dad passed away many years ago, and the Jeep became an occasional driver. Mom got sick a few years ago, and the Jeep was semi-permanently garaged. Recently, Mom passed after a long battle with cancer, and now the Jeep belongs to me. I don’t know much about cars, but I know keeping a vehicle in an unconditioned space for several years is bad for it. So what to do?

I’m a fan of a show called The Survival Podcast (TSP). In it, Jack Spirko, a renaissance prepper-cum-duck-farmer, talks about dozens of topics ranging from stocking a larder to bitcoin’s implications on the global economy. It’s a fantastically interesting show. TSP also has something called an Expert Council, comprised of subject matter experts from fields across the spectrum. One in particular stood out: Charles Sanville, the Humble Mechanic. I thought if anyone could help and offer guidance, he could. So I sent the following email to Jack.

Question for: Charles Sanville

Question: What should I do for an inherited 1979 CJ-7 that’s been garaged for the last 5 years and had some odd modifications done to it? It currently doesn’t run, but I’d like to keep it, and learn the basics of car maintenance and “restoration”.
Background:
My great-uncle bought an odd CJ-7 new in 1979 from the dealer. It has
  • A straight 6, automatic transmission (I think AMC 232?)
  • Power steering,
  • Manual breaks
  • All-time 4-wheel drive, Quadra-Trac, which makes the jeep really squirrely at speed).
  • Less than 20,000 original miles
  • Almost no rust

Over the years, it passed on from my great-uncle, to my father, then my mother. It’s a family heirloom at this point, and I have many fond memories of going camping, hiking, and to our families cabin in upstate NY. Heck, I ever learned to drive in it! I really want to keep this vehicle for weekend/occasional driving, camping, and because it’s all I have left of my family at this point. I’d love for my son to learn to drive in it some day.

There are a few known issues with the vehicle:
  • My dad didn’t believe in modern emissions regulations and pulled most of those components. There are hoses the terminate in a bolt and hose clamp. The Jeep ran after these modifications, but I’d like to get it back to “normal” running mode so that it doesn’t potentially mess up the engine.
  • Some of the control knobs inside come off.
  • All four whitewalls are flat and don’t appear to hold air.
  • The spare tire was side-mounted so a rear wooden cargo-box could be added. That box is now falling apart. Should I rebuild it or try to restore the spare tire to the rear?
  • It’s in a garage in upstate NY, and I need to get it hauled to my garage in PA.

I’m an IT Architect/engineer who used to build a lot of sets for theater, so I’m competent with tools and woodworking, but I have almost no experience with cars. I’ve changed oil a few times and that’s about it.

How do you get started with something like this? How do you figure out what was removed from the engine? Is a car this old worth restoring, or am I letting my sentimentality get in the way?
Any insight or advice you could provide would be greatly appreciated.
Sincerely,
-Derek M, in PA.
I sent it in wondering if the question was too specific for a followup on the show, but I figured it was worth a shot. A few weeks went by and no answer came, so I thought I’d have to figure it out on my own. Then, to my surprise, I heard my question on the air…

This is the start of a new series, documenting my Family’s 1979 CJ-7. Stay tuned for updates.

The Great Cleanup – Chicken Coop Restoration Part 2

March 4, 2017 at 8:01 pm

Welcome to Part 2 of the Coop Restoration series. In this post, I’ll go over the cleanup that I did this past weekend. The coop started out in rough shape. There were rolls of old insulation, mouse nests, mold… It had been used as a storage space for transient garbage for years. Below are some pictures after I pulled out the worst of the insulation. You can see some of the nest in the back left corner behind the cabinets and dog crate.

Once the big items were moved, sorted, and mostly thrown in the garbage, I did a preliminary sweep up. Turns out that half of the coop has unfinished hardwood floors! Bonus! After inspecting the chicken wire, I saw lots of rust, holes, and filth. There was no way to clean and reinforce it, so off it came. I also pulled off the old roosts and low panels as well.

I debated pulling out the old flooring and walls, but there’s only so much I can do in two weeks. The plan right now will be to disinfect them thoroughly, and lay some washable hardboard over them. This is the same material that I’ll be using for the lower two feet of wall as it’s easily cleanable and a great draft blocker.

After another sweep up, and vacuum, the place looked a lot better. The door was in pretty good shape, so that got left in place. I may have to pull it in the long run, though, as it currently swings inward and the wife and I are thinking about deep litter, but that’s an easy change at a later date.

Next up, in part 3: Framing and Re-Chicken-Wiring the coop!

To Export the Unexportable Key

March 1, 2017 at 5:04 pm

Every now and then, you have to export a certificate in Windows, and someone forgot to check that little box to let you be able to do it… What is an enterprising SysAdmin to do? Enter Mimikatz (source), a tool that lets you patch the Windows crypto api and do several cool (and frightening) things. The process is very simple.

To Export an Unexportable Private Key:

  1. Create a temp directory
  2. Download the latest version of Mimikatz
  3. Extract the appropriate version (32 or 64 bit) to the temp directory
  4. Open an admin command prompt
  5. Change to the temp directory
  6. Run mimikatz
  7. Type crypto::capi
  8. And finally type crypto::certificates /export

You’ll see all of the certificates in the MY store exported into the temp directory in pfx format. The default password is mimikatz. Want another cert store? Perhaps, the computer store? Simply run crypto::certificates /export /systemstore:LOCAL_MACHINE. Check out the github wiki for documentation on this and other cool features of this powerful tool.

Chicken Coop Restoration Part 1

February 28, 2017 at 6:01 pm

 

One of the wonderful things about our homestead is that we inherited several outbuildings. We have a large post-and-beam barn (40 x 60), equipment shed (16 x 24), storage shed (14 x 24 + lean-to), and a rather large chicken coop turned racing pigeon coop (14 x 24). Yes, you read that right. The previous owners really loved their racing pigeons and converted a perfectly good chicken coop into a palatial (for a pigeon) loft! Unfortunatly, the barn is the only structure in good shape, having been rebuilt by the previous owner. The rest of the outbuildings are in various states of disrepair.

Since we’re starting the new year off with a focus on sustainability, it’s time to look at our outbuildings and restore them to their former glory! Or at least, to a usable state. The first project will be to rebuild the chicken coop and get some birds in!

About the Chicken Coop

External view of the chicken coop

The Coop is a semi-insulated structure, elevated on piers, with a door on the short end closest to the house. It has several windows along the south wall, electricity, and a freeze-proof yard hydrant, and is in desperate need of a paint job amongst other things. Inside, there are two large rooms separated by wall. Each of those rooms has a wired off coop area and an open area. The previous owners must really have loved their racing pigeons to build such a large structure for them!

Original layout of chicken coop

Plan for the Chicken Coop

In addition to the basic cleanup of the building, the goal for the coop project is to make it able to hold a brooder in two weeks. As part of that, we want to do three main things: extend the interior coop wall to include the exterior chicken door, create removable roosting space, and build exterior-accessible nesting boxes.

  1. By extending just one section of the coop to include the exterior chicken door, we can keep more room for storage of supplies for the birds and other critters. If we end up running more birds than this space allows, I can always extend the entire wall.
  2. The roost space will be angled and removable. When brooding chicks, the roost will come out and the hover-brooder will go in the corner.
  3. Finally, having nesting boxes that we can access without having to go into the coop itself is just easier in the long run. I would very much like to have roll-out nesting boxes, but they tend to be expensive and we already have enough expenses rehabbing the coop this year.

Planned layout for chicken coop

So, what do you think?

Next Up, in part 2: The Great Cleanup!

My First Wine Kit – Winexpert World Vineyward Chilean Malbec

November 21, 2016 at 2:40 pm

Ready To GoI’ve brewed beer about a half a dozen times over the last few years. It’s not a hobby that I’m not particularly active in, but I do enjoy it once in a while. I’ve made ciders and ales a few times, even going as far as to make a Trippel once. The only thing I’ve bombed was a batch of mead which, for some reason, refused to ferment. Ah well, it was college. I blame distractions. Anyway, I thought it would be good to keep a journal of some of these activities. So here goes, my first foray into wine making: The Winexpert World Vineyard Chilean Malbec.

Authorized_Keys in Active Directory

November 21, 2015 at 6:21 pm

Now that we are implementing more Linux systems, I’m noticing some of the pain points of keeping certain things in sync. A big annoyance, for example, is keeping our infrastructure and users’ SSH keys in sync across all of our machines. There are several methods currently available, but I had issues with each. I’ve listed the two main methods below.

Via Configuration Management

A very DevOpsy way of tackling the problem would be to us a configuration management system like Chef to keep the files updated. In fact, there are several examples of this solution out there already. However, this seems a bit counter-intuitive to me. Why keep user account and related information in a config management system instead of a directory service? This is probably my Windows World bias, but there are others that agree.

Via Scripts/Dedicated Systems

From simple shell scripts, to complex systems, there are many ways to keep this data in sync. The simplest would appear to to be setting up NFS and pointing all users’ home directories there… But then you have to keep those NFS servers in sync and backed up across multiple sites, which can be problematic at scale.

Our Solution – AD/LDAP storage of SSH keys

To be up front, this was not my idea. There are many other folks who have implemented similar solutions. We are using this method specifically because we already have a robust AD infrastructure with all of our Linux authentication going through AD already (a post on this is soon to come). It probably doesn’t make sense for a group that already has a solid solution in, say, chef or puppet. For us, it did, and this is how we built it.

First, we had to extend the Active Directory schema. This is not something for the faint of heart, but is also not something to be afraid of. I followed the procedure listed here (after backing things up) and had everything ready to go in about 15 minutes. A note on the procedure: you do not need to use ADSIEdit to manage the custom attirbute afterwards. Just open AD Users and Computers and switch to the advanced view mode. Each item will then have an “attributes” tab in its properties page.

Once the schema was extended, the fun began. OpenSSH supports a config variable called “AuthorizedKeysCommand”. This allows us to call an arbitrary script to pull the users authorized_keys file. This serverfault post got me going on creating a custom command, but the output of SED wasn’t clean enough. I whipped up the following script in perl to get everything working nicely. It binds to AD using a username and password and then pulls all sshPublicKey values from the specified user account.

#!/usr/bin/perl
# Gets authorized keys from LDAP. Cleaner and supports any number of ssh keys, within reason. 
# Requires Net::LDAP.
use Net::LDAP;

$BINDDN="cn=service account,dc=example,dc=com";
$BINDPW="Password";
$SEARCHBASE="dc=example,dc=com";
$SERVER="domain or ip";
$SearchFor="samaccountname=$ARGV[0]";

$ldap = Net::LDAP->new( $SERVER ) or die "$@";
$msg = $ldap->bind( $BINDDN, password=> $BINDPW);

$result = $ldap->search( base => $SEARCHBASE,
                         filter => $SearchFor,
                        );

while (my $entry = $result->shift_entry) {
    foreach ($entry->get_value('sshPublicKey')){
        print $_ , "\n"
        } ;
}

$ldap->unbind;

Once the script is created, it can be called by adding “AuthorizedKeysCommand /path/to/script” to the sshd_config file. I also had to set the script to run as root by using the “AuthorizedKeysCommandUser root” command.

Next Steps

I want to improve this script in a few ways long-term…

  1. Since all of our Linux systems are part of our domain, there should be a way to have them bind to LDAP by using the machine’s Kerberos ticket. I don’t like using a username and password, but didn’t have the time to get the Kerberos bind working reliably.
  2. On the security front, this should be a TLS bind. No reason to have the data going over the wire cleartext.
  3. The script should not have to run as root…
  4. Cache the authorized_keys file on a per-user basis. We have a very robust AD infrastructure, but there is always a concern that it could become unavailable. The system’s resiliency would be greatly increased if it could cache the authorized_keys locally on a per-user basis, where sshd would normally look for it.
  5. Error Handling and Logging. It’s not fun, but it’s important. I wanted to get this solution out quickly, but it should be able to log to standard sources and handle some edge cases.
  6. Since the above is a lot of work, perhaps I can just improve a project like ssh-ldap-pubkey to support Kerberos.

 

External Links

I found the following links quite helpful in generating this solution.

Flexible Email Alerts for Logstash

November 13, 2015 at 5:02 pm

Logstash LogoMy company currently does a lot of it’s debug logging via email.  This means that every time an unhandled exception occurs in production, qa, uat, or integration, we get an email. Thank goodness for custom email rules and single instance storage in Exchange. Oh wait.

I have been a proponent of Logstash and the ELK stack for quite a while now. It is a wonderfully flexible framework for centralizing, enriching, and viewing log data. This past week, I built a proof of concept for management and they loved it. However, many folks wanted to know how we could send out emails from the logging system. I pointed them at the Logstash email output plugin, but they weren’t convinced. They wanted to see some flexible routing capabilities that could be leveraged in any config file, for any log type. Thankfully, this was pretty easy to accomplish.

Below I present a simple tag and filed based config for email notifications.

# This config is designed to flexibly send out email notifications 
# It *requires* certain fields to work 
# Create a tag "SendEmailAlert" 
# Required field emailAlert_to - the email address to send to 
# Required field emailAlert_subject - The subject of the email
# Required field emailAlert_body - The body, defaults to %message 
# 

output { 
  if "SendEmailAlert" in [tags] { 
    email { 
      address => "smtp.XXXXX.org" 
      username => "XXXXX" 
      password => "XXXXX" 
      via => "smtp" 
      from => "logstash.alert@XXXXXX.com" 
      to => "%{emailAlert_to}" 
      subject => "%{emailAlert_subject}" 
      body => "%{emailAlert_body}" 
      } 
   } 
} 

As the comments indicate, all you need to do is tag a message with “SendEmailAlert” and add the appropriate fields and voila: flexible email notifications. In order to use it, a simple mutate is all that is needed.

mutate {
    add_tag => ["SendEmailAlert"]
    add_field => { 
       "emailAlert_to" => "user@XXXXX.com"
       "emailAlert_subject" => "Test Alert"
       "emailAlert_body" => "%{message}"
    }
}

We could easily extend it further, but this has been fine for our POC thus far. We have also implemented similar notifications for Hipchat and PagerDuty.

What to do with an old Christmas tree farm?

October 21, 2015 at 4:29 pm
It's dark in there...

It’s dark in there…

As the missus and I sit and talk about our new homestead and the directions that we are thinking about taking it, one problem keeps coming up: the old Christmas tree stand. You see, dear reader, our homestead used to be a Christmas tree farm back in the 80s. Unfortunatly, the previous owners decided not to keep the farm going and let the trees grow up. On the surface this may not appear to be an issue, that is, until you consider planting densities.

Normal pine tree stands are planted at about 400-500 trees per acre. This allows for them to grow straight and healthy. Stands like that can be used for lumber and wood pulp and can net a good amount of money when they mature. However, Christmas tree farms are planted at 1,000 – 1,500 trees per acre. This is no problem if trees are kept small and regularly trimmed… Unfortunatly, that’s no the case here. Our stand is dense. It’s dark in there. This level of density leads to really unhealthy trees, and from the research I’ve been doing, it appears that there is not much that can be done.

It seems that our options are limited to the following:

  • Leave it be – The trees will keep growing, and will start dying off. This will likely result in a bad situation for both domestic and wild animals, not to mention the lack of productivity of that patch of the homestead.
  • Selective thinning – This would involve either getting a lumber/pulp company in to selectively harvest every other row of trees. This may not be an option because of the density. You can’t really get equipment in there. That means it might just be me with a chainsaw.
  • Harvest the whole thing – This is the option that I really don’t like, but seems to be the best all around. It would net some cash from the sale of the wood and would allow us to plant a new, healthy, forest and silvopasture using permaculture principles. The main problem here would be handling the stumps and the time it would take for a new forest to establish itself.

In case anyone is interested, I’ve also compiled a few links on the topic.

And here is a are some additional photos:

Finding a Home(stead)

July 12, 2015 at 3:36 pm

Finding a place to call your own is quite a step in life. To me, it means you’re ready to settle in, put down some roots, and potentially create a legacy. After discussing various options with my wife, we decided now would be a good time to consider buying a home. I thought would document the reasons for a the move for posterity’s sake.

Reasons

Investment

The thing that started us down this path was a simple realization: we pay a lot out in rent, and it would be really nice to “invest” that money instead. My wife and I currently live in a nice townhouse, in a good planned development near Princeton, NJ. Though we live within our means, we still pay out quite a bit for rent. For that, we get three bedrooms (really one bedroom and two offices), access to a pool, and lots of restrictions (including a no-charcoal-grill policy).

At the time, it made sense for us to move in to the development and not worry about anything. We could concentrate on paying off our debts, saving money, and helping my mother sell her old home in Queens, NY. Now that we’re done with all of that, though, it’s time to move on. Although many don’t consider a home an investment, putting equity in our pocket is certainly better than having it leave the family entirely.

Healthy, Self-Sufficient Lifestyle

My wife and I have recently started to adopt a “healthy lifestyle”. For us, that means eating more natural foods with fewer carbs and processed junk. The first thing we realized after making these changes was that the cost of high-quality food adds up quick. Then next thing we noticed was that we really did feel a lot better, so the premium was worth it.

Aside from food, we spend a lot of time commuting. Currently, my wife is commuting two hours, one way, by public transit, daily. This just isn’t healthy for so many reasons. I am in a better position, commuting a little over an hour one-way, by car, twice a day, but this takes quite a toll on us in many ways.

We don’t eat until 8:30-9:00 PM most nights. When you go to bed at 10 PM, this is a really bad thing. Second, we don’t have time for each other. Having an hour, two if you’re lucky, to catch up with your significant other is a terrible way to live a life. I love my wife, it’s why I married her! Finally, we don’t have time for our hobbies. My wife is a bit luckier in this regard as her hobby, knitting, is portable. I have taken to listening to podcasts on my commute as no one wants to listen to a table saw going at 9:30PM.

A Base of Operations with a Sense of Permanence

Living in a development has taught me two things: I love having someone else mow the lawn / shovel snow, and I hate not having a back yard to do projects in. Even with two offices, there’s no place for a shop, garden, or sheep. Yes, you read that right: sheep.

My wife is a knitter and she loves her hobby. She wants sheep, and I want her to be happy, so we need a place for sheep. My hobby is woodworking and building things. I’ve always been limited to a corner of a garage or basement, when I’ve been lucky enough to have a place at all. So room for a shop is a must for me. Finally there’s the garden. We had one in the past, and it was a wonderful experience. Now that we are focusing on healthier eating, what better way to get the best produce at the best prices than to grow it ourselves?

So on to the home search!