Judah Things

Category: Datacenter Diary

  • Refurbishing Old Switch

    Authors Note

    This post was abandoned in a pending state for a bit over a year. To be honest, since I was not successful I was a little disheartened from posting it. I changed my mind though, the post is worth publishing anyways, despite the failure. So here it is-

    You might have read in the datacenter diary that a switch was failing, it had a bad fan. Clearly the device had to go, but I had hopes of refurbishing it. If anything, its a test-run of doing more of this.

    Just out of curiosity I popped the lid open. The fans look like pretty standard 40mm fans. I can feel there is a ton of dust and gunk jamming one of the fans, so it’s definitely dead.

    Honestly its a lot smaller than I expected
    Looking at the fans model numbers

    It looks like these are readily available on AliExpress and eBay, so for $12 I grabbed a pair. They ship from mainland China so it took a week or five but it eventually arrived.

    Unfortunately, after powering the device back on there was a snap and a pop and a wisp of blue smoke. No activity on the fans and the console port won’t light up.

    It looks like this device is destined for the dumpster. They cant all be winners.

  • A Failing Switch

    Datacenter Diary #1

    Im starting off a new category on my blog, “Datacenter Diary.” It basically just chronicles my adventures in the “Server Canyon” (or other) realm.

    In this post I recount dealing with our first major hardware failure. Hardware failure happens all the time, but this was a single-point-of-failure device. Our main backbone switch.

    Network switch with a fan warning
    Fan light lit up but temp seems fine

    I’ll admit I might have been ignoring an ever-louder growing rattle for a few weeks, but when the warning light lit up it was time for action.

    I picked up a new (previously enjoyed and lovingly revitalized) switch from Unix Surplus. I have no special relationship with them, just a happy customer. I ended up with a fancy Dell/Force-10 S60-44T-AC-R. I picked this device because the Server Canyon network sees large bursts of traffic and this device has a pretty large packet buffer. Plus, it has two 10-gigabit ports that I will definitely take advantage of later.

    Force-10 glamor shot

    The first thing to do was copy the config from the old PowerConnect to the new Force10 switch. This turned out to be a little more difficult than expected since they were different product lines, but a switch is a switch and I got it sorted.

    Luckily, nearly every piece of server infrastructure has redundant uplinks. This means that I just needed to trunk the switches and start moving servers over. The management and client networks were only disrupted as long as it took to move the plugs over, but the storage network utilizes redundant links so no real downtime occured.

    Here you can see an in-progress picture of the migration:

    The process ended without much fuss, I only managed to miss tagging one VLAN.

    Next, a post-mortem on the old switch. That is a problem for another blog post though!