From Four Guys and a Garage to Something Nice

October 19, 2016

WARNING:

This blog entry is very, very, very, very, very long.  It has a huge amount of images, so be forewarned.

Introduction

Well, as many of you know, we have always been kind of the underground heroes of the Internet.  We have scraped by with used gear, volunteers, and parts that fail often.  We have told stories about our failures and the tribulations of being a non-profit.  While none of that has really changed, we have finally gotten something nice.  Something very nice.  Our main corporate sponsor stepped up and really built something that would be impressive for just about anyone.

Our new Data Center is beautiful.

All the other issues still exist.  We are perpetually short of gear, bandwidth, money, and many other things.  But we do have something nice now.

The Old Data Center

The building that we had our old data center in was sold and we had to move.  Our old space was a re-purposed lab that was made for benches, not 80 racks of gear.  The number of times we blew circuits and almost set the building on fire occurred often enough that the electricians only showed up in full body protection suits:

img_1794

img_4246-768x576

So it was time to find a new space before we were trying to steal wifi from Starbucks for more bandwidth, and power from the poles directly.

The Old Space

Yeah.  It was ugly.  It was crowded and noisy.  It frequently had a temperature of 90F.  It was a pit.  But it was our pit.  Our beloved pit of gear, blood, sweat, and tears.

Ah yes, we do still have P3’s running….  almost 190 of them in fact.

img_5017-e1476811024172-1024x768

img_5015-e1476811034837-1024x768

The beautiful orange disk trays are ones that we designed and had printed because when we received some blades none of them came with disk trays.  And of course this specific vendor does not allow any third-party vendor from selling trays which means you can only pay the full amount for a tray with a disk.  So we took the easier route and just made our own out of printed plastic:

img_6933-1024x768

Which of course stand out nicely all over our server systems:

img_5014-e1476811043402-1024x768

In the background you can see the very large HVAC unit that is the additional cooling we had to add in to achieve something less than 90F.

img_5012-e1476811055998-1024x768

Truly impressive enterprise cooling, honest!

img_6885-768x1024

And of course more orange disk trays in the back…

img_5011-e1476811069140-1024x768

Our main HDFS storage cluster which is most of the row.  It currently sits at about 3 petabytes of capacity.

img_5010-e1476811079492-1024x768

On the left is an elastic search cluster, with our OSX known file hash whitelisting systems in the middle.

img_5009-e1476811086154-1024x768

Then there are the brutes of the Data Center.  Our blade servers which house the minions that do all the work for us:

img_0031-e1476812066878-768x1024

img_4953-e1476812024970-768x1024

Stuff in the Cupboard

Because we had run out of space and power, we actually could not turn on all the gear we had available.  As die hard techies and data hoarders, this made us very unhappy. The new data center space will help us reach our goal of actually powering up all of the storage and compute capabilities that we currently have.

Our storage room mostly emptied, but there always seem to be some lingering boxes.

img_4925-e1476812057607-1024x768

img_4947-e1476812050909-768x1024

Some additional blade servers we might be able to turn on.  Assuming we actually have all the parts.  We have a lot of chassis, but not all the power supplies or blades.  The joy of being an impoverished non-profit. But we can dream.

Getting Ready To Move

After lots of preparation and hard work, the big day would final arrive. But first that meant loading hardware onto pallets. Lots of pallets:

move1-e1476811384234-1

move2-e1476811392966

The New Data Center

There are a lot of different statistics that can be mentioned. Most can be meaningless.  But we’ll give the overview:

  • 750kW of power – They brought in a new 500kW transformer to the building for us in addition to us taking half of the building power as well for that total.  We have been strictly instructed that we are not actually allowed to utilize 100% of that or we will burn down the building.  The power is broken down into buses where the total load is spread out over the row of racks it services.  So we have to balance the high-powered gear over several buses as to not blow anything up.  Not that we have ever done that before…
  • 84 racks – The racks are not 100% filled, but will be crowded.  We will need to more seriously plan to do more in-place upgrades instead of just adding in new gear.  At this point, despite the move we will be very much still at the limits of capacity to grow larger.
  • 18 Inter-Row Cooling Units (IRCs) – three per-row. Water cooled units that are part of a Pod cooling system to try and maximize the cooling.  We should have sufficient cooling for 800kW of power.

Images of the new racks with the IRC’s:

img_4948-e1476812042236-1024x768

img_4949-e1476812034927-1024x768

Here you can see the Pod cooling design.  The idea is the make more efficient mini-rooms of heat that can be better controlled.

img_6863-e1476813768401-1024x768

You can almost see all three of the pods:

img_6864-1024x391

From inside one of the pods:

img_6866-1024x463

The front of a managed IRC:

img_6903-e1476813830950-768x1024

Holy cow Batman, Dr. Freeze must be here to keep it that cold!

img_6904-e1476813824658-1024x768

The Move

It was a lot of hands on deck.  We kidnapped helpers off the street, we dragged friends into dark rooms, we did what it took to get everything going and moving. No children or small animals were hurt in the process.

Preparation

Nectar of the gods:

img_1478-768x1024

Beverages of the lessor deities:

img_5008-e1476812758145-768x1024

Ambrosia for the masses:

img_5013-e1476812750617-1024x768

Now that we are properly caffeined and sugared up what could go wrong?

Packing and Shipping

img_4986-e1476812015318-768x1024

Okay – our packing abilities are not the best.

img_4987-e1476812007384-768x1024

It slowly is getting emptied.

img_5033-e1476813116688-1024x768

Hmm, someone needs to fill those.

img_5068-e1476813109805

And off more gear goes. Ideally to the right location.

img_6872-e1476813101955-768x1024

img_6876-e1476813093394-768x1024

The entire storage cluster has moved.  Hopefully it arrives.

img_6877-e1476813087143-768x1024

All those carefully packed pallets then need unloading again:

img_6880-e1476813079336-1024x768

Hey guys…  I think we could have packed those a little more?

img_6905-e1476813066220-1024x768

img_6906-e1476813059830-768x1024

Sugar is a very important ingredient to heavy loading.

img_6907-e1476813051576-1024x768

I am sure not a single person helping thought about doing this…  Until they saw the warning sign:

img_6912-e1476813043520-1024x768

Lift with your back!

img_6913-e1476813037264-1024x768

img_6914-e1476813028326-1024x768

You are going on a journey. A long, dangerous journey…

img_6915-e1476813019264-1024x768

img_6917-e1476813001205-1024x768

And Into the New Space

Mounting all those rails is never amusing or fun.

img_5058-e1476813627788-768x1024-2

Lift with your back!

img_5064-e1476813618113-768x1024-2

img_5067-e1476813611321-768x1024-2

img_6899-e1476813601627-768x1024 img_6900-e1476813595181-768x1024

img_6908-e1476813588320-768x1024

And it Continues:

As of 2016-10-18 we are only on the third day of the move.  We believe that all the gear that is needed for our operations will be moved by Friday and hopefully also brought online.  The clusters we already moved have survived and are working.  Score!  We still need to:

  1. Mount it all in the new racks
  2. Wire it
  3. Pray it all powers up
  4. Power it up
  5. Sacrifice some chickens
  6. Bring up the clusters
  7. Rinse and repeat until everything has been moved

What does this mean for you?

After over a decade of continually expanding our capabilities and giving daily free of charge remediation reports to thousands of constituents around the world, this has been our first ever week of downtime for a major data center move. Once we are fully back online and serving you all again, we hope to be coming back stronger than ever – with firmer foundations in place to power us through another few more years of growth. Expect to hear more in the coming months.

What can you do to Help?

As always, we love to get more data in to help victims, but we are always seriously in need of bandwidth – particularly in our new data center.  And donations help pay for the beast that is now our backend (we add another 2+ terrabytes of storage per day, just to keep up with new data arriving).  The Data Center costs are covered for now, but there is a lot more that needs your help.

As a US 501c3 non-profit organization that does not charge for its services, we survive through tax deductible donations, sponsorships, as well as funded project work to expand out what we are able to do.

We share our data for no cost with direct network owners and National CERTs all over the world. We do not ask for credit, only the occasional support. If you think you might be able to help us out, please contact us by email at admin*at*shadowserver*dot*org.

Recent Articles