yes, indeed, it HAS been a while

YOU try building the biggest computer in the world. I've been so busy my brain is about to fall out of my head.

Anyways.

I just wanted to take a moment to say I have found one of the coolest programs in the world.

I was needing a way to load-balance our multiple login nodes, and searched for quite a while for a decent solution. I installed Linux-HA, and played with various other bloatware, but they all had too many levers and buttons to push. I just wanted a nice, simple piece of software that did what I wanted. And I found it!

You should check it out : Crossroads Load Balancer.

This guy is my new hero. It compiles quickly and successfully (why should that ever be an issue? but it is! getting Linux-HA installed required something like 28 perl dependencies, along with alot of other crap that necessitated upgrading almost the entire os. AND THEN you still have trouble getting it to work the way you want it to), and once installed, is very easy to configure.

Here is my configuration file, with important info changed to protect the innocent:

(to come later - the install I got going was on a test machine that someone turned off to make room for a grid cluster - I'm going to cannibalize that machine and use it's organs for more important business.)

If you are looking for something that just works, Use it.

cluster fork

You may or may not know, but cluster-fork is actually a command in the rocks universe. It allows you to run a command or set of commands on every machine in your cluster.

It's also a very apt way to describe why I haven't been updating this blog lately. I've been cluster-forked!

It's gotten to the point where I either do something useful, or update this blog. With the new machine, family and life, I've been afforded little time for such a thing as blogging. When I do get a chance, I will update as much as possible.

So, while I'm sitting here, I'll post a brief synopsis.

Uh, let's see.

We were testing and retesting the hardware, figuring out the correct bios settings to get the best performance out of the system, along with figuring out how to remotely manage and monitor thousands of machines in an easy and convenient manner. The blades are designed really well, and we are getting very nice performance percentages out of the new chips (wink wink). I know some of you are dying to know actual numbers, but I haven't had time to ask if I can write about it.

Additionally, we had to get the remote operating system install up and running smoothly, so that the blades come up uniformly and with only the necessary software and daemons to get the jobs run, when the hardware actually starts to arrive.

We installed all the disk servers with their necessary little bits and pieces, all the while learning the ins and outs of the unique and extremely engineered hardware from sun.

One might wonder how to go about controlling a massive cluster bigger than my house. Well, we're using sun's neat embedded service management voodoo hardware to monitor and remotely connect to the machines. Basically, it's a little computer that's embedded in the back of the machine you are running. It has it's own ip address, it's own processor, and can power on, turn off (gracefully or immediate) and monitor the server's health, all via ssh, or https(!).

There are 2 ways to do this, through ipmitool (a command-line interface to access, query and control the machines), or with the sun-produced browser-based java gui hoohadilly. I threw that last word in there for people who don't know what I'm talking about (hi mom!).

I use both. I am a strong advocate of command-line, script-based control of machines, and it will never be replaceable. However, I have taken a great liking to the java juju, I must admit. It's nice to pull up a browser and watch a computer boot in another place like I'm standing in the machine room with it. I can control it with both keyboard and mouse. It's a giant network-based kvm. I can access over 4000 machines from my one computer, and it's pretty cool.

Had to run the ethernet networking fabric for all of that. 12 48 port switches all connected to 2 24 port leaf switches that are uplinked via 10-GigE lines. That's alot of ethernet, btw. We run the lines, then velcro the bundles to the floor supports underneath our suspended floors.

Lots of 10-GigE cards installed. Lotsa fiber cards in PCI-E slots.

We have started receiving 18-wheeler shipments almost daily of huge amounts of hardware. 16 huge racks come in each shipment, and soon 200-400 blades will start arriving every day as well.

I'm sure that if our department weren't hosting two hpc conferences right now, we'd have actual photo evidence of all the work, but I will try to ask around for pics of what's going on.

This system didn't feel big until we got 45 racks and pushed those mammerjammers into place. That's when it felt huge. You can no longer walk in between rows through the spaces where the racks would go anymore, and the distance to go around to the sides, where the only openings are, is considerable. Now, you have to plan before you go somewhere, lest you have to turn back to get something you forgot. I'm not kidding. I jogged down the aisle and it took far too long to get to the end. It's really impressive.

Lots and lots of other details are being glossed over, because I am tired, and need to get up early to meet the next truck shipment in the morning.

more later...

*update
below, blogger says I posted this at 8:47 pm or something like that. that isn't true. it's 11:47pm. why must blogspot lie to the world and make little children cry?

PXE dust or, my boot over ib story

(editor's note - I wrote this during the night at midnight. upon further reflection, this is extremely stupid. but someone asked for an update, so here it is!)

One day there was a little infiniband card, freshly born into the world, wearing nothing but a blanket made of beautiful infiniband fabric.

This little card had no name, and was unable to get a name from his parents, for they could not speak his language. So he approached the gods in his area, and on bended knee, submitted a proposal that the god Mellan Ox strike him with lightning, and flash his memory so that he could speak to his parents, D.H. and C.P., in their native tongue.

So the great god Mellan Ox flashed his memory, and touched his tongue with a burning ember of rom, as well as a heaping pile of PXE dust, and he was thusly able to understand his parent's speech!

He felt very special, because there were not many other cards in the world that speak this language. His first order of business was to ask for a name. For this reason, he broadcast his message to his parents, shouting blindly from his crib in the night, but his parents could not hear him.

They did, however see that he was waving frantically to them from his crib. They did this by peering into a crystal ball called tcpdump, which told them not only his qpn number, but also the name given to him - his god-id, aka, guid. They were also able to see that he was asking for a name, but realized that the D.H. and C.P servant was blocking their ears.

They thusly asked their servant to re compile himself, which he did. This was achieved by removing the pounds of symbols near the rune-markings of his source-scrolls, allowing something called 'USE_SOCKETS'.

After the re-compile, the servant no longer prevented the little card's parents from hearing the little card's plaintive cries. D.H and C.P could now hear him asking for a name! They gladly responded, giving him the name '192.168.0.2'. Not original by any stretch of the imagination, but they heard that this new name was pretty popular, so they went with it.

Now, the little card was able to stand up in his crib, and speak with his parents very clearly. He was so happy that they could understand him! For the first time, he realized he was hungry, and asked for some food.

The first order of business was to put on his special tftp boots, which would allow him the strength needed to actually hold the platters of food that were soon coming.

He liked the taste of the PXE dust the god Mellan Ox had given him earlier, so he asked for some more. They brought it to him on a silver platter, along with heaping helpings of initrd meat and some binary image gravy.

He was able to put the initrd in his mouth and start chewing, but it wouldn't go down!

He was choking on the initrd because it didn't understand his delicate infiniband nature. The only thing that would help him digest the initrd meat would be the 7 tiny dancing ib drivers, who all dance on the head of a pin, and who don't really fit into the initrd meat without alot of effort.

Which is where our story will continue next time.....

back online

Ok, so I'm back from vacation, and back in swing on the new system.

We are in the midst of getting everything ready for when the large amount of hardware starts arriving. Fun things like filesystem structure, pxe booting over ib, provisioning the OS over ib using torrents, and the necessary setup that will easily allow us to install and keep track of 4000 machines and all their characteristics, much less making them all play nicely together.

The cabling alone will be a massive undertaking, and will need to be completed before the equipment begins to arrive. In fact, the filesystem hardware, all 1.7 Petabytes, will begin arriving in about a week or so. Not much time before the tsunami of hardware arrives.

I'm pretty jazzed about working on this new, cutting edge equipment. The constellation system, and all the new management hardware and software is really making my life much easier. I spend about half as much time in a 57 degree meat locker-like machine room, which means my body can get re-acclimated to a normal temperature again. I am constantly switching between a 50 degree difference in temperature. 57 inside, 107 outside. It's like a hot tub and cold dip, especially with the humidity lately.

geek gag order

This is from the comments in an earlier post:

Anonymous said...

Can you give us some linpack benchmarks on 'it'? I'm very curious.

July 24, 2007 8:23 AM

Super Geek said...

Ah yes, dear reader. I wish I could give you specifics, but I am under a very strict NDA, and if I were to put any results up here, I could strongly affect stock prices, either up or down, for this particular company.

So no, while I would LOVE to tell the world what I know, I just can't.

July 27, 2007 9:08 AM

vacation

been vacatin'

for geeks only

I have nothing to say but this:

Cpu0 : 0.4% us, 0.4% sy, 0.0% ni, 97.7% id, 1.5% wa, 0.0% hi, 0.0% si
Cpu1 : 0.3% us, 0.1% sy, 0.0% ni, 99.4% id, 0.1% wa, 0.0% hi, 0.0% si
Cpu2 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu3 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu4 : 0.0% us, 0.1% sy, 0.0% ni, 99.8% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu5 : 0.1% us, 0.1% sy, 0.0% ni, 99.7% id, 0.1% wa, 0.0% hi, 0.0% si
Cpu6 : 0.0% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu7 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu8 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu9 : 0.1% us, 0.1% sy, 0.0% ni, 99.5% id, 0.2% wa, 0.0% hi, 0.0% si
Cpu10 : 0.0% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu11 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu12 : 0.0% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu13 : 0.1% us, 0.1% sy, 0.0% ni, 99.7% id, 0.1% wa, 0.0% hi, 0.0% si
Cpu14 : 0.0% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si
Cpu15 : 0.1% us, 0.0% sy, 0.0% ni, 99.9% id, 0.0% wa, 0.0% hi, 0.0% si

Now, I can't tell you anything about this, but I can say it isn't intel, that it is really cool, and, not many people have been allowed to touch it or see it yet. But, here I am, touching all of it! Ha!

If you figured out what I've gotten a hold of, you can rest assured that no one will be disappointed with it, whatever 'it' is.