Typically a large computer cluster has a single head node and many compute nodes. These nodes are interconnected via a fast dedicated network (e.g. Gigabit Ethernet or Infiniband). The head node typically has two network adapters which bridges the WAN (or Internet) and the cluster.

For a two-node Beowulf cluster setup however, the above requirements (e.g. one head node and many compute nodes) can be greatly relaxed. Particularly in a two node environment, we do not have the luxury of having a dedicated head node and we have to use both nodes as compute nodes in the cluster. As a result, one of the two nodes will be both the head node and the compute node. As you will see later, the difference between the head node and the compute node in a two node cluster is not all that significant. Furthermore, since the cluster sits behind a firewall, there is no need to bridge the head node from the outside world via a second network adapter. The DSL router is sufficient to isolate the LAN from the Internet. This drops the two NIC requirements.

1. Choosing an MPI (Message Passing Interface) package
There are primarily two popular MPI implementations: MPICH and LAM/MPI. Some technical and performance differences exist between these two implementations (you can find more information in this research paper. This link has some high level information). Since MPICH2 implementation relies solely on SSH and does not require a daemon running on each machine in the cluster, it is presumably easier to setup in an environment that involves a large number of nodes. But for my particular setup, either implementation should serve the purpose. I chose to use MPICH2.

2. Setting up the MPI Environment
The two PCs (zeta, sigma) I use for the cluster both have a quad core processor (Q9450) and 8GB of RAM. The hardware configurations are slightly different (one Motherboard is based on Intel X38 chipset and the other one is based on P43) and the OSes are slightly different as well (one is running 64bit Ubuntu 8.04 and the other is running 64bit Ubuntu 8.10). None of these differences are material though.

In my setup, I used zeta (Ubuntu 8.04) as the head node since that is where my primary desktop is on. My new built (sigma, Ubuntu 8.10) runs in headless mode and serves as a compute node.

There is a lot of information on the Internet on how to setup an MPI environment. But most articles are rather dated. For Ubuntu cluster setup, this article is pretty much up-to-date (the article mainly targets Ubuntu 7.04 and there are some places that need to be updated and I will point those out as we move along). Some of the best references on MPI environment setup come from the MPICH2 distribution tar ball:

mpich2-1.0.8/doc/installguide/install.pdf

mpich2-1.0.8/doc/userguide/user.pdf

It definitely pays to read them through prior to setting up the environment. There are many useful trouble shooting tips in the installation guide as well, and they will come in handy should you run into any setup issues.

3. Setting up NFS on the Head Node
The installation guide that came with the MPICH2 distribution points out that NFS is not necessarily required for MPICH2 to function, but it would be necessary to duplicate the MPICH2 installation and all the MPI applications in the same directory structure on each node. Things become so much easier if all the nodes can use the same NFS directory where MPICH2 installation and MPI applications are. See Step 2 in the MPICH Cluster article.

4. /etc/hosts, mpd.hosts and SSH Setup
As indicated in the article I mentioned earlier, we need to change /etc/hosts on both nodes to contain the IP addresses of each server.

Pay special attention to the 127.0.1.1 address that points to the server name. For MPICH2 to work properly, this IP address must be changed to the real IP address of that server. For instance, here is the /etc/hosts on zeta:

127.0.0.1    localhost

#127.0.1.1    zeta

192.168.1.67    zeta

192.168.1.68    sigma

And similarly here is the /etc/hosts file on sigma:

127.0.0.1       localhost

#127.0.1.1      sigma

192.168.1.67    zeta

192.168.1.68    sigma

You will also need to create a mpd.hosts file on each of the nodes within the user’s home directory (note, this is the user you wish to run mpi under). Here is my mpd.hosts (I only have two machines in the cluster):

zeta

sigma

MPICH2 requires trusted SSH among compute nodes and head node. Please see Step 7 in this article I mentioned earlier. Note that in Ubuntu 8.04 and 8.10 the authorized keys file is called authorized_keys2 and the actual keys generated using ssh-keygen by default are id_dsa and id_dsa.pub. It does not really matter what type of keys you choose. Either DSA or RSA should be fine. This step should be repeated on each cluster node and the id_dsa.pub generated on one machine should be added to authorized_keys2 on the other machine. In my setup, I added the id_dsa.pub on zeta to the authorized_keys2 on sigma and vise versa. This is where we make distinction between  head node and compute nodes. Since head node needs to communicate with all the compute nodes, it needs to have all compute nodes’ public keys. Whereas each compute node only needs to have the head node’s public key. As you can see, in a two-node scenario, either node can be used as the head node.

5 Building and Installing MPICH2
The MPICH Cluster article on Ubuntu documentation has some great details on this so I am not going to repeat here. One thing you need to pay special attention is the .mpd.conf file (located in user’s home directory).

This file is used to make the communications among cluster nodes more secure. To ensure that computers within the same cluster can communicate with each other, each machine must have the same secret word. If the secret word does not match, you will receive a handle_mpd_output error (e.g. failed to handshake with mpd on <node name>; recvd output={})

6 Test Run
On zeta (head node) export the NFS node

exportfs -a

On sigma (compute node) mount the NFS exported from zeta on the same path so that the directory structures on both machines are the same.

sudo mount zeta:/mirror /mirror

On zeta (head node) start up the mpd on both nodes according to mpd.hosts

mpdboot -n 2

Right now running mpdtrace on either machine you should see both nodes get listed.

Running the interactive version of cpi on one core with an interval number of 100,000,000

mpiexec -n 1 ./icpi

it took 1.93 second to complete.

Running with all 8 cores with the same paramter:

mpiexec -n 8 ./icip

took only 0.31 second which is more than 6 times faster than the signle core version.

The following figure recaps the structure of my cluster setup:

ASUS P5QL PRO uses Intel’s P43 chip set which uses significantly less power than previous P3x ones. I chose one of the cheapest cases (Rosewill RZS05 SLV) since this PC I am building would not be the primary one. I also chose one of the cheapest and yet most energy efficient video cards (EVGA 256-P2-N429-LR GeForce 7200GS). I could have selected a G43 based mother board with integrated video, but it seems that my choice of G43 boards was quite limited. Price wise, the P43 plus video card solution is actually quite comparable to that with G43. And I believe that my solution only uses a few more watts than what could be achieved with G43. As a benefit of choosing a dedicated video card, the video performance is for sure to top that of a G43′s.

My previous built used two G.SKILL F2-6400CL5D-4GBPQ RAM (8GB). Since I do not plan to overclock this time, I chose two G.SKILL F2-6400CL5D-4GBNT(8GB) instead. Both of these RAMs has 5-5-5-15 timing and I believe that the only difference is that the latter does not have heat spreaders.

Here is the full list of the components I chose:

CPU: Intel Core 2 Quad Q9450
Mother Board: ASUS P5QL PRO
Video Card: EVGA 256-P2-N429-LR GeForce 7200GS
Memory: 8G (4GBx2) G.SKILL DDR2 SDRAM DDR2 800  F2-6400CL5D-4GBNT
Harddrive: Western Digital Caviar SE16 WD6400AAKS 640GB
DVDRW: LITE-ON 20X DVD±R DVD Burner Black SATA
Case: Rosewill RZS05 SLV Silver Sturdy SGCC Steel ATX Mid Tower
Power Supply: Antec Earthwatts EA500 500W 80 PLUS

This time I used the stock CPU heat sink. Running at stock frequency (2.66GHz), the stock cooler handles just fine.

ASUS P5QL PRO

When powered up, this configuration uses less than 100W of power (compared to the 200 plus watts my previous build uses)

CPU frequency:    3.200 GHz
Number of CPUs: 4
Number of threads: 4
Parameters are set to:Number of tests :  1
Number of equations to solve (problem size) : 10000
Leading dimension of array : 10000
Number of trials to run  : 10
Data alignment value (in Kbytes) : 1024

Maximum memory requested that can be used = 801248576, at the size = 10000

============= Timing linear equation system solver =================

Size   LDA    Align. Time(s)    GFlops   Residual      Residual(norm)
10000  10000  1024    16.427     40.5946  1.012665e-10 3.570760e-02
10000  10000  1024    16.398     40.6676  1.012665e-10 3.570760e-02
10000  10000  1024    16.395     40.6740  1.012665e-10 3.570760e-02
10000  10000  1024    16.473     40.4833  1.012665e-10 3.570760e-02
10000  10000  1024    16.391     40.6852  1.012665e-10 3.570760e-02
10000  10000  1024    16.394     40.6785  1.012665e-10 3.570760e-02
10000  10000  1024    16.427     40.5970  1.012665e-10 3.570760e-02
10000  10000  1024    16.397     40.6712  1.012665e-10 3.570760e-02
10000  10000  1024    16.394     40.6766  1.012665e-10 3.570760e-02
10000  10000  1024    16.396     40.6733  1.012665e-10 3.570760e-02

Performance Summary (GFlops)

Size   LDA    Align.  Average  Maximal
10000  10000  1024     40.6401  40.6852

End of tests

It is amazing to see that a personal PC nowadays can achieve 40+ GFLOPS! To put this number in perspective, take a look at the TOP 500 super computer ranking back in 2005. At that time this was the performance of a super computer (Since the benchmarking program used here is different and the conditions under which the tests are performed are not necessarily the same, direct numerical comparison might not be meaningful. Nevertheless, the general trend still holds.)!

It is also worth noting that Q9450 is an extremely overclockable CPU. To achieve a 3.2GHz core frequency, I only needed to raise the Front Side Bus (FSB) frequency from the default  333MHz to 400MHz (vcore is set at 1.2V manually). I was able to achieve a maximum of 3.4GHz without any stability issue. In fact, the only thing keeps me from achieving a higher clock rate is my DDR2 800 RAM (4x2GB, G.SKILL F2-6400CL5D-4GBPQ). With DDR2 RAM, Q9450 can easily archive 50 GFLOP in LINPACK test.

Since I have never been a fan of overclocking, and non of the component was chosen for such a purpose, I wasn’t keeping my hope high at all. As it turned out however, Q9450 seemed to overclock extremely well. By just bumping FSB from 333 MHz to 400 MHz, I was able to run it at 3.2 GHz without any issues. That’s a 20% jump and is quite impressive. What is more impressive is that this is achieved with mostly main steam components: ASUS P5E with 8GB (4GBx2) G.SKILL F2-6400CL5D-4GBPQ, Arctic Cooling Freezer 7 Pro. Even though I did not use Intel’s stock cooler, Arctic Freezer 7 is by no means a massive cooling solution and I am confident that Q9450 is able to achieve this frequency with the default heat sink as  well.

Without any voltage adjustment, I was only able to bring it to a maximum FSB of 430 MHz (3.44 GHz). At 440 MHz (3.52 GHz) I was able to boot but the system became somewhat unstable. I think more or less this was due to the fact that the DDR2-800 RAM was 10% overclocked at that point.

Given the above preliminary results, I am pretty sure that going above 4 GHz is definitely achievable with proper cooling and better RAM.

Q9450 Retail Box

Q9450 Retail Box (Top)

Intel Q9450

Intel Q9450

Arctic Cooling Freezer 7 Pro Versus Stock HS

I cannot believe how small the stock cooler is. It’s barely the height of Freezer 7 Pro’s bottom heat pipe (and Freezer 7 Pro can only be considered as an average cooling solution)! Intel’s new High-K 45nm process lowered the overall processor heat dissapation significantly, so it’s not surprising that the heat sink of this new chip is smaller than previous ones.

Stock HS

Stock HS

8GB (4GBx2) G.SKILL F2-6400CL5D-4GBPQ

8GB G.SKILL F2-6400CL5D-4GBPQ works quite well here. In fact, no voltage adjustment was necessary for the motherboard (Asus P5E) to recognize it. But G.SKILL does recommand manually setting the RAM voltage at 1.9V (the voltage range for F2-6400CL5D is 1.8 ~ 1.9) to ensure maximum stability.

The BIOS came with the motherboard is version 303, which is quite old (released 10/31 last year). Nevertheless, I booted up with Q9450 with no issue except an unsupported CPU message at boot time. I then flashed the BIOS to version 605 which officially support this stepping of Q9450. I  am not sure why Asus distinguish on its website a "DOS" version of the BIOS, maybe it’s named specifically for it’s DOS utilities to recognize it. Anyway, you can safely flash it with the EZFlash utility inside the BIOS.

I spent a few hours last night, quickly put my new system together. By the time I was done, it was already late. So I have not got a chance to install OS yet. Nevertheless it posted first time without any issues.

Here is a list of the parts I used in my new system:

• Case/Power: Antec TITAN650 (with 650W power supply)
• DVDRW: SAMSUNG SH-S203N (SATA)
• Video Card: EVGA 256-P2-N751-TR GeForce 8600GT 256M
• Motherboard: ASUS P5E (X38)
• Harddrive: Western Digital Caviar SE16 WD6400AAKS 640GB
• Memory: 8GB (4GBx2) G.SKILL F2-6400CL5D-4GBPQ
• CPU: Intel Q9450
• CPU Cooler: Arctic Cooling Freezer 7 Pro

Like my previous builds, I tend to put more emphasis on processing power and memory than the graphic system as I only need the horsepower for scientific calculations and data processing.

I had planned to use ASUS P5K-E WIFI-AP motherboard originally, but I noticed that there is still no current BIOS support for Q9450 (C1).

Anyway, in the next couple of days I will post more information on my new build and possibly a few pictures as well.

Many forum posts have suggested that Q9450 and X3350 were largely identical and had recommended buying the cheaper server version of the chip (X3350) instead of Q9450. But are they really identical as claimed by so many people? I decided to figure it out myself.

After some digging, I found the data sheet for both Q9XXX and X33XX on Intel’s web site. After comparing the two specifications, it became clear that while the majority characteristics of these two chips are identical, there are some differences never the less. Here are some of the highlights:

Pin differences:

The following are the pins that Q9450 has but not defined for X3350

• DPRSTP# (Deep Sleep State to Deeper Sleep State transition when asserted)
• DPSLP# (Sleep State to Deep Sleep State transition when asserted)
• SLP# (Sleep State)

The  absence  the above three signals in X3350 is not a surprise at all since it is a server class CPU, which usually means 24×7 operation.

A few test pins are different as well:

On Q9450, according to Intel’s documentation on the placement of pull-up resistors, TESTHI10 pin cannot be grouped with other TESTHI pins. While on X3350, this limitation applies to not only TESTHI10, but TESTHI11 and TESTHI13 as well.

Other electrical differences:

Q9450:

ICC_VCCPLL ICC for PLL land — 260 mA

RON Buffer On Resistance 7.5 – 11 Ω (GTL+ Signal Group)

X3350:

ICC_VCCPLL ICC for PLL land — 130 mA

RON Buffer On Resistance 7.49 – 9.16 Ω (GTL+ Signal Group)

 

So, it does seem that the two processors are largely compatible. However given the TEST pin pull-up register placement differences, it is possible that certain workstation motherboards’ (e.g. workstation bords not specifically stated supporting X3350) designs do not meet Intel’s guideline for X3350 and may result some stability issues.

The minor current requirement should not be an issue in my opion, but the missing SLEEP states pins in X3350 could result in some stability issues as well, since those pins are reserved in X3350 and reserved pins in general requires a static reference voltage or simply unconnected. But on a board build for workstations specifically, those pins may change states as the processor goes into and comes out of sleep state.

To summarize, I personally would not recommend putting a non-officially supported CPU in for reasons stated above. In the case of Q9450 and X3350 though, the differences are minute enough and from a technical stand point, it is possible that most motherboards that support Q9450 may also be compatible with X3350. Of course, even a motherboard is electronically compatible with X3350, we would still need the blessing of the BIOS. And since X3350 is officially unsupported, such blessing is totally at the mercy of each motherboard manufacturer and can not be guaranteed.

I guess I will stick with a Q9450. Hopefully there will be more in stock soon as the IDF starts next week.

Despite the reports however, it seems that the new Q9300, Q9450 and Q9550 are nowhere to be found. Typically, one would guess that the release of any new products, especially the new quad cores, would be covered with much fanfare. But this time, it seems a bit odd that there was nothing in the news at all.

Only a handful of the sites list these processors. The big ones like NewEgg hasn’t shown any yet (NewEgg did drop the price of the current quad on 3/16, which might suggest that they had already got the shipment from Intel). But most of them list them as back ordered and even the ones listed on Intel’s official site do not seem to have these processors in stock. And the listed prices were way above the $316 mark.

I have been waiting for the arrival of these 45nm quad cores (I am thinking of getting a Q9450) for a while, and I might have to wait a few more weeks before it becomes broadly available.

See also Technical Difference Between Q9450 And X3350