Grid Engine on the new Amazon Compute Cluster Instances

13 Jul 2010 Grid Engine on the new Amazon Compute Cluster Instances

{ crossposted to and }

Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

I’m thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the “real” production environment.

In the next few days I’ll be blogging over on, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I’m going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later — I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):


qhost output showing system resources (click for full-size):


Related Posts
Filter by
Post Page
Employee Posts Tech Notes Training Presentations Events Screencasts
Sort by

Grid Engine for Users

2011-03-10 16:08:58


Grid Engine & Amazon EC2

I delivered the following presentation at the 2009 Sun HPC Workshop where
2009-09-09 09:10:43


SGE Video Tips: Portable Grid Engine

A short video showing how to use IP Aliasing to reliably run Grid Engine on laptop devices that may frequently
2007-12-15 19:12:58


  • Ben Mabey
    Posted at 18:10h, 11 January Reply

    Any updates on sharing your chef recipies for setting up grid engine? Even without a tutorial/blog post I think making the recipies available (via github or similar) would be a great contribution.

Post A Comment