Grid Engine on the new Amazon Compute Cluster Instances

{ crossposted to x.bioteam.net and gridengine.info }

Amazon made a very important announcement today, releasing new EC2 server types and network configurations that significantly enhance the Amazon AWS environment for people who are interested in cluster computing, compute farming and high performance computing (HPC) on the cloud.

The announcement is here for those who are interested:

http://aws.typepad.com/aws/2010/07/the-new-amazon-ec2-instance-type-the-cluster-compute-instance.html?

I’m thrilled that this news is now public, the service is up for use and I can finally start testing, blogging and benchmarking in the “real” production environment.

In the next few days I’ll be blogging over on http://www.bioteam.net, concentrating initially on seeing how storage and storage IO speeds differ on the new instance types. For life science types like myself, one of the biggest hassles in the cloud is due to the fact that we tend to be more performance bound by the speed of storage and file IO than anything else. The 10GbE networking changes and non-oversubscription of the network links along with the ability to group nodes together may mean very very interesting things are now much more feasible on the AWS platform.

Because I’m going to first concentrate on storage and IO stuff on the new offering I wanted to quickly show Grid Engine running on the new server types.

Even a single node SGE cluster can do reasonable work now as the cc1 instance type includes a pair of quad-core Nehalem CPUs along with ~23GB memory and a 10GbE ethernet backend.

We will be blogging and talking much more about how to use Chef Server to orchestrate self-assembling Grid Engine clusters and compute farms on this new service but since that may not happen until later — I just wanted to throw up a teaser post showing SGE 6.2u5 running in single-node mode on the new HPC offerings from Amazon.

qstat output showing 16 CPUs (click for full-size):

sge-cc1-1.png

qhost output showing system resources (click for full-size):

sge-cc1-2.png

Filed Under: Employee PostsNews

Tags: , , ,

About the Author

Chris is an infrastructure geek specializing in the applied use of IT to enable and enhance scientific research in life science informatics environments.

Comments (1)

Trackback URL | Comments RSS Feed

  1. Ben Mabey says:

    Any updates on sharing your chef recipies for setting up grid engine? Even without a tutorial/blog post I think making the recipies available (via github or similar) would be a great contribution.

Leave a Reply




If you want a picture to show with your comment, go get a Gravatar.