High Speed Data Transfer

03 Apr 2012 High Speed Data Transfer

I recently had the opportunity to present a brief overview of some of the work that BioTeam has been doing with the New York Genome Center.  The format was a “webinar,” sponsored by Aspera and hosted by Bio-IT World.   I spoke briefly about the data motion needs of the center.  The most interesting piece to me was an example where we achieved sustained data transfers of 700Mb/sec coast to coast over the commercial internet.  At those data rates we do not need to rely on primitive hacks like fedexing hard disks.   Michelle Munson, CEO of Aspera, spoke second.  She explained how their software works as well as some of their plans for the future.

The entire presentation is available Under this link.

My slides are available as a PDF:  2012 Aspera Webinar

2 Comments
  • Indresh Singh
    Posted at 08:59h, 16 April Reply

    Hi Chris,

    Have you looked into transferring data/file using UDT (UDP-based Data Transfer)? It has much better performance then TCP and it’s free and open source.

    Cheers…
    Indresh

    • cdwan
      Posted at 09:05h, 17 April Reply

      I have to admit that I’ve been burned a number of times over the last few years by using software that has not had a broad industrial use case to flush out the bugs. For something as critical and low level as data motion, we really will not have time to help an open source project discover scaling problems, data corruption issues, or other “fun” problems.

      That said, it looks like UDT has been around for a while, at least in the Supercomputing community. I’ll check it out. Thanks for the pointer!

Post A Comment