See the talk at SC26: Thursday, November 19, 10:00 AM, AWS booth theater, Chicago.

Thanks for coming to our SC26 talk. Here’s the map, the checks, and where to read more.

From BioTeam’s talk at the AWS in-booth theater at SC26

Two AWS Paths, One BioTeam Field Map

Scientific teams keep asking us the same question: where should our workload run? A wrong answer shows up months later, as an idle cluster or a rebuild.

Production pipelinesgo to AWS HealthOmics.
Slurmgoes to AWS PCS.
Either way, check before you commit.

Find your path in three questions

These are the questions we ask first. Answer them for one workload.

1. What kind of work is it?
2. Who runs it day to day?
3. How do you want to pay?
One more: licensed apps across many queues?

Your starting point shows up here once you answer questions 1 and 3.

The field map

Your workload looks likeStart hereWhy
A core facility: one production pipeline, samples in wavesAWS HealthOmicsScales to zero. Nothing to patch.
Research computing: Slurm jobs all day, a small platform teamAWS Parallel Computing Service (PCS)Keep Slurm. Hand off the controller.
Licensed software and many queues, such as computational chemistryCount your queues firstPCS if they fit. AWS ParallelCluster if not.

For the director who signs off: every platform choice is a bet on what your data needs will be in 2030. Some bets you can undo. Some you can’t. A scoped proof of concept tells you which one you’re making.

Check the terrain before you commit

None of these are blockers. They’re what you want to know in week one, not month six. Tick them off as you go.

AWS HealthOmics

AWS PCS

If the checks don’t fit, AWS ParallelCluster is the self-managed path. Our write-up compares the two.

What happens when a check gets skipped

From BioTeam projects.

The run that stopped before any science

A protein tool downloaded its model weights at startup, the way it does on a laptop. On HealthOmics, the run had no route to the internet, so it failed. We built the tool into the container and staged the weights in S3.

For: teams running AI models in HealthOmics pipelines
Check: network access

The quota that had to be raised twice

Clinical genomics teams moving a pipeline to production often need a higher run-start quota before go-live. We’ve seen the increase disappear when a team splits staging and production into new AWS accounts. Quotas live in each account and each Region.

For: clinical and production genomics labs
Check: quotas, per account and per Region

The history that deleted itself

We often find Slurm clusters where accounting purges its own records every month, or where Slurm doesn’t record GPU use at all. When the team asks whether it needs more GPUs, nobody can answer from data.

For: research computing teams on Slurm
Check: accounting

One config entry, two costs

We’ve seen one wrong GPU entry in a Slurm configuration make a queue fail most of its jobs. The same kind of error made jobs draw more license tokens than the hardware called for. Lost science, and money spent on licenses.

For: Slurm sites with GPUs or licensed software
Check: custom Slurm settings

The license string that ran long

A licensed modeling suite needed a 1,278-character license configuration. That was 254 characters more than PCS supported. BioTeam brought AWS the use case, and AWS doubled the limit to 2,048.

For: teams running licensed software on Slurm
Check: custom Slurm settings

Check your own Slurm cluster

Is your cluster keeping its job history? Accounting is the first thing BioTeam asks for on a Slurm assessment, and we often find it purging its own records. These three checks take a few minutes on any self-managed cluster.

  1. Purge and archive settings

    Run this on the host that runs slurmdbd. If PurgeJobAfter is set and ArchiveJobs isn’t yes, Slurm deletes old job records instead of archiving them. Only root or the Slurm user can read the file, so use sudo.

    sudo grep -iE '^(Purge|Archive)' /etc/slurm/*dbd.conf

    The file sits next to slurm.conf. On AWS ParallelCluster, use /opt/slurm/etc/*dbd.conf; the * also catches the included file that holds the database settings. No output means no purge is set: Slurm keeps every record, and the database grows until someone sets a retention period. Lines starting with # are switched off.

  2. GPU tracking

    Look for gres/gpu in the output. If it’s missing, Slurm doesn’t record GPU use, and “Do we need more GPUs?” has no answer in your data.

    scontrol show config | grep -i AccountingStorageTRES
  3. Your oldest job record

    This lists the three earliest jobs Slurm still remembers. If the oldest is only a few weeks old, your history is being deleted. Run it as root to see every user’s jobs. The command -v part finds sacct when sudo’s path doesn’t include it.

    sudo "$(command -v sacct)" -a -X -S 2000-01-01 -n -P -o Submit,JobID | sort | head -3

Found a gap? Pick a retention period, turn on archiving before the next purge, and size and back up the archive. For example, to keep 12 months and archive anything older:

# slurmdbd.conf: archive old records
ArchiveDir=/var/spool/slurm/archive
ArchiveJobs=yes
ArchiveSteps=yes
PurgeJobAfter=12months
PurgeStepAfter=12months

# slurm.conf: record GPU use
AccountingStorageTRES=gres/gpu

The archive directory must exist and be writable by the Slurm user. Restart slurmdbd and slurmctld after the change.

On AWS PCS, accounting is one setting away. Turn it on when you create the cluster, or later, and PCS runs the accounting database for you.

Read more

Case studies and write-ups from BioTeam, plus the AWS pages we point teams to.

Which workload costs your team the most time?

Tell us about it. We’ll tell you which path we’d check first, and what a scoped proof of concept would look like.

  1. Pick one workload, the one that costs your team the most time.
  2. Run it on one pipeline or one cluster.
  3. Measure cost, turnaround, and admin hours.
  4. Decide from the numbers. Expand, or switch paths. Either answer is a good outcome.

About BioTeam

BioTeam is scientists, data scientists, and technologists. We’ve worked where science meets IT since 2002, and we’ve run science on AWS since 2007. Scientists know what they want to do with their data. We make sure they can.

We’re vendor neutral, and we’re an AWS Partner with the AWS Life Sciences Competency. See BioTeam’s latest insights at bioteam.net/blog.

AWS Partner, Advanced Tier Services AWS Life Sciences Competency