See the talk at SC26: Thursday, November 19, 10:00 AM, AWS booth theater, Chicago.
Thanks for coming to our SC26 talk. Here’s the map, the checks, and where to read more.
From BioTeam’s talk at the AWS in-booth theater at SC26
Scientific teams keep asking us the same question: where should our workload run? A wrong answer shows up months later, as an idle cluster or a rebuild.
These are the questions we ask first. Answer them for one workload.
| Your workload looks like | Start here | Why |
|---|---|---|
| A core facility: one production pipeline, samples in waves | AWS HealthOmics | Scales to zero. Nothing to patch. |
| Research computing: Slurm jobs all day, a small platform team | AWS Parallel Computing Service (PCS) | Keep Slurm. Hand off the controller. |
| Licensed software and many queues, such as computational chemistry | Count your queues first | PCS if they fit. AWS ParallelCluster if not. |
For the director who signs off: every platform choice is a bet on what your data needs will be in 2030. Some bets you can undo. Some you can’t. A scoped proof of concept tells you which one you’re making.
None of these are blockers. They’re what you want to know in week one, not month six. Tick them off as you go.
If the checks don’t fit, AWS ParallelCluster is the self-managed path. Our write-up compares the two.
From BioTeam projects.
A protein tool downloaded its model weights at startup, the way it does on a laptop. On HealthOmics, the run had no route to the internet, so it failed. We built the tool into the container and staged the weights in S3.
For: teams running AI models in HealthOmics pipelines
Check: network access
Clinical genomics teams moving a pipeline to production often need a higher run-start quota before go-live. We’ve seen the increase disappear when a team splits staging and production into new AWS accounts. Quotas live in each account and each Region.
For: clinical and production genomics labs
Check: quotas, per account and per Region
We often find Slurm clusters where accounting purges its own records every month, or where Slurm doesn’t record GPU use at all. When the team asks whether it needs more GPUs, nobody can answer from data.
For: research computing teams on Slurm
Check: accounting
We’ve seen one wrong GPU entry in a Slurm configuration make a queue fail most of its jobs. The same kind of error made jobs draw more license tokens than the hardware called for. Lost science, and money spent on licenses.
For: Slurm sites with GPUs or licensed software
Check: custom Slurm settings
A licensed modeling suite needed a 1,278-character license configuration. That was 254 characters more than PCS supported. BioTeam brought AWS the use case, and AWS doubled the limit to 2,048.
For: teams running licensed software on Slurm
Check: custom Slurm settings
Is your cluster keeping its job history? Accounting is the first thing BioTeam asks for on a Slurm assessment, and we often find it purging its own records. These three checks take a few minutes on any self-managed cluster.
Run this on the host that runs slurmdbd. If PurgeJobAfter is set and ArchiveJobs isn’t yes, Slurm deletes old job records instead of archiving them. Only root or the Slurm user can read the file, so use sudo.
sudo grep -iE '^(Purge|Archive)' /etc/slurm/*dbd.conf
The file sits next to slurm.conf. On AWS ParallelCluster, use /opt/slurm/etc/*dbd.conf; the * also catches the included file that holds the database settings. No output means no purge is set: Slurm keeps every record, and the database grows until someone sets a retention period. Lines starting with # are switched off.
Look for gres/gpu in the output. If it’s missing, Slurm doesn’t record GPU use, and “Do we need more GPUs?” has no answer in your data.
scontrol show config | grep -i AccountingStorageTRES
This lists the three earliest jobs Slurm still remembers. If the oldest is only a few weeks old, your history is being deleted. Run it as root to see every user’s jobs. The command -v part finds sacct when sudo’s path doesn’t include it.
sudo "$(command -v sacct)" -a -X -S 2000-01-01 -n -P -o Submit,JobID | sort | head -3
Found a gap? Pick a retention period, turn on archiving before the next purge, and size and back up the archive. For example, to keep 12 months and archive anything older:
# slurmdbd.conf: archive old records
ArchiveDir=/var/spool/slurm/archive
ArchiveJobs=yes
ArchiveSteps=yes
PurgeJobAfter=12months
PurgeStepAfter=12months
# slurm.conf: record GPU use
AccountingStorageTRES=gres/gpu
The archive directory must exist and be writable by the Slurm user. Restart slurmdbd and slurmctld after the change.
On AWS PCS, accounting is one setting away. Turn it on when you create the cluster, or later, and PCS runs the accounting database for you.
Case studies and write-ups from BioTeam, plus the AWS pages we point teams to.
BioTeam moved an aging on-premises pipeline to WDL on HealthOmics, with containers, automated runs, and more than 100 refactored tests.
What we’ve learned moving pipelines, whatever the workflow language.
Public protein-science workflows, including ones BioTeam contributed. They stage model weights in S3, the pattern from the talk.
For runs that need to reach the internet through your own network.
The comparison behind “count your queues first.”
BioTeam rebuilt an on-premises Slurm cluster with high availability, while researchers kept working on it.
Current defaults for queues, node groups, and clusters.
The license string story from the talk. AWS has since raised the limit to 2,048 characters.
How a licensed suite’s recommended layout maps onto PCS queues and node groups. AWS has since made the queue and node group limits adjustable through a support case.
Licensed chemistry and notebook tools on Slurm in AWS, with jobs that wait for a free license seat.
Sixteen years of chemistry HPC, ending with a full cloud environment that deploys in about a day instead of weeks.
How we place workloads when the answer isn’t one platform.
Tell us about it. We’ll tell you which path we’d check first, and what a scoped proof of concept would look like.
BioTeam is scientists, data scientists, and technologists. We’ve worked where science meets IT since 2002, and we’ve run science on AWS since 2007. Scientists know what they want to do with their data. We make sure they can.
We’re vendor neutral, and we’re an AWS Partner with the AWS Life Sciences Competency. See BioTeam’s latest insights at bioteam.net/blog.