Local LLMs for real HPC work — a series
- Part I: Two Macs, a test harness, and the speed numbers (published 15-Oct-2026)
- Part II: Writing complex real-world AWS IAM policies (published 20-Oct-2026)
- Part III: Schrödinger postmortem root cause analysis (this post)
- Part IV: Letting a local model audit and fix a repo on its own (future post)
- Part V: Giving agents just enough tools: sandboxes, proxies and read-only MCPs in a test harness (future post)
TL;DR
Five real Schrödinger job failures from client clusters. Each model gets the job’s jsc postmortem bundle (logs, job records, subjob directories), a docs-search tool, and one instruction: find out why it failed, cite the log lines that prove it, and say what the underlying cause probably is. Three attempts per failure, per model. No AI judge: the answer is checked against the bundle itself.
- Claude Opus 5.5 passes 15 of 15. So does Swift1.5 (a fine-tune of Qwen3.8-Flash-Next), a local model tying Opus outright.
- The top six local models find the right cause on almost every attempt (mean scores 0.86–1.00). The thing that fails them is citation discipline: a log quote copied onto the wrong line, from the wrong file, with a
...in the middle, or stitched together from fields of a JSON log line. - No local model invented a log line. Of 503 local citations, 462 are exact; I checked the 41 that aren’t against the bundles, and every one is real bundle text copied badly.
- One mechanical repair turn fixes the copying. When a harness checks the citations and tells the model which ones are wrong (without telling it the answer), DeepSeek-V4-Flash goes from 9 of 15 on its first answers to 15 of 15. On first answers alone, one local model clears the bar I set for “usable”; with the repair turn, four do.
- Speed: Opus takes ~1.5 minutes per failure; the local models that tie or nearly tie it take 4–8 minutes on one Mac Studio. That is fine for a postmortem, which nobody needs in under ten minutes.
What ran
Same box and same models as Part II: one Mac Studio (M5 Ultra, 256 GB unified memory) behind oMLX, every local model a quantized community build (4 to 6 bits), Claude Opus 5.5 through the Anthropic API as the baseline. The model table, with links to every model card and quant, is in Part II’s “What ran” section. The agent harness is oh-my-pi, same prompt and tools for every model.
Why this task
Failed Schrödinger jobs are a regular ticket in our consulting work. The Schrödinger Suite is the commercial molecular-modeling platform a lot of pharma and biotech drug-discovery groups run: scientists build a calculation in the Maestro desktop and submit it, and the Job Server sends it to the cluster, where workflows like FEP+ free-energy runs fan out into many Slurm subjobs that can run for days. When one fails, the user sees “job failed”; the answer is somewhere in a postmortem bundle of a few hundred to tens of thousands of lines spread across the driver log, the Job Server log, per-subjob supervisor logs and JSON job records. Most of the work is reading: which subjob died first, what killed it, and what that implies about the cluster.
It is also the task where “where does this text go?” matters most. The bundles contain hostnames, paths, user names and scheduler details from client clusters. A model that runs on a box I own is the only way some of these bundles would ever be fed to a model at all.
Who is this for? HPC and Bio-IT folks who support Schrödinger, or any scientific software that runs under a scheduler and fails in ways that need log archaeology. If you have never looked inside a postmortem bundle, Schrödinger’s support article on how to generate one is the place to start: $SCHRODINGER/jsc postmortem <jobid> zips up everything the Job Server knows about a job (supervisor and launch logs, Job Server log entries, the failed subjobs), and the cause is in there, usually.
The test, briefly
Five failures, taken from eight real bundles from three client clusters (the three bundles of one ten-minute scheduler event count as one failure; one bundle with no defensible single answer was dropped). Four Schrödinger workflows are involved: IFD-MD (induced-fit docking with molecular dynamics), FEP+ free-energy perturbation and its force-field builder, and residue scanning in BioLuminate, Schrödinger’s biologics package.
| Failure | Job and task | What the bundle shows | What an expert should propose next |
|---|---|---|---|
| pm1 | IFD-MD, an MD subjob | Slurm sbatch fails with “I/O error writing script/environment” for an MD subjob |
the Slurm controller’s state directory is out of disk |
| pm2 | FEP+ core-hopping run | a subjob hits ENOSPC on its temporary directory after eight days | the user pointed -TMPDIR at a small local /tmp instead of cluster scratch |
| pm3 | BioLuminate residue scanning | one subjob aborts (SIGABRT) with a core dump | cannot be known from the bundle: get the subjob’s backend log or open a Schrödinger case |
| pm4 | FEP+ force-field builder | the builder is killed by the cgroup OOM killer | an 8 GB node is too small for the QM fragment jobs Slurm packed onto it |
| pm5 | FEP+ run | the driver is killed and requeued mid-run; the restarted backend fails on missing earlier-stage outputs | something outside the job stopped the node (maintenance) |
The answer is one JSON block: job type, failure category (one of twelve), the proximate cause, one to five pieces of evidence as {file, line, quote}, a root-cause hypothesis, and next steps.
There is no AI judge. A Python grader checks each answer against the pristine bundle:
- category is in the accepted set for that failure;
- evidence-verbatim: every quoted log line actually appears on the cited line (±2) of the cited file. This is the check that catches invented evidence;
- evidence-decisive: at least one verified citation is on a line that proves the cause (for pm1, the
sbatchI/O error, not a downstream “job failed”); - required facts: three or four per failure (for pm4, that it was the OOM killer and that the node was too small);
- forbidden causes: one or two known-wrong explanations per failure, negation-aware, so “no core dump” does not count as claiming a crash. The negation handling is a word-level rule, so it can misfire in either direction; the one misfire I know of is in a run not shown here.
A sample passes only when every check passes.
The model works inside a macOS sandbox: read-only access to its own copy of the bundle, no internet, and two network ports, one for the model and one for a proxy to our hpc-docs documentation server (Schrödinger docs, the Schrödinger knowledge base, Slurm docs, our consulting notes). Our hpc-docs index includes BioTeam’s own write-ups of two of these five incidents, so the proxy hides exactly those two documents. Otherwise a model could look the answer up.
Like Part II, the test is frozen once the reference run is registered. Before freezing I read every failed sample, Opus first, then every local model’s. That review is where most of the interesting findings came from.
Lessons learned first
- Local models are good at this. The top six locals score 0.86–1.00 on mean check score; what they get wrong is rarely the cause.
- Citation discipline is the gap, and it is mechanical. Opus copies every quote exactly, 75 citations out of 75. The local models copy 462 of 503 exactly, and every one of the 41 misses is a copying error; none is invented. A quote on the wrong line still fails the sample, because the next person reading the answer will go looking for that line.
- A linter-style repair turn closes the gap. One extra turn that lists the bad citations (and nothing else) rescued all six of DeepSeek’s failed samples without changing anything else in its answers.
- Two local models found a cause Opus missed. On pm5, MiMo and Qwen independently found evidence of a second explanation in the bundle that all three Opus attempts walked past. I had to widen the answer key.
- Context window size can decide a score. Laguna-S-2.1 scored 0.2 at a 262K context window because the harness’s compaction threw away the answer format; the same weights at 512K scored 0.86.
- Speed without reading doesn’t help. Inkling-Small used the same effort as Opus (96 seconds, 20 tool calls) and passed 2 of 15.
Results: correctness
Read the bar and the pass count separately. DeepSeek’s mean of 0.970 says it gets almost every check right; its 11 of 15 says four samples each had one bad citation. That is the whole story of the middle of this chart.
Swift, Qwen and MiMo are within noise of Opus at 15 samples: the 90% confidence interval on the paired difference in pass rate includes zero for all three (Qwen −0.067 [−0.209, 0.075], MiMo −0.133 [−0.307, 0.041], a t-interval over the five failures; Part II’s correction explains why not a bootstrap). GLM is clearly behind (−0.467 [−0.751, −0.182]) and DeepSeek only just (−0.267 [−0.533, −0.001]). GLM’s extra problem is specific: it called the pm5 driver’s clean exit an application crash in all three attempts (there was no signal and no core dump).
I set a “usable” rule before running any local model: pass rate within 0.15 of Opus and zero verbatim failures. A model that is right but cites the wrong line costs the reader the time it takes to discover that. On this board only Swift meets it. Qwen misses on one miscited path.
Every citation, checked
The local models made 503 citations across 105 answers (three answers had no parseable JSON at all). 462 are exact. The other 41: 5 with the right text a few lines off, 4 with the wrong file or a mistyped path, 12 elided with ... in the middle, and 20 stitched or altered (fields of one JSON log line glued together, a word or a path changed).
I checked all 41 against the bundles with a script: does the whole quote appear anywhere in the bundle, and if not, do its pieces? Every one is real log text copied imperfectly: either the whole quote exists elsewhere in the bundle, or every part of it does except one word, field or path.
Note: the prompt says to copy the line exactly and never to elide. Elision fails the check by design. “Stitched” is mostly JSON log lines: the model quotes a few fields of one record as if they were one contiguous string. It reads fine and won’t match a grep.
The case Opus missed
On pm5 the driver is killed and requeued, and the restarted backend dies looking for files from an earlier stage. All three Opus attempts stopped there: an external event stopped the node, the requeue lost state, rerun the stage. That was also my answer key.
MiMo (one attempt) and Qwen (one attempt) kept reading. Seconds after the restarted backend exited, the job server started failing every hosts lookup on a malformed entry, and the hosts file in the bundle had been edited inside the maintenance window. The more likely story is that the maintenance edit broke host lookup and the requeued driver died of that. Both explanations are defensible from the bundle, so the answer key now accepts either, with the hosts-file lines added as decisive evidence.
That is a local model reading a bundle more carefully than Opus did, on a real incident. It is one failure out of five, but it is the reason I read every failed sample before trusting any score: here the grader was wrong about two correct local answers.
The Opus review found two required-fact patterns that were too narrow and had rejected three correct Opus answers on pm4 and pm5. The local review found four more problems: a forbidden-cause check that matched “no OOM”, too few decisive lines on pm4, the pm5 answer key above, and one in the other direction, a mistyped relative path that the grader resolved to a different file with the same name. Every fix except that last one was in favor of a correct answer.
Results: speed
The local models read more than Opus does: two to three times the tool calls, and more documentation lookups. On this task that is a feature. Swift and Qwen at four and a half minutes per failure are well inside the time it takes a human to download the bundle and unzip it.
Inkling is the same cautionary row as in Part II. It works at Opus’s pace and Opus’s tool count and gets 2 of 15, because it reads too little of the bundle and paraphrases what it cites: 14 of its 63 citations are not exact, and on pm1 it never names the sbatch I/O error in any attempt.
A repair turn
Every failure in the middle of the correctness chart is the kind a linter catches. So I added one to the harness, as a separate version of the test with everything else byte-identical:
- The model answers as before.
- A checker that knows nothing about the right answer looks at the JSON: is it parseable, are the fields there, is the category one of the twelve, and does every quote appear on its cited line?
- If anything fails, the same session gets one more turn with the list of problems (“evidence 2: quote not on line 1750 (±2) of jobserver.log”), the rule, and an instruction to fix only what is listed. It never gets the correct line, the correct quote, or any hint about the cause.
- The grader scores the final answer, and separately scores the first answer, so every run measures both.
All six DeepSeek repairs took 47–124 seconds and changed only the flagged evidence entries; category, cause and next steps were untouched in every one. MiMo’s and Qwen’s single miss is on the same failure (pm1, one attempt each, never naming the full controller disk), a reasoning miss the checker cannot see by design.
With the repair turn, DeepSeek, MiMo and Qwen all meet the usable rule, and Swift already did without it. That is four local models I would hand a Schrödinger postmortem to, with the citations machine-checked before a human reads the answer.
Note: MiMo and Qwen needed no repair in this run, where in the earlier one each had a verbatim miss. Fifteen samples is not many; the earlier run is in the correctness chart above, and I’m not going to pretend either run is the “real” one.
The same repair turn on a harder task, a checklist audit of a whole cluster’s recon evidence, fixed every formatting and citation slip, and the best local model still stops at 9 of 12 against Opus’s 12, because the misses there are judgment calls. That one becomes a post if a local model clears the bar. A linter fixes copying. It does not fix reasoning.
Runtime churn
The model server and the harness both moved during these runs, as in Part II. Before trusting a comparison across the upgrade, I re-ran MiMo on the new stack: 0.978 mean and 12 of 15 against 0.926 and 13 of 15 before, with the same failure kinds in both runs. No evidence the runtime moved the score.
The server has moved again since (oMLX 0.7.3 development builds). On the first of those builds, all three local runs of this test had at least one first answer with a broken format; none of the three runs on the earlier release candidate did. That may be the server or may be chance at three runs, and the repair turn hides it either way, so I’m checking it before I trust a first-answer number from the new builds.
The Laguna result in the lessons list is the other runtime story. At a 262K context window the harness compacted the conversation in 11 of 15 samples, and the compaction summary dropped the required answer format, so seven answers had no JSON block at all. At 512K, zero compactions and a 0.86 mean. I moved three models to 512K that day.
What’s next
Part IV is the audit-and-fix task, where a local model edits a real infrastructure repo on its own and the repo’s own linters and tests decide whether it worked. An oMLX build that loads Nemotron 3.5 Super VL arrived after these runs. Its early results (2 of 15 postmortems on its first answers, 3 of 12 on the recon checklist even with the repair turn) made it clear it would not get close, so I pulled it from the testing pool instead of spending more Studio days finishing its runs. MiniMax-M3.1-Flash loads and passes its smoke tests as of 27-Oct-2026; its runs start when the Studio is back from client work.
Have a failed Schrödinger job of your own? Send me the postmortem.zip (dag@bioteam.net) and I’ll run it through the models. Scrub it first: jsc postmortem already replaces the job name, and -replace <string> hides hostnames, user names and project paths. It stays on the Studio, runs only on the local models, the answers go back to you alone, and I delete the bundle afterwards.
