Local LLMs for real HPC work, Part II: writing complex real-world AWS IAM policies

Local LLMs for real HPC work — a series

• Part I: Two Macs, a test harness, and the speed numbers (published 15-Oct-2026)
• Part II: Writing complex real-world AWS IAM policies (this post)
• Part III: Root-causing failed Schrödinger jobs from the logs (future post)
• Part IV: Letting a local model audit and fix a repo on its own (future post)
• Part V: Giving agents just enough tools: sandboxes, proxies and read-only MCPs in a test harness (future post)

---

TL;DR

This is an AWS-specific test. Seven AWS IAM policy-writing tasks, all taken from real AWS ParallelCluster work, each answer checked by AWS IAM Access Analyzer and the AWS IAM policy simulator against 71 to 223 allow/deny scenarios. Five answers per task, per model.

• Claude Opus 5.5 scores 0.992 and passes all 35 samples. On the hardest task it fell into one trap in 2 of 20 attempts across my reference runs, so the task discriminates even at the top.
• The best local models score 0.83–0.87 (MiMo-V2.6-Flash, DeepSeek-V4-Flash, Swift1.5, Qwen3.8-Flash-Next). On the six everyday tasks they sit at 0.94–0.98, close enough that I would use them with review.
• On the hard task, a 6,100-character AWS IAM permissions boundary for a shared production account, locals pass 1 sample in 35. They allow another team’s EBS snapshot to be copied, let an AMI be shared, blow past AWS’s 6,144-character cap on a managed policy, or never produce a policy at all.
• Speed is a separate question. Opus takes ~2 minutes per policy and ~9 on the hard one; the local models that are closest on correctness take 5–16 minutes per policy and 36–121 minutes on the hard one, on one Mac Studio.

The deep details on the policies themselves are in the ParallelCluster guardrails post. This post only explains what the test needs.

---

What ran

Everything local ran on one Mac Studio (M5 Ultra, 256 GB unified memory, ~243 GiB usable by models) behind oMLX, an MLX-based model server. Every local model is a quantized community build from Hugging Face; the number after “oQ” is the main bit-width (4, 5 or 6 bits, with attention and routing kept at 8 bits). Nothing here is an unquantized model, and nothing dense above 27B fits the box: the biggest thing in the test is a 320B-total / 18B-active mixture at 5 bits.

Model	Params (total / active)	Quant	Resident
MiMo-V2.6-Flash-MOPD (Xiaomi)	309B / 15B	oQ4e	160 GB
DeepSeek-V4-Flash-0731	284B / 13B	oQ4e	154 GB
Swift1.5-Qwen3.8-Flash-Next (ukisai fine-tune of Qwen’s model)	125B / 6B	oQ4e	100 GB
Qwen3.8-Flash-Next (Alibaba)	125B / 6B	oQ4e	99 GB
GLM-5.3-Flash (Z.ai)	320B / 18B	oQ5e	209 GB
Laguna-S-2.1 (poolside)	118B / 8B	oQ6e	86 GB
Inkling-Small (Thinking Machines)	276B / 12B	4-bit, experts only	143 GB
Claude Opus 5.5 (baseline)	—	Anthropic API	—

Part I has the hardware, the server and the model cards. The agent harness is the same for every model: the same prompt, the same tools, the same scorer, the same five samples per task.

The question I’m really asking

A reader of Part I put it better than I had: I’m not really testing Qwen or MiMo. I’m asking whether a small organisation with a modest purchase can functionally replace a frontier-model subscription for real engineering work. The combination under test is local open-weight models plus limited infrastructure: one box that sits on a desk and runs quietly off a regular outlet.

Here that box is one 256 GB Mac Studio. My answer so far, from the thread where this came up: “yeah, maybe — but incredibly slow relative to Anthropic”, and “the trend is positive”. This post is the first task-specific evidence behind both halves of that sentence.

Who is this for? HPC, Bio-IT and cloud engineers who write least-privilege AWS IAM policy for a living, or who are deciding whether a Mac Studio is a sensible thing to put on an ops team’s desk. If you have never written an AWS IAM policy, the guardrails post explains the problem; this one assumes you have.

The test, briefly

An AWS IAM policy is the JSON document that says what an AWS identity may do: which actions, on which resources, under which conditions. Each task is a written brief, the way a colleague would describe the need: what the role must be able to do, what it must never do, the naming conventions, the account layout. The model answers with a short rationale and one complete policy document. It has read-only reference tools: AWS’s published Service Authorization Reference of actions, resource types and condition keys, plus Access Analyzer’s policy validator to check a draft before answering. I would never ask an engineer, human or model, to write AWS IAM policy without the documentation open, so that is the arm I report. (A closed-book arm existed in earlier versions; Opus refused the hardest task closed-book, and I dropped it.)

The seven tasks:

Task	What the AWS IAM policy is for	Source
CLI operator, base	the role that creates, updates and deletes Slurm clusters with the ParallelCluster CLI	public AWS reference policy, corrected
CLI operator, IAM mode	the extra IAM rights that role needs to create its own instance roles, without being able to escalate itself	public AWS reference policy, corrected
CLI operator, image build	the role that bakes a custom cluster image with EC2 Image Builder	public AWS reference policy, corrected
Operator in a shared account	an operator who may touch only its own stacks, with Lambda, Image Builder and log changes allowed only through CloudFormation	deployed, sanitised
Image-build boundary	the permissions boundary (the ceiling) on the image-build machine	deployed, sanitised
Deployer with bounded roles	a deployer that may create or change roles only if each new role has one of two approved boundaries attached	deployed, sanitised
Workload boundary	the boundary on every workload role of the platform (head and compute nodes, AWS Backup, image builds) in an account other teams share: eight deny statements, tag-on-create, ownership of every EC2 resource it changes, 6,100 of the 6,144 characters AWS allows	deployed, sanitised; two independent reviews; CloudTrail-verified dry runs

“Sanitised” means every account number, name and ARN replaced by fictional values (111122223333, acme-hpc, /lab/). The deployed policies went through seven cluster create/delete cycles on a client engagement before they became test answers. The three public-reference tasks exist because the AWS documentation’s policies disagree with what a deployment needs, which was the first lesson of this whole exercise.

There is no AI judge. Each answer is scored by AWS’s own tools, read-only: is it valid (Access Analyzer, 15 %), does it allow what the job needs (policy simulator, “must allow” scenarios, 30 %), does it block what it must (simulator, “must deny” scenarios, 35 %), and does it grant anything beyond the reference (Access Analyzer’s policy comparison, 20 %). The worst deny scenarios are marked critical: a policy that lets one of those calls through is treated as unsafe, and its score cannot go above 0.25 out of 1.0 no matter how good the rest of it is. A policy that does too little gets the same ceiling, so denying everything is not a way to score well. The hard task alone has 183 allow and 40 deny scenarios, 18 of them critical.

The test is frozen: once the reference model’s run was registered, any change to a task, a scenario or the scorer is a new version, and every model here was scored on the same one. Before freezing, I reviewed the reference model’s misses and then every local model’s, and fixed more than a dozen places where the scorer was unfair to correct alternatives. The scorer also has to classify 21–48 deliberately broken policies per task (over-broad, under-broad, invalid, escalating) correctly before a version can be frozen.

Lessons learned first

• The everyday AWS IAM policies are within reach of local models. On the six tasks that are not the workload boundary, MiMo, DeepSeek, Swift and Qwen average 0.94–0.98 against Opus’s 0.99. With a human reviewing the diff, I would use them for this work today.
• The hard policy separates everyone, including Opus. The workload boundary is deny-heavy and full of resource-ownership reasoning, and the real limiter is size: AWS caps a managed policy at 6,144 characters and the correct answer needs 6,100 of them, so every statement has to earn its place and there is no room for a safe but verbose version. Locals pass 1 sample in 35. Opus passed 5 of 5 on the frozen run, but across all 20 of my reference-run samples it fell into the standalone-CreateTags trap twice. The task is hard for a reason: the deployed policy went through two review rounds and a live dry run before we got it right.
• Local models fail in three ways, and none of them is “wrong action name”. When the model can look actions and condition keys up and validate its draft, we don’t see invented action names or made-up condition keys; the policies are valid AWS IAM. What we see instead are capability failures: (1) allowing an adoption or exfiltration path (copying another team’s snapshot, sharing an AMI, tagging a foreign resource), (2) writing a correct policy that is too long for an AWS managed policy, and (3) running out of output budget before producing a policy at all.
• Speed and correctness pick different winners. The fastest local model (Inkling-Small, 1.6 minutes per policy) has the lowest score (0.569, 10 critical misses). The highest-scoring local model (MiMo) takes two hours on the hard task.
• A fine-tune that promises fewer thinking tokens kept its parent’s score and saved about a quarter of the wall clock on the hard task, nothing on the easy ones. Which is fine: a free speed-up that costs no correctness is worth having.
• The runtime moves under you. During these runs the agent harness and the model server both got upgrades. I re-ran the best local model on the new stack before trusting any comparison across the upgrade; details at the end.

Results: correctness

The same numbers as a table, with the hard task separated out:

Model	Mean [90% CI]	Pass rate	Critical samples	Six everyday tasks	Workload boundary	Δ vs Opus [90% CI]
Claude Opus 5.5	0.992 [0.990, 0.994]	1.00	0 / 35	0.99	1.00	—
MiMo-V2.6-Flash	0.866 [0.786, 0.937]	0.86	4	0.98	0.20	−0.126 [−0.347, −0.010]
DeepSeek-V4-Flash	0.861 [0.779, 0.932]	0.83	4	0.95	0.34	−0.131 [−0.293, −0.023]
Swift1.5	0.845 [0.758, 0.918]	0.83	2	0.95	0.20	−0.147 [−0.350, −0.015]
Qwen3.8-Flash-Next	0.830 [0.735, 0.914]	0.83	3	0.94	0.15	−0.162 [−0.371, −0.015]
GLM-5.3-Flash (5-bit)	0.695 [0.577, 0.807]	0.63	2	0.81	0.00	−0.296 [−0.513, −0.113]
Laguna-S-2.1	0.597 [0.442, 0.749]	0.52	3 / 21	0.67	0.15	−0.412 [−0.607, −0.223]
Inkling-Small	0.569 [0.466, 0.667]	0.57	10	0.64	0.15	−0.423 [−0.616, −0.239]

Three things to read off that table.

The top four locals are not distinguishable from each other at five samples per task. The differences between MiMo, DeepSeek, Swift and Qwen (0.83–0.87) are smaller than a single critical miss on one task, which moves a task mean by 0.15. Every one of them is distinguishable from Opus: the confidence interval on the paired difference excludes zero for all seven locals.

The six everyday tasks are where the local models earn their keep. MiMo at 0.98 and the Qwen family at 0.94–0.95 on those six means a few sufficiency misses (a describe call the simulator wanted allowed, a log-group ARN form) and the occasional extra statement, caught by the containment check. Reviewable, fixable, the kind of thing I correct in a human’s draft too.

GLM-5.3-Flash’s problem is specific: it writes correct policies that are too long. All five of its workload-boundary answers were valid JSON at 6,254–8,028 characters against a 6,144-character cap, and the cap is not negotiable: it is the limit AWS puts on a managed policy. The 4-bit build of the same model scored 0.80 on the five original tasks where the 5-bit build scored 0.77 (overlapping intervals, and the 4-bit run predates two scorer fixes), so I can’t blame the extra bit for anything. Nor can I credit it.

Note: Laguna’s row covers 21 samples. On the base CLI task it hit the one-hour-per-sample time budget on every attempt, looking actions up one at a time (up to 407 tool calls on a single sample), and those samples are excluded rather than scored as zero. That is the honest number, and it is also the slowest model in the table.

Results: speed

Brian’s comment on Part I was “let’s not confuse speed with functionality”, so speed gets its own chart and its own verdict.

Model	Median minutes, all tasks	Median tool calls	Median minutes, workload boundary	Longest sample
Claude Opus 5.5	2.1	16	8.7	11 min
Inkling-Small	1.6	8	1.6	2 min
Swift1.5	4.9	35	35.6	37 min
GLM-5.3-Flash	5.1	21	52.9	82 min
Qwen3.8-Flash-Next	6.1	31	39.6	48 min
MiMo-V2.6-Flash	11.5	26	120.9	160 min
DeepSeek-V4-Flash	16.4	27	63.6	84 min
Laguna-S-2.1	34.4	58	74.8	180 min (time budget)

On correctness alone, MiMo, DeepSeek, Swift and Qwen are usable for the everyday policies. On the clock, Swift and Qwen are the ones I would actually wait for: five or six minutes a policy is a coffee; MiMo’s two hours on the hard task is an afternoon. Opus does the whole seven-task set in about an hour of wall clock; MiMo needs roughly thirteen. One Studio serving one request at a time is the constraint, and nothing in this post changes that.

Inkling-Small is the cautionary row. It is the fastest thing I have run and it behaves like Opus (eight tool calls, short answers, no dithering), but it decides without looking things up: it allowed ssm:SendCommand to every instance in the account in 4 of 35 samples, reads of other teams’ S3 buckets in 3, and direct Lambda, launch-template and SNS calls that the task says must go through CloudFormation. Speed here is skipped verification. I dropped it from the Studio after the run.

The hard task, sample by sample

Of 35 local samples: 1 pass (DeepSeek, 0.967, at 6,069 characters and 64 tool calls), 15 critical escalations, 8 over the cap, 5 too narrow to do the job, 6 with no policy in the answer at all. The critical misses cluster on resource ownership, not on the trap I expected:

• d-share-image (make an AMI public or shared): 9 misses across the models.
• d-createvolume-account-snapshot / d-snapshot-foreign-volume / d-snapshot-foreign-instance (create from, or snapshot, something another team owns): 14 misses combined.
• d-createtags-standalone-createaction (the adoption trap): 5 misses, plus Opus’s 2 of 20.

The pattern is the same in every failing local answer I read: the boundary allows CreateSnapshot, CopySnapshot, ModifyImageAttribute and friends on resources tagged lab:managed, and forgets that the tag condition says nothing about who owns the source. The deployed policy handles it with ec2:Owner, aws:ResourceAccount and a multi-key negated condition on account-owned AMIs. Those are exactly the statements that cost us a review round as humans.

The “no policy” samples are a budget problem, and partly mine. Each sample gets 64,000 output tokens and three hours. Qwen’s and Laguna’s no-policy answers ran the token budget out while still reasoning; one of Laguna’s produced a 683-character fragment that parsed, allowed nothing, and so earned the 0.25 that a safe-but-useless policy gets. I kept the 64K cap for comparability across the series. A 128K cap might turn one or two of those into scored answers; it would not turn them into passes, because the samples that did finish failed on ownership.

Opus, for the record, needed 12–18 tool calls and 8–11 minutes per sample on this task, and its two misses across 20 samples were both the Null ec2:CreateAction presence test described in the guardrails post, which the policy simulator does not catch and a live dry run did.

Runtime churn

Part I’s “obsolete by the time you read this” section was not a joke. Between the first and last runs in this post, the agent harness went from 18.4.8 to 18.12.2, the model server from a dev build to a release candidate, and three models moved from a 262K to a 512K context window. Any of those could move a score, so before adding the two newest models I re-ran the best local model on the new stack: audit-and-fix came back identical (96/96 before and after), the postmortem task came back inside its own confidence interval, and the six everyday AWS IAM tasks matched the earlier run to within 0.01 per task (the workload-boundary rerun is still going as I write this). The leaderboard rows record which harness and server each run used, and I treat any comparison across that line as suspect until the rerun says otherwise.

One concrete thing the new harness surfaced:

That is a model behaviour (the Qwen family’s known tendency to loop inside a tool argument), not a harness bug, and it is the kind of thing a sample-level retry exists for.

What’s next

Part III is the Schrödinger postmortem task, where the local models do much better and one of them ties Opus. Nemotron 3.5 Super VL goes through this same test as soon as the model server can load it (the MLX support arrived on 19-Oct; the server release that bundles it has not). The hard task stays as it is: I would rather a benchmark that nothing local can pass yet than one I keep making easier.

Questions, or want the scoring scenarios for your own policies? dag@bioteam.net.