AWS ParallelCluster Firewall Destinations
When Internet egress is blocked by default inside your cloud environment, what external sites and destinations need to be allowed?
What & Why
This is a quick late-2025 update to a personal blog post I wrote years ago over at https://bad.dev/blog/pcluster-destinations/ where I did some testing to see what external destinations were required to operate my Slurm HPC clusters on AWS.
I decided to revise and update this work in 2025 because I’ve encountered a few more clients who deny Internet access by default in their research & scientific computing VPC. Most of these clients are using Palo Alto Next-Gen firewalls or similar devices to screen traffic and only allow access to named fully qualified hostname (FQDN) destinations.
This time I broke the testing down into two different experiments in order to test both:
- ParalleCluster Image Build Pipeline (uses AWS Ec2 Image Pipeline to build custom AMIs for your HPC cluster)
- ParallelCluster HPC Cluster (the HPC cluster itself)
Methodology
Our testbed was pretty simple and may not be 100% accurate as not all software may honor system level HTTP proxy settings and may have directly tried to access the internet which would have been allowed in our VPC.
What we did:
Squid Proxy Server
We launched a small Amazon Linux 2023 instance and manually configured Squid as a simple proxy server for HTTP and HTTPS traffic. The important parts of the config file looked like this:
# Allow from VPC CIDR Range
acl vpc_cidr src xx.xx.xx.xx/yy
http_access allow vpc_cidr
# Multiple styles of log formatting so we have parsing options
access_log /var/log/squid/common.log common
access_log /var/log/squid/access.log combined
# Custom formatted proxy log
logformat minimal %tl %>a %Ss/%03>Hs %rm %ru
access_log /var/log/squid/minimal-access.log minimal
These are the proxy values we tried to configure into the Linux OS for each test. We also set the proxy values in the configuration files for yum, dnf and apt as needed.
http_proxy="http://172.31.87.105:3128"
https_proxy="http://172.31.87.105:3128"
no_proxy="localhost,127.0.0.1,169.254.169.254,*.ec2.internal,*.s3.amazonaws.com,*.amazonaws.com"
ParallelCluster Image Build Pipeline
Image Build pipeline does not explicitly support a config directive for proxy server usage so we had to manually configure things
- Launch a generic Linux OS AMI from AWS Marketplace (Amazon Linux 2023, Ubuntu 24.02 LTS, RedHat 9)
- Login to the running instance and configure proxy settings in multiple locations
- /etc/environment
- .bashrc
- Set proxy host entries in the appropriate package manager config file for Yum/DNF (Amazon Linux, RHEL), Apt (Ubuntu)
- Turn the lightly customized server into a new AMI owned by our AWS account
- Feed this “Parent Image” into the ParallelCluster build pipeline
- When pipeline exits, grab the squid proxy logs and save them for processing
The .yaml file fed into the Parallelcluster “pcluster build-image …” command contained build instructions to update OS packages and explicitly install GPU drivers and Lustre filesystem support. A generic version of our build file looks like this:
Build:
InstanceType: g4dn.xlarge # build on a GPU node to force nvidia install
ParentImage: ami-xxxx # our custom image configured to use proxy server
SubnetId: subnet-xxxx # HPC subnet
SecurityGroupIds:
- sg-xxxx
UpdateOsPackages:
Enabled: true
Installation:
NvidiaSoftware:
Enabled: true
LustreClient:
Enabled: true
Parallelcluster Deployment
Parallelcluster config files support Proxy directives so we also launched our clusters with that value explicitly set.
- Install the latest release of AWS Parallelcluster, at the time we wrote this post that was version 3.14.0
- Launch a quick cluster without an RDS database for accounting data
- Configure the HeadNode with a GPU in case NICE DCV server needs egress access
- Configure a cheap/small server in a 1-node interactive-q partition
- Run a quick job submission test, then stop the cluster and grab the proxy logs
- Destroy HPC cluster stack
The Ubuntu version of our proxy test cluster configuration file looked like this:
Image:
Os: ubuntu2404
CustomAmi: ami-xxxx
HeadNode:
InstanceType: g4dn.xlarge
Networking:
SubnetId: subnet-xxxx
SecurityGroups:
- sg-xxxx
Proxy:
HttpProxyAddress: http://172.31.87.105:3128
Ssh:
KeyName: pcluster-training-2025-common
Dcv:
Enabled: true
Iam:
InstanceRole: arn:aws:iam::xxxx:role/scientificComputing-default-Ec2-Role
Imds:
Secured: true
Scheduling:
Scheduler: slurm
SlurmQueues:
- Name: interactive-q
Networking:
SubnetIds:
- subnet-xxxx
SecurityGroups:
- sg-xxxx
Proxy:
HttpProxyAddress: http://172.31.87.105:3128
ComputeResources:
- Name: t3amed
InstanceType: t3a.medium
MinCount: 1
MaxCount: 1
Iam:
InstanceRole: arn:aws:iam::xxxx:role/scientificComputing-default-Ec2-Role
Monitoring:
Logs:
CloudWatch:
Enabled: true
RetentionInDays: 1
DeletionPolicy: Delete
Dashboards:
CloudWatch:
Enabled: true
DetailedMonitoring: false
Alarms:
Enabled: true
Results
ParallelCluster Image Builder External Destinations
##
## Internet destinations seen in proxy server logs for evaluation in potential firewall rules
##
# Ubuntu 22.04 LTS Image Builder Pipeline (pcluster v3.14.0)
api.snapcraft.io
dashboard.snapcraft.io
cloudfront.cdn.snapcraftcontent.com
d1uj6qtbmh3dt5.cloudfront.net
irc-apigee-service.emaasmw.intel.com
analytics.softwareimprovement.intel.com
security.ubuntu.com
esm.ubuntu.com
pypi.org
files.pythonhosted.org
static.crates.io
index.crates.io
# Amazon Linux 2023 Image Builder Pipeline (pcluster v3.14.0)
al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com
rhui.us-east-1.aws.ce.redhat.com # yes, really
# RedHat Enterprise (RHEL) 9 Image Builder Pipeline (pcluster v3.14.0)
rhui.us-east-1.aws.ce.redhat.com
ParallelCluster HPC Cluster Deployment External Destinations
One interesting pattern is the S3 dualstack URI seen for Amazon Linux 2023. The S3 destination shows up in our proxy logs because we did not configure a “no_proxy” statement for the pattern “*.s3.dualstack.<region>.amazonaws.com“. For people with S3 gateway endpoints in their VPC this is not an issue.
Other than the no_proxy config error that logged the S3 dualstack URI, it appears that ParallelCluster on Amazon Linux 2023 does not need any external, non-AWS destinations to deploy and function.
##
## External Destinations seen in proxy logs when running ParallelCluster v3.14.0
##
# Ubuntu 22.04 LTS
api.snapcraft.io
cloudfront.cdn.snapcraftcontent.com
dashboard.snapcraft.io
# Amazon Linux 2023
al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com
# RedHat Enterprise Linux (RHEL) 9
d2lzkl7pfhq30w.cloudfront.net
mirrors.fedoraproject.org
rhui.us-east-1.aws.ce.redhat.com
Other Considerations
AWS Services with multiple hostname patterns
Few people run VPCs where every AWS service is accessed via VPC endpoints or PrivateLink routes, so there is often traffic to AWS services that goes out over the egress network path.
In one past project we had a customer who was authorizing traffic to CloudFormation but did not also allow traffic to the “coudformation-waitcondition” endpoint which ParallelCluster needs to signal it’s progress through the stack set. It was important to authorize egress to both of these destination patterns for the CloudFormation service:
cloudformation-waitcondition-us-east-1.s3.amazonaws.com cloudformation.us-east-1.amazonaws.com
Route 53
We also often see research VPCs where DNS is centrally managed and thus the security guardrails assume “nobody will need Route 53” but this is not true for ParallelCluster by default which makes custom private zones when allowed and updates the zone each time a compute node scales in or out. This may mean that some may have to explicitly authorize:
route53.amazonaws.com
And finally if you want to see a list of amazonws.com destinations to compare against your VPC endpoints, PrivateLinks or firewall rules here are the hosts we saw in our proxy log server entries. Remember that we deployed a simple cluster without RDS and without fancy KMS encryption and other things enabled
Note: BioTeam makes heavy use of Amazon SSM, our IAM role allows SSM and we log into our HeadNode via SSM. Despite this no Amazon SSM traffic was picked up in our proxy logs, leaving us to believe that amazon-ssm-agent was communicating directly, bypassing the logging proxy.
## ##Amazon Destinations ## ## Cloudfront Distibutions d1uj6qtbmh3dt5.cloudfront.net d2lzkl7pfhq30w.cloudfront.net ## S3 Bucket Destinations ec2imagebuilder-toe-us-east-1-prod.s3.us-east-1.amazonaws.com fsx-lustre-client-repo.s3.amazonaws.com parallelcluster-xxxx-v1-do-not-delete.s3.amazonaws.com us-east-1-aws-parallelcluster.s3.us-east-1.amazonaws.com al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com ## CloudFormation cloudformation-waitcondition-us-east-1.s3.amazonaws.com cloudformation.us-east-1.amazonaws.com ## Other/Misc dynamodb.us-east-1.amazonaws.com ec2.us-east-1.amazonaws.com route53.amazonaws.com
