Here we will explore three ways to connect two services on AWS, a retail shop and a pharmacy, and what each one means for security, cost and failure. The code for all three is at github.com/danieljohnmorris/floci-three-ways and runs locally on floci, the emulator covered in part one, so the whole thing can be reproduced without an AWS bill.
- Perimeter and addresses. Two VPCs, NAT gateways, WAF IP allowlists, API Gateway. Callers are identified by their NAT gateway’s public IP.
- Queues between VPCs. Same two VPCs, but the services never call each other. SQS sits outside both, with dead-letter queues and redrive.
- Identity over addresses. Callers are IAM roles. Security groups reference security groups. VPC endpoints replace NAT. EventBridge carries the async parts.
The two services
A retail shop sells things. Some of them need a prescription. A pharmacy holds prescriptions and patient records and is the side with the tighter access rules. The one interaction that matters: a customer orders a prescription-only item, retail asks pharmacy whether the prescription is current and belongs to this customer, pharmacy answers, retail fulfils or rejects.
Both services are small Hono apps on Node 22. One app object serves Lambda through hono/aws-lambda and a container through @hono/node-server, so the same code runs behind API Gateway or inside ECS. Each owns a Postgres database and creates its own tables on first use. Retail also takes a simulated payment webhook, verified by HMAC over the raw body, so each architecture has a third-party inbound to compare.
One scenario runs against all three: seed a prescription, order with the right customer and see FULFILLED, order with the wrong customer and see REJECTED, order a non-prescription item and see it fulfilled without asking pharmacy, post a webhook with a bad signature and get 401, then find the correlation ID from the first order in the retail Lambda’s CloudWatch logs.
A second suite reads the deployed configuration back through the SDK and asserts the security properties directly: that the pharmacy Web ACL’s default action is block and its IP set holds retail’s NAT address, that no security group admits 0.0.0.0/0, that every method on the pharmacy API is AWS_IAM. Those assertions are the same against the emulator and against a real account.
Architecture A: perimeter and addresses
Two VPCs, 10.10.0.0/16 for retail and 10.20.0.0/16 for pharmacy, each with public, private and isolated subnet tiers across two zones and one NAT gateway on an Elastic IP. Retail is a Lambda behind a REST API Gateway. Pharmacy is a Fargate service behind an internal ALB with an HTTP API in front. Each side has a REGIONAL Web ACL whose default action is block and whose one rule allows an IP set. Retail’s IP set holds pharmacy’s NAT address, pharmacy’s holds retail’s, and the API Gateway resource policy repeats the same address in aws:SourceIp. A third copy sits in the app as ALLOWED_CIDRS.
The caller’s identity is the address its traffic leaves from. The Elastic IP lives in the base stack, with RemovalPolicy.RETAIN, and the base stack has termination protection. If the address changed on every app deploy, the other side’s allowlist would be wrong after every deploy. I destroyed both app stacks and redeployed them to check: both NAT addresses came back unchanged, the retail allowlist was rebuilt holding pharmacy’s /32 and pharmacy’s holding retail’s, the three orders written before the destroy were still in Postgres, and the same seven scenario steps and 22 configuration assertions passed again.
Four stacks deploy in about 35 seconds once the images are built. Base stacks publish their IDs to SSM parameters and app stacks read them back through AWS::SSM::Parameter::Value<String>, so nothing crosses between stacks as a CloudFormation export and an app stack can be deleted without an export lock.
A’s standing cost is two NAT gateways and two Elastic IPs before any data. Every new caller means an edit to someone else’s IP set. Third-party inbound, the payment webhook here, arrives from addresses you do not control and have to keep current.
Architecture B: queues between VPCs
Same two VPCs, no NAT gateways. Neither service has an inbound listener from the other. Retail puts a PrescriptionCheckRequested message on one queue, pharmacy consumes it and puts PrescriptionCheckCompleted on a second, retail consumes that through a Lambda event source mapping and applies the result. Both queues are KMS-encrypted, redrive to a dead-letter queue after three receives, and carry a queue policy naming only the two roles. Each VPC reaches SQS, Secrets Manager and CloudWatch Logs through interface endpoints, so traffic to AWS never leaves the VPC.
Ordering POST /orders returns PENDING and the test polls GET /orders/:id until it flips. The same seven steps pass, plus 24 configuration assertions.
In B the caller’s identity is an IAM principal. With floci’s IAM enforcement on and two service users whose policies grant only what each side needs, the denials are real:
$ aws sqs receive-message --queue-url .../ftw-b-check-request # as retail-svc
An error occurred (AccessDeniedException) when calling the ReceiveMessage operation:
User is not authorized to perform: sqs:ReceiveMessage
$ aws sqs purge-queue --queue-url .../ftw-b-check-request # as pharmacy-svc
An error occurred (AccessDeniedException) when calling the PurgeQueue operation:
User is not authorized to perform: sqs:PurgeQueueRetail can send and nothing else. Pharmacy can receive and reply and cannot purge. The test key does all of it, which is the bypass part one describes.
A poison message with no customer or SKU took about 90 seconds to exhaust its three receives at a 30 second visibility timeout, then appeared in the dead-letter queue. One start-message-move-task put it back:
$ aws sqs start-message-move-task --source-arn arn:aws:sqs:eu-west-2:000000000000:ftw-b-check-request-dlq
{ "TaskHandle": "task-292d7b6f-…" }
$ aws sqs list-message-move-tasks --source-arn ...
"Status": "COMPLETED", "ApproximateNumberOfMessagesMoved": 1Nothing timed out and nothing cascaded. The order sat at PENDING and the queue depth was the signal. The alarm to set is on ApproximateAgeOfOldestMessage.
The trade is that the trust boundary moves from “who can reach me” to “what is in this message”. Pharmacy validates every field before acting, and both consumers dedupe on an idempotency key in a processed_messages table, because SQS standard queues deliver at least once. A compromised retail could flood the request queue. It could not read pharmacy’s database or call an endpoint pharmacy did not choose to expose.
Architecture C: identity over addresses
One VPC, 10.60.0.0/16, private and isolated tiers, no NAT gateway, six interface endpoints and an S3 gateway endpoint. Six security groups carry seven ingress rules, every one of them sourced from another security group: the pharmacy ALB admits the retail Lambda’s group, each database admits its own app’s group, the endpoints admit both app groups on 443. There is no CIDR anywhere inside the VPC.
Pharmacy sits behind an internal ALB and a REST API Gateway where every method is AWS_IAM and the resource policy names one principal, the retail Lambda’s execution role. Retail signs its outbound call with SigV4 using that role, and publishes an order event to an EventBridge bus with a seven-day archive. A rule fans the event out to a queue with its own dead-letter queue. Nothing consumes that queue in this build. It is where an audit or analytics consumer would attach, and adding one changes nothing on the ordering path. The only public edge is retail’s customer API behind a Web ACL running the AWS managed common rule set and a rate limit, with no IP allowlist at all.
Adding a caller means adding a role ARN to the pharmacy resource policy. Rotating a caller means changing a role. A third party still hits the public edge, where the webhook’s HMAC signature does the work an IP list did in A.
The same seven steps pass, and the configuration assertions confirm what the design says: the check method is AWS_IAM, every ingress rule in the VPC is security-group sourced, describe-nat-gateways returns nothing, and the rule’s target queue has a dead-letter queue. That is 22 assertions.
Destroying the two app stacks left the VPC, all six security groups with their seven rules, both databases, both secrets, the bus and all three queues. Redeploying took about ten seconds.
This is also the shape closest to how I run personal projects on a single Docker network behind one reverse proxy: services addressed by name, nothing exposed except the edge. The CDK version adds a real identity layer, IAM, on top of the same picture.
What the emulator could and could not show
Everything above deployed and ran on floci. It could not demonstrate three things, and part one explains why.
The WAF allowlists in A never blocked a request. The AWS_IAM on pharmacy’s API in C never blocked an unsigned request: a plain container on the same Docker network got {"ok":true,"service":"pharmacy"} from it without a signature. And IAM enforcement in B and C was action-level, so a policy naming a queue or a bus ARN was denied and only the account-wide wildcard allowed. The events:PutEvents denial in C came with the same log line as SQS: resource=*.
For each of those the repo asserts the configuration instead: Web ACL default action and IP set contents, method authorisation type, security group rule sources, NAT gateway count. Those assertions run unchanged against a real account, which is where the enforcement happens.
The emulator also taught me things about my own CDK. CloudFormation on floci cannot update a stack, so a second cdk deploy over an unchanged template tried to delete and recreate it and left UPDATE_ROLLBACK_FAILED. cdk deploy --exclusively on the app stacks is the only safe redeploy. Dynamic references like {{resolve:secretsmanager:…}} arrive in Lambda and ECS environment variables as literal text. AWS::SQS::QueuePolicy and AWS::WAFv2::IPSet report CREATE_COMPLETE and create nothing readable, so B carries a small custom resource that calls SetQueueAttributes with the same document, and A carries one that creates the IP set and Web ACL through the SDK. Both are clearly labelled and both are switchable off for a real account.
Stateful things do not belong in the app stack
Every build has two stacks per service for the same reason. The base stack holds the VPC, the database, the secrets, the log groups and, in A, the Elastic IP, with retention policies and termination protection. The app stack holds the compute, the API, the queues and the roles, and can be destroyed and recreated at will.
I have lost a database to a stack delete before, and I have seen an Elastic IP released by a stack change and every allowlist that named it go stale at once. Splitting the stack in two prevents both. SSM parameters rather than CloudFormation exports are the other half: an export locks the exporting stack until every importer is gone, which turns a routine app redeploy into a dependency puzzle.
The destroy-and-redeploy check in each build tests that. The base stacks survived, the app stacks came back, and the data was still there.
When is an Elastic IP required?
Only when something outside your control identifies you by address. In A that is every callee’s IP set, so the NAT Elastic IP is load-bearing and must outlive the app stack. In B and C nothing identifies a caller by address, and there is no Elastic IP to protect.
Third parties that insist on a fixed egress address still exist. That forces one NAT Elastic IP per VPC whatever the architecture, and it is the one reason left for having one. Inbound fixed addresses are a different question again: an ALB has none, an NLB with an Elastic IP or Global Accelerator does, and neither was needed here.
Side by side
| A. Perimeter | B. Queues | C. Identity | |
|---|---|---|---|
| Caller identity | NAT Elastic IP | IAM principal on the queue | IAM role on the API |
| Breaks when an IP changes | Every allowlist naming it | Nothing | Nothing |
| Third-party inbound | IP list in the stack | Not applicable | Signature at the edge |
| NAT gateways | 2 | 0 | 0 |
| Elastic IPs that must survive a stack delete | 2 | 0 | 0 |
| CIDR-sourced security group rules | 0 inside the VPC, allowlists at the edge | 2, VPC range to each endpoint group | 0 |
| Failure mode | Timeout cascades | Backlog and dead-letter queue | Timeout cascades on the sync path, bounded on the bus |
| Redrive | None | start-message-move-task | Bus archive and replay, plus a DLQ per consumer queue |
| CloudFormation resources | 142 | 139 | 114 |
| CDK lines | 700 | 683 | 749 |
On cost, eu-west-2 prices from the AWS Pricing API: a NAT gateway is 0.05 USD an hour and 0.05 per GB, an interface endpoint is 0.011 an hour per zone and 0.01 per GB, a public IPv4 address is 0.005 an hour. A’s two NAT gateways and two addresses come to about 80 USD a month before any data. B and C each run six interface endpoints across two zones, about 96 a month before data. At idle the endpoints cost more than A’s two NAT gateways and two addresses. Per gigabyte they cost a fifth as much, and the traffic never leaves the VPC.
A is the shape I have run in production. It carries a list of other people’s IP addresses in its source code, and each of those addresses breaks the allowlist when it changes. B removes the list by removing the call. C removes the list by making the caller an IAM role. Choosing between those two is choosing whether the interaction is a question that needs an answer now or an event the other side can process when it gets to it. Prescription checks before fulfilment are the first kind, so C is where I would take a design like this next, with B’s queues for everything that is not on the ordering path.