SAA-C03 · Networking

Why can't my private subnet reach S3? Gateway and interface endpoints

With no internet gateway or NAT, a private subnet can still reach AWS services. A gateway endpoint adds a route; an interface endpoint adds an address.

5 min readVideo comingChecked against the AWS documentation on Ends at Shortcut to S3

An instance in a private subnet tries to write to an S3 bucket, and the request times out. Nothing is wrong with the bucket or the permissions. The request has nowhere to go. The fix is a VPC endpoint, and the two kinds work in completely different ways. One is a route. The other is an address.

Why the request has nowhere to go

Your code sends the request to the service’s usual name, such as s3.us-east-2.amazonaws.com, and that name resolves to a public address. The AWS PrivateLink guide says it plainly: “The default service endpoints are public interfaces, so you must add an internet gateway to your VPC so that traffic can get from the VPC to the AWS service.” A private subnet’s route table holds the local route and nothing that matches a public address, so the packet is dropped at the first hop.

A NAT gateway would get it there, through the internet gateway, but it also opens a path to the rest of the internet. If the VPC must stay off the internet, that answer is out. VPC endpoints reach the service without an internet gateway or a NAT device.

Gateway endpoint: a route in your route table

Gateway endpoints “provide reliable connectivity to Amazon S3 and DynamoDB without requiring an internet gateway or a NAT device for your VPC”. Those two services are the ones that offer them.

When you create a gateway endpoint, you choose route tables, and AWS adds a route to each one. “The destination is a prefix list for the service owned by AWS and the target is the gateway endpoint.” You can’t edit that route by hand; it comes and goes as you associate and disassociate route tables. And “Instances in subnets that aren’t associated with these route tables use the public service endpoint, not the gateway endpoint.” So a common failure is an endpoint that exists while the subnet’s route table was never associated with it.

The price is the headline: “There is no additional charge for using gateway endpoints.” The limits matter as much. A gateway endpoint “is available only in the Region where you created it”, and “Resources on the other side of a VPN connection, VPC peering connection, transit gateway, or Direct Connect connection in your VPC cannot use a gateway endpoint”. For those clients you need an interface endpoint, “which is available for an additional cost”.

Your instances’ security groups and network ACL must still allow traffic to the service.

Interface endpoint: an address in your subnet

Other services, such as Amazon SQS, use an interface endpoint, powered by AWS PrivateLink. Instead of a route, “For each subnet that you specify from your VPC, we create an endpoint network interface in the subnet and assign it a private IP address”. The local route already reaches that address.

What makes your code find it is private DNS. When private DNS is on, and the VPC has both DNS hostnames and DNS resolution enabled, AWS creates a hidden private hosted zone. It resolves “the default DNS name for the service” to the private IP addresses of the endpoint network interfaces. Your code keeps calling sqs.us-east-2.amazonaws.com and lands on the endpoint, “without requiring that you make any changes to those applications”.

Two more details trip people up:

  • The endpoint has security groups of its own. If you don’t choose any, “we associate the default security group for the VPC”. The endpoint’s group must allow the expected traffic from your instances. For SDK calls over HTTPS, that means “the security group must allow inbound HTTPS traffic”.
  • It is billed. “You are billed for each hour that your interface VPC endpoint is provisioned in each Availability Zone. You are also billed per GB of data processed.”

Amazon S3 supports interface endpoints too, for clients a gateway endpoint cannot serve, such as an on-premises network. From inside the VPC, the gateway endpoint does the job at no additional charge.

Endpoint policies: which buckets, not just which service

An endpoint reaches the whole service by default: “The default policy allows full access.” The endpoint itself does not stop requests to buckets in other accounts. An endpoint policy narrows it. The guide’s example “Restrict access to a specific bucket” allows actions only on the named bucket’s ARN, so requests for any other bucket through that endpoint are refused.

Don’t confuse it with a bucket policy that uses the aws:sourceVpce condition. That denies access to your bucket “unless the specified gateway endpoint is used”. It protects your bucket, not someone else’s, and it also blocks the Console’s access to that bucket. The endpoint policy protects the path.

What the exam guides ask

The SAA-C03 guide covers this in Task 1.2 “Design secure workloads and applications” and Task 4.4 “Design cost-optimized network architectures”. The SOA-C03 guide has Skill 5.1.2 “Configure private networking connectivity” and Skill 5.1.4 “Optimize the cost of network architectures”. The SAP-C02 guide has Task 1.1 “Architect network connectivity strategies” and Task 2.3 “Determine security controls based on requirements”.

Scenario questions on the topic usually describe private instances that reach S3 through a NAT gateway, or a VPC with no internet access, and ask for the most cost-effective or most secure design. The wrong answers follow a pattern:

  • A NAT gateway. It reaches the service through an internet gateway, and opens the rest of the internet with it.
  • An interface endpoint for S3, used from inside the VPC. It works, but it is billed by the hour and by the gigabyte. The gateway endpoint is not.
  • A gateway endpoint for a service other than S3 or DynamoDB. Other services take an interface endpoint.
  • A gateway endpoint used from on premises or across a transit gateway. It cannot be.
  • A security group rule to limit which buckets are reached. Security groups filter addresses and ports. Bucket names belong in the endpoint policy.

When you troubleshoot for real, check the endpoint type first. Gateway endpoint: is the subnet’s route table associated with it? Interface endpoint: does the name resolve to the endpoint, and does its security group let your instance in?

Play it

In Shortcut to S3, Northpine Labs runs a genomics pipeline in pipeline-vpc, which has no internet gateway and no NAT gateway and must stay that way. pipeline-worker, in a private subnet, takes jobs from an SQS queue and writes results to the northpine-results bucket. It must not copy data to any other bucket, including one in another AWS account, and the finance team sets a strict monthly budget for the VPC’s networking. The palette offers both endpoint types for S3, plus a NAT gateway and an internet gateway. Send traffic and watch where each request stops: at a route table with no match, at an endpoint’s security group, at the budget, or at a bucket it should never have reached. Shortcut to S3 is part of the paid Networking domain. The free first stage, Open for Business, is a Network build too and plays the same way.

Video comingA short walk-through of this stage in the game is being recorded. Until it is here, the stage itself is one press away.