Private Link is the IP filtering of the cloud

2022/01/30

Tags: privatelink network

Use cases for Private Link and differences in its implementation across the major Cloud Providers.

Changelog

Private Link allows secure connectivity to services hosted on different Cloud provider accounts.

Private Link

Note on terminology

Use case #

We run a SaaS company. We offer a simple service to send emails:

curl -XPOST -H 'Authorization: ...' https://customerA.my-saas.io/emails -d '{
	"from": "alice@customerA.com",
	"to": "bob@example.com"
	"subject": "foo",
	"body": "bar"
}'

The service is easy to use and developer-friendly. It catches on.

Security comes in layers #

Cecilia from MegaCorp, Inc. wants to use my-saas.io in her latest project. This is a big contract for us! However, she adheres to defense in depth.

The base URL of her account https://megacorp.my-saas.io is easy to guess. Even if we switch the vanity domain megacorp.my-saas.io for something more random, like db1534b8-753f-4531-b7dc-fefb3d6f53f1.my-saas.io it can leak. The host is present in the DNS queries, SNI headers, request logs, etc.

Cecilia would like us to drop any connections to https://megacorp.my-saas.io unless they come from their VPC.

Solution sketch #

PROXY protocol v2 #

How can we attribute connections to specific customers? All the connections come from the same IP address (that one of the Private Link Service).

PROXY protocol v2 (or PP2 for short) to the rescue. Cloud providers allow us to enable PP2 on the service. Each connection will be annotated with extra attributes:

What’s a PP2? It’s a protocol popularized by the HAProxy. Quoting their docs:

The PROXY protocol provides a convenient way to safely transport connection information such as a client’s address across multiple layers of NAT or TCP proxies. It is designed to require little changes to existing components and to limit the performance impact caused by the processing of the transported information.

In practice, PP2 adds a binary (but unencrypted) blob at the start of the byte stream. Since it goes at the start of the connection, it will go even before the TLS handshake. You want to consume it, and then hand the rest of the byte stream over to the http library/application logic.

Note that PP2 is an option. By default it is disabled and you need to explicitly enable it.

Back to our solution #

  1. Customers fill in their link IDs.
  2. We add a proxy (can work at L7 or L4):
    1. It reads the PP2 header to understand the source of the connection.
      1. Public internet connections have no link IDs and can be processed the same way.
    2. It reads the SNI header to understand the target resource.
    3. If the target resource set up a link ID, then it compares if the source link ID matches:
      1. Match ==> Accept the connection and route it to the target.
      2. No match ==> Drop the connection.

All connections to customer resources are dropped, except if they originate from their configured VPC.

Private Link

Similarities with the classic IP filtering #

IP filtering

Conceptually, IP filtering is a similar solution. It is simpler, as source IPs are part of the TCP/UDP connections, no need to involve PP2. The connections are direct, with no extra Services or Endpoints.

In the cloud this is not feasible, there are no static IPs on either side:

Alternative solution: VPC peering #

VPC peering allows two VPCs to share network address space—similarly to how you may use a switch to join two network segments.

VPC peering

Diagram from https://docs.aws.amazon.com/vpc/latest/peering/what-is-vpc-peering.html

This sounds like a good idea, but has a couple of drawbacks for our SaaS use case:

  1. The private IPs between the VPCs can’t overlap, which means you need to coordinate the address space with all your customers.
  2. The access is bidirectional as if the resources were in the same network.
    • You can and should set up access-control rules, but an error in them may result in your customers being able to access your infrastructure and/or you being able to access your customers’ infrastructure.

Security considerations #

Can you trust the PP2 header? Can it be spoofed?

The scenario we need to watch out for is someone adding their own PP2 header, and therefore spoofing the connection source.

Differences between cloud providers #

AWSAzureGCP
NameAWS PrivateLinkAzure Private LinkGCP Private Service Connect
Allows cross-region connections?Yes, as of November 2024 available in major regionsYes, endpoint and service can be in different regionsYes, endpoint and service need to be in the same region, but you can connect to the endpoint from other regions with Global Access
PP2 header formatLinkLinkLink
Can share LB with public internet traffic?YesNo, see belowNo, a dedicated forwarding rule required
Learn morePrivateLink product pagePrivate Link overview and docsPrivate Service Connect guide

If you want to parse the PP2 headers in go—or compare notes on implementation—check tlvparse package in pires/go-proxyproto. A go library which supports all of them.

Quirks #

At least one overlapping AZs is required in AWS #

Both the source and the target VPCs need at least one overlapping Availability Zone to form the Private Link connection.

Check the limitations section of the docs, esp.

When the service provider and the consumer have different accounts and use multiple Availability Zones, and the consumer views the VPC endpoint service information, the response only includes the common Availability Zones. For example, when the service provider account uses us-east-1a and us-east-1c and the consumer uses us-east-1a and us-east-1b, the response includes the VPC endpoint services in the common Availability Zone, us-east-1a.

Note that the letters describing the AZs (e.g., a in us-east-1a) are randomized between the accounts. AWS internally uses numbers to represent the AZs and randomize the number => letter mapping per account4.

You can check your AZ mapping with the CLI:

$ aws ec2 describe-availability-zones --region us-east-1 | jq -c '.AvailabilityZones[] | { id: .ZoneId, name: .ZoneName } ' | sort
{"id":"use1-az1","name":"us-east-1c"}
{"id":"use1-az2","name":"us-east-1e"}
{"id":"use1-az3","name":"us-east-1d"}
{"id":"use1-az4","name":"us-east-1a"}
{"id":"use1-az5","name":"us-east-1f"}
{"id":"use1-az6","name":"us-east-1b"}

Azure requires different LBs #

You need a separate LB to handle Private Link traffic and public internet traffic. As soon as you attach Private Link Service to your LB it stops serving public internet traffic. At least this is what I’ve found in my tests, as far as I know, this limitation is not documented.

Create two different LBs and test the configuration in a pre-prod account before changing the production configuration.

DNS support #

How do customers access our service over Private Link? The endpoints get a static IP (from the source IP subnet) and a DNS alias.

curl -XPOST -H 'Authorization: ...' https://10.0.10.63/emails -d '{  # How do we pass the domain?
  …
}'

The problem is that 10.0.10.63 (or a random DNS alias) won’t match our TLS certs for *.my-saas.io.

This means that our Private Link customers need to manually create DNS records pointing from 10.0.10.63 => *.my-saas.io"

curl -XPOST -H 'Authorization: ...' https://megacorp.my-saas.io/emails -d '{
  …
}'

How do they know that everything is configured correctly? If they forgot the DNS record the megacorp.my-saas.io will connect to their resources over the public internet. This will likely result in a connection drop, which should hint at the problem.

Another suggestion would be to use a different domain—unresolvable from the public internet—for the Private Link traffic.

Private DNS integration #

The cloud providers offer automatic DNS integration. You can configure your Private Link Service, that any endpoint connecting to it will automatically add my-saas.io (or other domain) to the private DNS of its VPC5.

None of the providers support wildcard domains. In our example, we use the domain to differentiate between customers, but if we can use a different mechanism, e.g., http header, or part of the request, then we can simplify the Private Link setup for our customers using the Private DNS integration.

curl -XPOST -H 'Authorization: ...' https://my-saas.io/emails -d '{
	"account": "megacorp",
  …
}'

Summary #

We’ve learned about Private Link and how it can act as an IP filtering for the Cloud. We’ve presented a fictional SaaS company, and solved their use case with Private Link. We’ve discussed security considerations, differences in implementation, and quirks of different cloud providers.

Resources #

  1. See Q: How secure is an AWS PrivateLink connection? ↩︎

  2. See Proxy protocol. ↩︎

  3. I haven’t seen this documented anywhere, but as soon as we associate a private link service with an Azure Load Balancer, the public internet traffic stops flowing. This is worth a separate investigation and a blog post. ↩︎

  4. Learn more about the ID to name mapping in the docs. ↩︎

  5. Learn more VPC Endpoint Private DNS. ↩︎

>> Home