Sydney to Auckland latency tests.

As some of you may know I am an AWS cloud architect that lives in New Zealand. So most of the work I do is in the AP-SouthEast-2 region, or Sydney if you’re old fashioned. In my previous post I talked about the new Auckland local zone that went live in 2023. This is the harbinger of a full region, which my spies tell me will be launched sometime this year.

With that on the horizon I am predicting a lot of my future work will involve moving AWS resources between Sydney and Auckland. Suddenly the speed between these two regions has become an area of great interest to me.

What is a man to do? The answer is obvious: Poke It!

My setup:

• I am located in Wellington, New Zealand.
• I am using a MacBook pro connected to my network by ethernet.
• My internet connection is fibre with 2degrees.

Here is a speedtest run on my mac:

   Speedtest by Ookla

      Server: 2Degrees - Auckland (id: 5749)
         ISP: 2degrees
Idle Latency:    10.26 ms   (jitter: 0.76ms, low: 9.58ms, high: 10.65ms)
    Download:   530.19 Mbps (data used: 482.2 MB)
                 24.63 ms   (jitter: 1.86ms, low: 10.54ms, high: 31.70ms)
      Upload:   518.11 Mbps (data used: 502.2 MB)
                 20.52 ms   (jitter: 0.51ms, low: 9.50ms, high: 28.25ms)
 Packet Loss:     0.0%

Testing Parameters

• All instances will be T3.Medium
• All instances will run Amazon Linux 2023
• Each ping result is the average of 100 attempts

I will attempt to test the average speed between Sydney and Auckland using a variety of different network configurations.

AWS Setup

I have created a basic VPC in the AWS Sydney “ap-southest-2” region. From here I wil build out increasingly more complex scenario’s.

Test Scenario 1 - Home to Sydney

I deploy an EC2 instance in Public Subnet 1 with a public IP address, I then ping it from my home network as shown in the diagram.

Result

37.09 ms

Test Scenario 2 - Sydney, Loopback

This time I ping localhost on the EC2 instance. Why? to establish a baseline, this is as fast as it will go.

Result

0.045 ms

That’s not even one millisecond, we are in the submillisecond realm here. I could have written 45 μs “microseconds” but those of us that deal with latency are used to milliseconds, so that’s what I’ll use.

Test Scenario 3 - Sydney, Same Subnet, Private

Another EC2 instance gets created in the same Subnet as the first. Security groups are in place to ensure they can communicate, and I initiate a ping between instance 1 to instance 2 using their “Private” IP addresses.

Result

0.207 ms

still in the submilliseconds, you might think that that is a lot slower than the previous test, and you would be right, but its important to remember that we are still not reaching even a single millisecond. These two instances are in the same subnet and the same AZ, so theoretically they could be on the same hypervisor host.

Test Scenario 4 - Sydney, Same Subnet, Public

Here we do the same test but using the Public IP address of the second instance. This will cause the first instance to use its public interface, the connection will be routed through the Internet Gateway, but then immediately go back. This could be considered a “hairpin” connection.

Result

0.289 ms

from 0.270 to 0.289, an increase of only 0.019 milliseconds.

Test Scenario 5 - Sydney, Different AZ, Private

Now we remove the second instance from Subnet 1 and recreate it in Subnet 2. Its important to remember that Availability Zones or AZ’s are different physical locations within the same region. To quote AWS themselves:

An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region.

Source

Result

0.885 ms

This genuinely impressed me, I knew the AWS “backbone” is fast, but to get a latency under one millisecond for distinct datacenters within the same region is amazing.

Test Scenario 6 - Sydney, Different AZ, Public

Same as before, but this time, again we will be using the public IP of the second instance.

Result

0.915 ms

This path is partially outside of AWS infrastructure, and yet we are only seeing an increase of 0.03 milliseconds. Sydney must have very good network infrastructure.

Test Scenario 7 - Auckland Local Zone

I have added a Subnet to the VPC, but this one is special, is located in the ap-southeast-2-akl-1a Availability Zone, or the Auckland AZ.

I create another instance in this subnet. Then I ping this instance from my home network using its Public IP.

Result

10.79 ms

Thats really good, about the same as my initial ookla speedtest.

Test Scenario 8 - Sydney to Auckland, Private

Now we will be “jumping the pond” using AWS network infrastructure. Im pinging from instance 1 to instance 2 using instance_2’s private IP address.

Result

35.7 ms

This is not too bad considering it needs to go there and back again. That’s approximately 18 ms per trip.

Test Scenario 9 - Sydney to Auckland, Public

Doing the same, but this time using the public IP addresses, the pattern im using is becoming predictable now.

Result

27.8 ms

This is surprising, I’ve tested multiple times, each time doing 100 passes, the results are consistent, the public internet seems to be faster than AWS dedicated line. ¯\(ツ)/¯

Test Scenario 10 - Sydney to Sydney, VPC Peering

I’m increasing complexity by adding an account with a VPC. I’ve connected the vpc’s using VPC peering. the EC2 instances are deployed in the same availability zone.

Result

0.3 ms

Almost as fast as being in the same VPC

Warning:
Availability Zones across accounts might not have the same name, what is called 2A in one account might be a different AZ in another. to ensure you are using the same physical Availability zone, go into “AWS Resource Access Manager” and look for the following table:

Test Scenario 11 - Sydney to Sydney, VPC Peering Different AZ

Same as before but this time the Availability Zones are different.

Result

0.8 ms

Almost exactly as fast as when they were in the same VPC, VPC Peering does not seem to introduce any noticable latency.

Test Scenario 12 - Sydney to Auckland, VPC Peering

Now we ping from a Sydney AZ in one VPC to the local zone AZ in the other VPC

Result

35.7 ms

That makes sense, it needs to travel the Tasman again.

Test Scenario 13 - Auckland to Auckland, VPC Peering

Local zone to local zone, different VPC though,still using VPC peering, anyone want to take some bets? will this be fast or slow?

Result

0.17 ms

I would call that blazingly fast,. The same as being in the same VPC and the same subnet

Test Scenario 14 - Sydney to Sydney, Transit Gateway

I repeat the experiment with the two accounts and VPC’s, but this time we use transit gateway instead of VPC Peering. As before we are using the same Availability Zone for each account, making sure we checked the AZ ID’s in the “AWS Resource Access Manager” AZ ID table.

Result

0.40 ms

Just shy of half a millisecond, that’s pretty fast in my opinion.

Test Scenario 15 - Sydney to Sydney, Transit Gateway Different AZ

Transit gateway again, different accounts and VPC’s different Availability Zones, still all in Sydney.

Result

0.998 ms

Just 2 microseconds shy of a millisecond, slower than the previous one, but still lightning fast.

Test Scenario 16 - Sydney to Auckland, Transit Gateway

This is where it gets interesting, one EC2 instance is located in Sydney in the first account and VPC, the other EC2 instance is located in the Auckland local zone which sits in the second account and VPC.

Result

36.39 ms

Jumping the Tasman comes with a cost, no matter how you do it.

Test Scenario 17 - Auckland to Auckland, Transit Gateway

This is it, the one we’ve all been pondering, what happens when we use Transit Gateway as a means to communicate between the same Local Zone? Will it be fast like VPC Peering?

Result

63.077 ms

The slowest of them all, this is about as slow as you can make it.

For those of you who don’t know, Transit Gateway works through Transit Gateway Attachments, These attachments are deployed in either dedicated subnets or existing subnets in your Availability Zones.

It is not possible to deploy a Transit Gateway Attachment in a Local Zone.

Therefor a Local Zone must use a Transit Gateway Attachment in one of the home region Availability Zones.

The path the ping test takes is as follows:

• Server1 pings Server2 using its private IP.
• The ICMP packet gets routed to the Transit Gateway Attachment in Sydney (≈ 15 ms)
• The ICMP packet enters the Transit Gateway Network in Sydney in account1 and exits in Sydney in account2
• The ICMP packet gets routed to Server2 in Auckland from Sydney (≈ 15 ms)
• Server2 receives the ICMP packet and returns the ping to Server1
• The ICMP packet gets routed to the Transit Gateway Attachment in Sydney (≈ 15 ms)
• The ICMP packet enters the Transit Gateway Network in Sydney in account2 and exits in Sydney in account1
• The ICMP packet gets routed to Server1 in Auckland from Sydney (≈ 15 ms)
• Server1 receives the ping and returns the result.

In short, the ping travels from Sydney to Auckland to Sydney back to Auckland, jumping the Tasman Sea for a total of four times.

https://docs.aws.amazon.com/vpc/latest/tgw/what-is-transit-gateway.html

Overview

Here are all the test in a neat little overview. As you can see loopback is fastest, no surprises there. Local Zone through Local Zone using Transit Gateway is slowest by far. Avoid that configuration if you can.

Conclusion

• Local zone to local zone via transit gateway is a terrible idea.
• You should use AWS Local Zones to deploy workloads closer to your end-users for low-latency requirements. AWS Local Zones have their own connection to the internet and support AWS Direct Connect, so resources created in the Local Zone can serve local end-users with very low-latency communications.
• Local zones leverage the transit gateway attachment in the main region. (Sydney in our case)

Gotcha’s

Availability Zone Names

ap-southeast-2a for me might not be ap-southeast-2a for you, or even in another account that you own. Check the actual AZ ID’s to be sure.

Local Zone Transit Gateway Attachments

You can not create a transit gateway attachment in the local zone Subnets. I’ve done some testing and it looks like the Auckland local zone uses the transit gateway attachment in the APSE2-AZ2 availability zone, which for me is ap-southeast-2c If you have all three Availability Zone’s in Sydney and attach your transit gateway attachment to all three, this should not be a problem, but if you only have 1 or 2 AZ’s , be careful, your local zone might not be able to connect to transit gateway. If you have trouble connecting your Local Zone to Transit Gateway, try adding a Transit Gateway Attachment to the APSE2-AZ2 Availability Zone.

References

Local Zones Documentation

Transit Gateway Documentation

Vpc Peering Documentation