BlogIPv6 on EC2
(Updated: )

IPv6 masquerading for egress on microk8s on EC2

Benjamin Knofe

Staff Platform Engineer

devops

Share on social

a stock photo showing IPv6 on a mobile device screen

tl;dr

If your host has an IPv6 address (check with ip a ) and your microk8s runs in dual-stack mode, then you can do a

microk8s kubectl patch ippools/default-ipv6-ippool --type=merge -p '{"spec":{"natOutgoing":true}}'

and after your calico-node is restarted, it should work. For testing if it works, you can run:

microk8s kubectl run my-ping6-test --rm -ti --image docker.io/ubuntu:latest -- /bin/bash -c "apt update; apt install inetutils-ping -y; ping6 2606:4700:4700::1111"

which will ping Cloudflare’s 1.1.1.1 DNS service with its official IPv6 address.

The slightly longer post

The adoption of IPv6 throughout the internet goes slow, but it makes progress. One significant change was when AWS started charging for in-use IPv4 addresses this February.

For several years now the death knell has rung on IPv4 addresses and the predicted force move to IPv6’s larger addressable space. Market forces have provided a more gentle transition, with increased prices for IPv4 addresses and the implementation of dual stack solutions that allow support for both address types. The change of pricing at AWS was discussed at length on Hacker News last year.

Advantages of IPv6

Moving to IPv6 has other developer experience advantages, especially when running on AWS and using their VPC offering: Every time you create a new VPC, you will just get your own block of IPv6 addresses and that’s it, you can use it as you want from now on, no EIP allocation, NAT’ing or anything complicated, just assign IPs from that space and get going. It’s awesome!

More generally, the vast address space of IPv6 eliminates the need for network address translation (NAT), allowing every device to have a unique global address, which simplifies network design and improves end-to-end connectivity. Auto-configuration capabilities streamline network setup, reducing the reliance on DHCP for address assignment. Without private address collisions, networks can interoperate without the complex workarounds required in IPv4. IPv6 enhances multicast routing efficiency, supporting more robust and scalable distribution of data. Its simplified header format and improved routing mechanisms enable more efficient packet processing. IPv6 introduces true QoS through flow labeling, allowing for better traffic prioritization. Security is fundamentally integrated, with support for authentication and privacy directly in the protocol. The architecture of IPv6 is designed to be extensible, accommodating future needs with flexible options and extensions.

No operations discussion would be complete without considering how end-to-end testing and synthetic monitoring will be affected. Of course, Checkly also supports IPv6!

Getting started — Prerequisites

  • an IPv6 enabled host
  • a dual-stack (or IPv6 only) microk8s cluster

IPv6 enabled host

When on AWS, to assign an EC2 instance automatically an IPv6 address, make sure to create a subnet with a IPv6 CIDR range. Terraform example would be something like this:

resource "aws_vpc" "main" {
  cidr_block                       = "10.0.0.0/16"
  assign_generated_ipv6_cidr_block = true
}

data "aws_availability_zones" "available" {
  state = "available"
}

resource "aws_subnet" "main" {
  count             = length(data.aws_availability_zones.available.names)
  vpc_id            = aws_vpc.main.id
  cidr_block        = cidrsubnet("10.0.0.0/16", 4, count.index + 1)
  ipv6_cidr_block   = cidrsubnet(aws_vpc.main.ipv6_cidr_block, 8, count.index + 2)
  availability_zone = data.aws_availability_zones.available.names[count.index]
}

Then spin up an instance in that subnet, connect to it and run ip a. It should look something like this:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
    link/ether 02:cc:f3:a3:2f:09 brd ff:ff:ff:ff:ff:ff
    altname enp0s5
    inet 10.0.18.64/20 metric 100 brd 10.0.31.255 scope global dynamic ens5
       valid_lft 3087sec preferred_lft 3087sec
    inet6 2001:0db8:85a3:0000:0000:8a2e:0370:7334/128 scope global dynamic noprefixroute 
       valid_lft 382sec preferred_lft 72sec
    inet6 fe80::cc:f3ff:fea3:2f09/64 scope link 
       valid_lft forever preferred_lft forever

Dual-stack microk8s cluster

To set up a dual-stack microk8s cluster, these exact steps will get us working for our demonstration.

Before installing microk8s, create a file at /var/snap/microk8s/common/.microk8s.yaml with the following contents:

/var/snap/microk8s/common/.microk8s.yaml
---
version: 0.1.0
extraCNIEnv:
  IPv4_SUPPORT: true
  IPv4_CLUSTER_CIDR: 10.3.0.0/16
  IPv4_SERVICE_CIDR: 10.152.183.0/24
  IPv6_SUPPORT: true
  IPv6_CLUSTER_CIDR: fd02::/64
  IPv6_SERVICE_CIDR: fd99::/108
extraSANs:
  - 10.152.183.1
extraKubeletArgs:
  --cluster-domain: cluster.local
  --cluster-dns: 10.152.183.10
addons:
  - name: dns

This is called the microk8s launch configuration[4]. It includes all the default settings plus everything necessary for IPv6.

Then install microk8s with snap install microk8s --classic

Test if everything is working so far

First check if pods get an IPv4 and an IPv6 address assigned, for example by running

microk8s kubectl -n kube-system describe pod

It should show something like this for each pod there:

IPs:
  IP:           10.42.0.2
  IP:           2001:cafe:42::2

Enable IPv6 masquerading in microk8s

I am not sure why but microk8s’ calico does not automatically allow NAT’ing outside of the cluster, to enable it, just run a

microk8s kubectl run my-ping6-test --rm -ti --image docker.io/ubuntu:latest -- /bin/bash -c "apt update; apt install inetutils-ping -y; ping6 2606:4700:4700::1111"

And should be able to see pinging work with something like that (after some installation output from apt):

PING 2606:4700:4700::1111 (2606:4700:4700::1111): 56 data bytes
64 bytes from one.one.one.one: icmp_seq=0 ttl=56 time=1.184 ms
64 bytes from one.one.one.one: icmp_seq=1 ttl=56 time=2.961 ms
64 bytes from one.one.one.one: icmp_seq=2 ttl=56 time=1.254 ms
64 bytes from one.one.one.one: icmp_seq=3 ttl=56 time=1.163 ms

Success! Look at that gorgeous 128-bit IPv6 address.


If you'd like to share your own experience with IPv6 or the other trials and tribulations of modern operations, join the Checkly Slack to meet the team!

Share on social