Skip to content

Calico verbose panic log: ip6tables-legacy-save failed

Problem

When you start the new version of Calico (e.g. 3.22) with docker, you may see the following panic logs repeatly from Calico log output when ipv6 related kernel parameter (/proc/sys/net/ipv6/conf/all/disable_ipv6) is disabled.

2022-05-31 02:35:15.284 [PANIC][14115] felix/table.go 769: ip6tables-legacy-save command failed after retries ipVersion=0x6 table="raw"
panic: (*logrus.Entry) 0xc00143d4f0

goroutine 498 [running]:
github.com/sirupsen/logrus.Entry.log({0xc0001321e0, 0xc0011704b0, {0x0, 0x0, 0x0}, 0x0, {0x0, 0x0}, 0x0}, 0x0, ...)
    /go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:128 +0x56c
github.com/sirupsen/logrus.(*Entry).Panic(0xc000a18be0, {0xc0018c2b58, 0x1, 0x1})
    /go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:173 +0xfb
github.com/sirupsen/logrus.(*Entry).Panicf(0xc000a18be0, {0x2d636b2, 0x2d21148}, {0xc0018c2c10, 0x70222c5d2232332f, 0x2273656c69666f72})
    /go/pkg/mod/github.com/projectcalico/logrus@v1.0.4-calico/entry.go:221 +0x70
github.com/projectcalico/calico/felix/iptables.(*Table).getHashesAndRulesFromDataplane(0xc00005f600)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:769 +0x3cc
github.com/projectcalico/calico/felix/iptables.(*Table).loadDataplaneState(0xc00005f600)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:606 +0x196
github.com/projectcalico/calico/felix/iptables.(*Table).Apply(0xc00005f600)
    /go/src/github.com/projectcalico/calico/felix/iptables/table.go:990 +0x373
github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply.func3(0x0)
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1858 +0x52
created by github.com/projectcalico/calico/felix/dataplane/linux.(*InternalDataplane).apply
    /go/src/github.com/projectcalico/calico/felix/dataplane/linux/int_dataplane.go:1857 +0x671
2022-05-31 02:35:16.721 [WARNING][14248] felix/table.go 814: iptables save failed error=exit status 1

Reason and explanation

New version of Calico supports ipv6 by relying on the following Linux kernel modules, and load them by running command ip6tables-legacy-save. However, these modules cannot be loaded properly if this command is triggered inside docker with isolated namespace.

ip6t_rpfilter
ip6table_nat
ip6table_raw
ip6table_mangle
ip6table_filter
vip6_tables

Solution

Manually run one of the ip6tables-* command (e.g. ip6tables-legacy-save) as root user, or use modprobe to load the corresponding modules.

sudo modprobe ip6t_rpfilter
sudo modprobe ip6table_nat
sudo modprobe ip6table_raw
sudo modprobe ip6table_mangle
sudo modprobe ip6table_filter
sudo modprobe ip6_tables

net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.default.disable_ipv6
net.ipv6.conf.lo.disable_ipv6

Disclaimer

License under CC BY-NC 4.0
Copyright issue feedback me#imzye.me, replace # with @
Not all the commands and scripts are tested in production environment, use at your own risk
No privacy information is collected here

Try iOS App

Yi's Blog