Can't get NAT64 to work. Do I need to masquerade traffic?

[dekiesel]

Hi,

I am running openthread border router (using this docker container).
I am able to control my thread devices from Home Assistant, but I am unable to let thread devices connect directly to ipv4 services (mqtt, in my case).

I can see that NAT64 is translating the requests

17:05:34.905155 IP 192.168.255.1.49370 > 10.1.101.5.1883: Flags [SEW], seq 3290775174, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70234635 ecr 0], length 0
17:05:37.980221 IP 192.168.255.1.49370 > 10.1.101.5.1883: Flags [SEW], seq 3290775174, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70237836 ecr 0], length 0
17:05:41.181269 IP 192.168.255.1.49370 > 10.1.101.5.1883: Flags [S], seq 3290775174, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70241036 ecr 0], length 0
17:05:44.411568 IP 192.168.255.1.49370 > 10.1.101.5.1883: Flags [S], seq 3290775174, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70244237 ecr 0], length 0
17:07:13.578977 IP 192.168.255.1.49371 > 10.1.101.5.1883: Flags [SEW], seq 436685625, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70333437 ecr 0], length 0
17:07:16.580411 IP 192.168.255.1.49371 > 10.1.101.5.1883: Flags [SEW], seq 436685625, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70336438 ecr 0], length 0
17:07:19.780928 IP 192.168.255.1.49371 > 10.1.101.5.1883: Flags [SEW], seq 436685625, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70339639 ecr 0], length 0
17:07:22.981550 IP 192.168.255.1.49371 > 10.1.101.5.1883: Flags [SEW], seq 436685625, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70342840 ecr 0], length 0
17:07:26.181465 IP 192.168.255.1.49371 > 10.1.101.5.1883: Flags [S], seq 436685625, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 70346040 ecr 0], length 0

10.1.101.5 being the host the mqtt server is running on.
My network is divided into multiple VLANs, all in the 10.1. range.

My thread device can't reach my mqtt server and I'm thinking it's because NAT64 is translating the IPs intro a range that doesn't exist in my network? 192.168.255.x isn't available in my network.

Can I masquerade that traffic? Can I somehow use a different ip address space?

Here is my docker-compose:

otbr:~/app# cat docker-compose.yaml 
services:
  otbr:
    image: bnutzer/otbr-tcp
    network_mode: host
    restart: unless-stopped
    privileged: true
    cap_drop:
      - NET_ADMIN   # Should prevent iptables/ipset updates
      - NET_RAW     # No raw network access
    devices:
      - /dev/net/tun
    environment:
      - RCP_HOST=wohnzimmer-SLZB-MR1.lan
      - RCP_PORT=7638
      - OTBR_WEB_ENABLE=1
        #- OTBR_WEB_LISTEN_ADDRESS=10.1.100.0/24
    volumes:
      - ${OTBR}:/var/lib/thread
    logging:
      driver: "json-file"
      options:
        max-size: "208m"

And here is the output of ip -a

otbr:~/app# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0@if88: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP qlen 1000
    link/ether 6e:c4:f7:78:96:74 brd ff:ff:ff:ff:ff:ff
    inet 10.1.101.247/24 brd 10.1.101.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fde0:d605:9475:0:6cc4:f7ff:fe78:9674/64 scope global dynamic flags 100 
       valid_lft 5060sec preferred_lft 2360sec
    inet6 2001:16b8:c122:700:6cc4:f7ff:fe78:9674/64 scope global dynamic flags 100 
       valid_lft 5060sec preferred_lft 2360sec
    inet6 fd54:bac3:aabe:339d:6cc4:f7ff:fe78:9674/64 scope global dynamic flags 100 
       valid_lft 1759sec preferred_lft 1759sec
    inet6 fe80::6cc4:f7ff:fe78:9674/64 scope link 
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN 
    link/ether 02:42:86:e2:19:f2 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
5: wpan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq_codel state UNKNOWN qlen 500
    link/[65534] 
    inet6 fdce:bcd:a00d:ec9:0:ff:fe00:fc11/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fd13:bdbc:61a:1:45e8:43c8:a5b4:4d3c/64 scope global flags 02 
       valid_lft forever preferred_lft forever
    inet6 fdce:bcd:a00d:ec9:0:ff:fe00:fc38/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fdce:bcd:a00d:ec9:0:ff:fe00:fc10/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fdce:bcd:a00d:ec9:0:ff:fe00:fc00/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fdce:bcd:a00d:ec9:0:ff:fe00:2c00/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fdce:bcd:a00d:ec9:76b5:e71b:826e:fc6d/64 scope global deprecated flags 02 
       valid_lft forever preferred_lft 0sec
    inet6 fe80::e4ea:8a12:71ef:640a/64 scope link deprecated flags 02 
       valid_lft forever preferred_lft 0sec

I'm thankful for any help!! :)

[jwhui]

It looks like your NAT64 translation is working correctly inside the OTBR, but the resulting IPv4 traffic isn't routable on your LAN because it isn't being masqueraded.

The Issue:
In your tcpdump output, we can see packets leaving with a source IP of 192.168.255.1. This is an internal address used by the NAT64 translator. When your MQTT server (10.1.101.5) receives this packet, it has no route back to 192.168.255.1, so it cannot reply.

The Fix:
The OTBR needs to perform "masquerading" to replace that 192.168.255.1 source address with the OTBR's own LAN IP (10.1.101.247). The OTBR-agent normally sets up these iptables rules automatically, but it requires the NET_ADMIN capability to do so.

In your docker-compose.yaml, you have explicitly dropped NET_ADMIN:

    cap_drop:
      - NET_ADMIN   # Should prevent iptables/ipset updates

To fix this, you should remove NET_ADMIN (and likely NET_RAW) from the cap_drop section. For an OTBR to manage its networking and firewall rules correctly, it typically needs these capabilities.

Updated docker-compose.yaml snippet:

    # ...
    privileged: true
    # Remove or comment out the cap_drop section
    # cap_drop:
    #   - NET_ADMIN
    #   - NET_RAW
    # ...

Once NET_ADMIN is restored, the OTBR should be able to configure the necessary iptables rules, and your Thread devices should be able to reach your MQTT server.

[dekiesel]

Hi and thanks for your help!

I removed both capabilities, restarted the container and tried again, but I am unfortunately seeing the same issue:

otbr:~# tcpdump -ni any host 10.1.101.5
tcpdump: WARNING: any: That device doesn't support promiscuous mode
(Promiscuous mode not supported on the "any" device)
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
08:56:39.154791 wpan0 In  IP 192.168.255.1.49275 > 10.1.101.5.1883: Flags [SEW], seq 751140670, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 158826428 ecr 0], length 0
08:56:42.138350 wpan0 In  IP 192.168.255.1.49275 > 10.1.101.5.1883: Flags [SEW], seq 751140670, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 158829428 ecr 0], length 0
08:56:45.341638 wpan0 In  IP 192.168.255.1.49275 > 10.1.101.5.1883: Flags [SEW], seq 751140670, win 1333, options [mss 474,nop,wscale 0,sackOK,TS val 158832630 ecr 0], length 0

Should I try masquerading the traffic myself? Or would that interfere with OTBR?

[jwhui]

Have you tried explicitly adding NET_ADMIN with cap_add?

[dekiesel]

Weird, I thought I had added a comment already. Apologies!

I noticed, that the container didn't set the iptables-rules described in _nat64.

I manually added these rules to my /etc/profile and everything works as expected now.

iptables -t mangle -A PREROUTING -i wpan0 -j MARK --set-mark 0x1001
iptables -t nat -A POSTROUTING -m mark --mark 0x1001 -j MASQUERADE
iptables -t filter -A FORWARD -o eth0 -j ACCEPT
iptables -t filter -A FORWARD -i eth0 -j ACCEPT

I haven't had time to explore why the container wasn't able to set the rules.

[jwhui]

Glad to hear that you have it working. If you ever figure out why the Docker image is not setting the rules automatically, please let us know. Thanks!