0

I'm looking for is a good explanation of how to interpret the output of iptables -L and ip rule show in conjunction with the ip route show table <table_name> commands.

I am struggling to understand the overly complicated policy-based routing of my OnePlus 8 Pro while the WiFi hotspot service is turned on. I have CCNA and Cisco Certified Specialist - Enterprise Core certifications, so I'm not clueless when it comes to routing and policy-based routing, but I still need help.

I don't understand much about Linux's iptables. I understand some of the output of ip rule show (like the priority and which routing table is checked if the rule is matched). I have a firm grasp on ip route when using a single table, but less so when combined with policy-based routing.

Unfortunately, my OnePlus 8 Pro seems to be combining iptables with policy-based routing and multiple routing tables for really no good reason at all. My system is rooted and I have busybox installed to give me most Linux standard command-line tools.

So here is the mess that I'm trying to understand:

First, here are the interfaces listed for the device:

OnePlus8Pro# ifconfig dummy0 Link encap:Ethernet HWaddr ae:22:9c:ac:69:85 inet6 addr: fe80::ac22:9cff:feac:6985/64 Scope: Link UP BROADCAST RUNNING NOARP MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:113 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 TX bytes:15971 wlan0 Link encap:Ethernet HWaddr 4c:4f:ee:96:b3:ed Driver cnss_pci UP BROADCAST MULTICAST MTU:1500 Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:3000 RX bytes:0 TX bytes:0 rmnet_data3 Link encap:UNSPEC inet6 addr: fe80::a00e:387b:f22e:c336/64 Scope: Link inet6 addr: 2607:fc20:1bc3:a194:a00e:387b:f22e:c336/64 Scope: Global UP RUNNING MTU:1500 Metric:1 RX packets:10 errors:0 dropped:0 overruns:0 frame:0 TX packets:18 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:804 TX bytes:1284 rmnet_data0 Link encap:UNSPEC inet6 addr: fe80::11c2:944d:89dd:e766/64 Scope: Link UP RUNNING MTU:1500 Metric:1 RX packets:51 errors:0 dropped:0 overruns:0 frame:0 TX packets:36 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:7720 TX bytes:2487 rmnet_mhi0 Link encap:UNSPEC Driver mhi_netdev UP RUNNING MTU:65535 Metric:1 RX packets:48471 errors:0 dropped:0 overruns:0 frame:0 TX packets:42815 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:23427476 TX bytes:11997668 lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 inet6 addr: ::1/128 Scope: Host UP LOOPBACK RUNNING MTU:65536 Metric:1 RX packets:59 errors:0 dropped:0 overruns:0 frame:0 TX packets:59 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:5850 TX bytes:5850 wlan1 Link encap:Ethernet HWaddr 62:59:38:18:bc:69 Driver cnss_pci inet addr:192.168.42.149 Bcast:192.168.42.255 Mask:255.255.255.0 inet6 addr: fe80::6059:38ff:fe18:bc69/64 Scope: Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:4599 errors:0 dropped:0 overruns:0 frame:0 TX packets:1756 errors:0 dropped:44 overruns:0 carrier:0 collisions:0 txqueuelen:3000 RX bytes:859009 TX bytes:143474 rmnet_data2 Link encap:UNSPEC inet addr:33.215.218.82 Mask:255.255.255.252 UP RUNNING MTU:1440 Metric:1 RX packets:30742 errors:0 dropped:0 overruns:0 frame:0 TX packets:35534 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:21959066 TX bytes:11141381

Next, here is the routing:

OnePlus8Pro# ip route 33.215.218.80/30 dev rmnet_data2 proto kernel scope link src 33.215.218.82 192.168.42.0/24 dev wlan1 proto kernel scope link src 192.168.42.149

And now, the ip rules (policy-based routing):

OnePlus8Pro# ip rule show 0: from all lookup local 10000: from all fwmark 0xc0000/0xd0000 lookup legacy_system 10500: from all iif lo oif dummy0 uidrange 0-0 lookup dummy0 10500: from all iif lo oif rmnet_data0 uidrange 0-0 lookup rmnet_data0 10500: from all iif lo oif rmnet_data2 uidrange 0-0 lookup rmnet_data2 10500: from all iif lo oif wlan1 uidrange 0-0 lookup local_network 13000: from all fwmark 0x10063/0x1ffff iif lo lookup local_network 13000: from all fwmark 0xd0001/0xdffff iif lo lookup rmnet_data0 13000: from all fwmark 0x10064/0x1ffff iif lo lookup rmnet_data2 14000: from all iif lo oif dummy0 lookup dummy0 14000: from all fwmark 0xc0000/0xc0000 iif lo oif rmnet_data0 lookup rmnet_data0 14000: from all iif lo oif rmnet_data2 lookup rmnet_data2 14000: from all iif lo oif wlan1 lookup local_network 15000: from all fwmark 0x0/0x10000 lookup legacy_system 16000: from all fwmark 0x0/0x10000 lookup legacy_network 17000: from all fwmark 0x0/0x10000 lookup local_network 18000: from all iif wlan1 lookup rmnet_data2 19000: from all fwmark 0x64/0x1ffff iif lo lookup rmnet_data2 22000: from all fwmark 0x0/0xffff iif lo lookup rmnet_data2 32000: from all unreachable

A list of all the named routing tables:

OnePlus8Pro# show table all | grep "table" | sed 's/.*\(table.*\)/\1/g' | awk '{print $2}' | sort | uniq 1014 1025 dummy0 local local_network rmnet_data0 rmnet_data2

Output of ip route show table <table_name> for each named routing table:

OnePlus8Pro# ip route show table 1014 OnePlus8Pro# ip route show table 1025 OnePlus8Pro# ip route show table dummy0 default dev dummy0 proto static scope link OnePlus8Pro# ip route show table local broadcast 33.215.218.80 dev rmnet_data2 proto kernel scope link src 33.215.218.82 local 33.215.218.82 dev rmnet_data2 proto kernel scope host src 33.215.218.82 broadcast 33.215.218.83 dev rmnet_data2 proto kernel scope link src 33.215.218.82 broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1 local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1 local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1 broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1 broadcast 192.168.42.0 dev wlan1 proto kernel scope link src 192.168.42.149 local 192.168.42.149 dev wlan1 proto kernel scope host src 192.168.42.149 broadcast 192.168.42.255 dev wlan1 proto kernel scope link src 192.168.42.149 OnePlus8Pro# ip route show table local_network 192.168.42.0/24 dev wlan1 proto static scope link OnePlus8Pro# ip route show table rmnet_data0 OnePlus8Pro# ip route show table rmnet_data2 default via 33.215.218.81 dev rmnet_data2 proto static mtu 1500 33.215.218.80/30 dev rmnet_data2 proto static scope link

And last but not least, the output of iptables -L:

OnePlus8Pro# iptables -L Chain INPUT (policy ACCEPT) target prot opt source destination bw_INPUT all -- anywhere anywhere fw_INPUT all -- anywhere anywhere Chain FORWARD (policy ACCEPT) target prot opt source destination oem_fwd all -- anywhere anywhere fw_FORWARD all -- anywhere anywhere bw_FORWARD all -- anywhere anywhere tetherctrl_FORWARD all -- anywhere anywhere Chain OUTPUT (policy ACCEPT) target prot opt source destination nm_qti_filter_ssdp_dropper all -- anywhere anywhere oem_out all -- anywhere anywhere fw_OUTPUT all -- anywhere anywhere st_OUTPUT all -- anywhere anywhere bw_OUTPUT all -- anywhere anywhere Chain bw_FORWARD (1 references) target prot opt source destination bw_costly_rmnet_data2 all -- anywhere anywhere bw_costly_rmnet_data2 all -- anywhere anywhere Chain bw_INPUT (1 references) target prot opt source destination bw_global_alert all -- anywhere anywhere bw_costly_rmnet_data2 all -- anywhere anywhere RETURN esp -- anywhere anywhere RETURN all -- anywhere anywhere mark match 0x100000/0x100000 MARK all -- anywhere anywhere MARK or 0x100000 Chain bw_OUTPUT (1 references) target prot opt source destination bw_global_alert all -- anywhere anywhere bw_costly_rmnet_data2 all -- anywhere anywhere Chain bw_costly_rmnet_data2 (4 references) target prot opt source destination bw_penalty_box all -- anywhere anywhere REJECT all -- anywhere anywhere ! quota rmnet_data2: 9223372036854775807 bytes reject-with icmp-port-unreachable Chain bw_costly_shared (0 references) target prot opt source destination bw_penalty_box all -- anywhere anywhere Chain bw_data_saver (1 references) target prot opt source destination RETURN all -- anywhere anywhere Chain bw_global_alert (3 references) target prot opt source destination all -- anywhere anywhere ! quota globalAlert: 2097152 bytes Chain bw_happy_box (1 references) target prot opt source destination RETURN all -- anywhere anywhere match bpf pinned /sys/fs/bpf/prog_netd_skfilter_whitelist_xtbpf bw_data_saver all -- anywhere anywhere Chain bw_penalty_box (2 references) target prot opt source destination REJECT all -- anywhere anywhere match bpf pinned /sys/fs/bpf/prog_netd_skfilter_blacklist_xtbpf reject-with icmp-port-unreachable bw_happy_box all -- anywhere anywhere Chain fw_FORWARD (1 references) target prot opt source destination Chain fw_INPUT (1 references) target prot opt source destination Chain fw_OUTPUT (1 references) target prot opt source destination Chain nm_mdmprxy_doze_mode_skip (0 references) target prot opt source destination Chain nm_mdmprxy_iface_pkt_fwder (0 references) target prot opt source destination Chain nm_qti_filter_ssdp_dropper (1 references) target prot opt source destination DROP udp -- anywhere anywhere udp dpt:1900 DROP udp -- anywhere anywhere udp dpt:1900 Chain oem_fwd (1 references) target prot opt source destination Chain oem_out (1 references) target prot opt source destination Chain st_OUTPUT (1 references) target prot opt source destination Chain st_clear_caught (2 references) target prot opt source destination Chain st_clear_detect (0 references) target prot opt source destination REJECT all -- anywhere anywhere connmark match 0x2000000/0x2000000 reject-with icmp-port-unreachable RETURN all -- anywhere anywhere connmark match 0x1000000/0x1000000 CONNMARK tcp -- anywhere anywhere u32 "0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x0&0xffff0000=0x16030000&&0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x4&0xff0000=0x10000" CONNMARK or 0x1000000 CONNMARK udp -- anywhere anywhere u32 "0x0>>0x16&0x3c@0x8&0xffff0000=0x16fe0000&&0x0>>0x16&0x3c@0x14&0xff0000=0x10000" CONNMARK or 0x1000000 RETURN all -- anywhere anywhere connmark match 0x1000000/0x1000000 st_clear_caught tcp -- anywhere anywhere state ESTABLISHED u32 "0x0>>0x16&0x3c@0xc>>0x1a&0x3c@0x0&0x0=0x0" st_clear_caught udp -- anywhere anywhere Chain st_penalty_log (0 references) target prot opt source destination CONNMARK all -- anywhere anywhere CONNMARK or 0x1000000 NFLOG all -- anywhere anywhere Chain st_penalty_reject (0 references) target prot opt source destination CONNMARK all -- anywhere anywhere CONNMARK or 0x2000000 NFLOG all -- anywhere anywhere REJECT all -- anywhere anywhere reject-with icmp-port-unreachable Chain tetherctrl_FORWARD (1 references) target prot opt source destination bw_global_alert all -- anywhere anywhere tetherctrl_counters all -- anywhere anywhere [goto] state RELATED,ESTABLISHED DROP all -- anywhere anywhere state INVALID tetherctrl_counters all -- anywhere anywhere [goto] DROP all -- anywhere anywhere Chain tetherctrl_counters (2 references) target prot opt source destination RETURN all -- anywhere anywhere RETURN all -- anywhere anywhere

I'm trying to process/understand all these rules the same way the phone does, and I'm lost. I can't find a good explanation of lines like 10500: from all iif lo oif rmnet_data0 uidrange 0-0 lookup rmnet_data0 in ip rules show. I also don't understand the marking in iptables and how to know how it applies in lines like 13000: from all fwmark 0xd0001/0xdffff iif lo lookup rmnet_data0.

My ultimate goal is to write a shell script that will fix the broken mess that is the WiFi Hotspot routing on this phone. I plan to launch the script after I start up the WiFi Hotspot each time. However, in order to do that, I need to understand what this routing mess is doing first.

I provided all of the output that I believe is relevant to understanding the routing here. Any help is appreciated.

Improve this question
3
  • If you're looking at iptables -L, it's missing the columns that indicate if a particular rule applies to a specific interface only. I recommend iptables -L -vn to see the complete definition of each iptables rule. Commented Feb 21, 2021 at 7:09
  • My own advice on this is to never use iptables -L or similar (-v is mandatory for some usefulness anyway) but use instead iptables-save -c or multiple iptables -S which have an output in the same format as the command you'd type to add rules in the first place. Using iptables -L requires to learn twice how to use iptables. Commented Feb 21, 2021 at 17:47
  • Related: android.stackexchange.com/a/241910/218526 Commented May 25, 2022 at 12:56

1 Answer 1

Reset to default
2

Fairly late on the party here, but. It's pretty difficult to make sense of Android routing without looking at AOSP code, which will help you cast light on a lot of this. You could figure out what it does, but why it does it would be really difficult. Android has much more complicated rules than a regular unix machine because it's typically connected to many networks at the same time ; some processes have access to some networks while others don't ; each UID might have a different default network ; apps can request another network than their default by using a number of API. A unix server or desktop machine doesn't have any of this complexity to deal with, and can get by with much simpler routing.

If you're interested in how rules work, start here for the latest code (you can look back to older branches for the specific version of Android that you're interested in). Device manufacturers may have (and often have) changes on top of that, so details may vary on any specific device.

Generally the idea is that routing starts by looking at each rule in priority order (lower first). If the packet matches the rule, then the kernel will lookup the associated table ; if it finds the destination in there, that's where it sends the packet, otherwise it goes back and evaluates the next rule.

A rule like

10500: from all iif lo oif rmnet_data0 uidrange 0-0 lookup rmnet_data0

is described in man ip-rule. In a nutshell :

  • 10500 is the priority. The routing evaluates the rules in increasing order. Two rules with the same priority have undefined order.
  • from all iif lo oif rmnet_data0 uidrange 0-0 are filters. Each of them is two words :
    • from all means any source will match
    • iif lo (for input interface local) matches only traffic generated locally (not forwarded traffic)
    • oif rmnet_data0 (for output interface) matches only traffic destined for the rmnet_data0 interface
    • uidrange 0-0 matches only the traffic coming from a UID comprised between 0 and 0 (so only UID 0 in this case). Android has a different UID for each app.
  • lookup rmnet_data0 simply means that if the packet matches all the filters, then you should lookup table rmnet_data0 to try and find the route. If there's one, follow it, otherwise go to the next rule.

As for the marking, here is where you want to start. Android uses the firewall mark to manage network permissions and explicit asks from apps. As you can see in the code, the bottom 16 bits in the firewall mark are the ID of the network (Android assigns a 16-bits network ID to each network). Upper bits are used for permissions, accounting...

When you see a filter like fwmark 0xd0001/0xdffff, this means to mask the mark on the socket with the right part (0xdffff here) and compare it to the left part (0xd0001 here). In this case, looking up that code again, 0xdffff will keep bits 0~15 (the netId), bit 16 ("explicitly selected") and bits 18 and 19 (permissions). After that masking, traffic with a socket mark of 0xd0001 will match this rule, so it must be bound for network with netId 1, explicitly selected (which means the application has used some API to ask for this network rather than the default), it must have both permission bits (so it must be a privileged app that holds the android USE_RESTRICTED_NETWORKS permission).

As for how the socket gets marked, this is yet another story, but to keep it short Android uses hooks in syscalls to ship the socket to the (Android-only) netd daemon when you call connect or some Android API that lets you select a network, and netd will mark that socket for you. You can see the logic in this code if you're interested in the details. There are also some iptables rules to mark accepted sockets and a host of technical details that you can lookup in the code if you want to learn more.

Hope this helps,

Improve this answer

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.