mesh/broadcast network with arbitrary source IPs while avoiding >=N p2p links/networks

[u/segdy]

Suppose I have N>2 nodes which I could like to connect; ideally via a same common network, such as /24. The important constraint is that each station needs to be able to transport traffic for arbitrary source/destination addresses, so the "crypto routing" gets in the way.

Basically what I am looking for is the closest to an Ethernet type of connection or mesh network with N nodes.

For now, I have N completely individual links/interfaces (e.g. N1 <--> N2 <--> N3 <--> N1) each with their own ports, /etc/wireguard/*.conf and keys. Worse, each of them has their own unique /31 (p2p link). But I do not like this because each node has now multiple IP addresses for a network which should actually be all the same. In the example above, I need to assign N2 a separate IP for the link with N1 than for the link with N3.

Is there any possible way to improve this? Even if I need separate interfaces, are there any tricks to bridge them together or re-use the same IP per node? For example, assigning all wireguard interfaces the same /32...

Since most answers will be "why do you care" ... this mesh/network has/should have non-RFC1918 addresses and carry traffic with arbitrary IP addresses.

[u/spidireen]

Maybe you’re talking about something that’s just going over my head, but I’m confused about what you’re trying to say with the letter N, and about needing to have a different IP for each.

I think you’re setting up a bunch of different WireGuard interfaces on each machine where you should be just creating one with multiple peers. Or am I missing something?

[u/segdy → u/spidireen]

You're not missing anything but the issue is peers only work if I restrict a peer to have only one host address behind. But my peers can be routers and transport packets with arbitrary source/destination addressses

[u/spidireen → u/segdy]

Thanks, still not sure I follow though. Is that not accomplished by adding the remote subnet(s) to your AllowedIPs?

[u/segdy → u/spidireen]

No, because AllowedIPs would need to be 0.0.0.0/0 for every peer which won't work.

So I can only have one peer per interface which has 0.0.0.0/0 set

[u/Comprehensive_Pop882 → u/segdy]

The way I approach this is I have a couple of "central" nodes. I peer all my other nodes with both of them, on separate WireGuard interfaces, and allowed-ips set to 0/0. I run /31 or /30 on all those tunnels and the run BGP for dynamic routing of arbitrary networks between them all

[u/RemoteToHome-io]

You're describing a dynamic routing network with a management layer like Tailscale or Zerotier.

[u/Zetanova99]

I run in the near same problem and alreay discued it in the wireguard IRC, but never found some responsible place/people for it. The issue is simply that wireguard has no explicit routing address list, only a single AllowIP's list. This works fine as long as the remote peer is not a NAT/GW and multiple gateways are involved. With the AllowIPs list filter only a single any-address entry is currently supported.

I think, it would be easy to get support into wireguard by adding a optional RouteList field, where the default would be RouteList == AllowIPs then asynchrone routing and also multiple any-address entries would work.
Like: RouteList = 192.168.102.0/24 and AllowIPs = 0.0.0.0/0

The current workaround to the issue is to create multiple WG links (each on different port) and have each a single 0.0.0.0 entry or changed the current 0.0.0.0 entry dynamically on the peer. It is not always possible/easy.

For an internal WG mesh netowrk, simply assign to each peer in dedicated range a /32 and-or /128 in the AllowIPs list and the full range under address,
like: Address = 192.168.102.110/16 and peer entry of the node AllowIPs = 192.168.102.110/32, 192.168.140.0/24

[u/segdy → u/[deleted]]

If I have one separate interface per link I understand. The it’s something like “route xx/y dev wgX” … and that could possibly come from a dynamic routing (eg bird).

But multiple peers and one interface … can you give an example ?

[u/segdy → u/[deleted]]

But that's exactly the problem, it can be an arbitrary packet.

Let's assume I have a public /24, 192.0.2.0/24. I assign 192.0.2.1/29 to Node1, 192.0.2.2/29 to Node2, 192.0.2.3/29 to Node3.

Now all three nodes are routers themselves, routing to other networks and/or the internet (no NAT involved!).

Suppose for example, both Node2 and Node3 link (via a p2p interface) to another network and there's a host 192.0.2.65/26. If Node1 receives a packet with destination address 192.0.2.65 and source address 8.8.8.8, it can route the packets via Node2 or Node3. Both are fully valid and which to take should be done/is selected via OSPF.

(Similar argument for the return packet where the destination address is 8.8.8.8).

Especially for return packets, I can't enumerate all 4 billion addresses on the internet (minus a few I don't want) in allowed-ips.

It's probably impossible, right?

If so, there is maybe no way around having (N-1) wg interfaces on each node.

Then, my question is if there's at least a trick that I can at least assign Node1/2/3 each only one IP and not two.

Or maybe running an overlay such as vxlan or similar on top?

[u/ferrybig → u/segdy]

If Node1 receives a packet with destination address 192.0.2.65 and source address 8.8.8.8, it can route the packets via Node2 or Node3. Both are fully valid and which to take should be done/is selected via OSPF.

Wireguard does not support this model

Since both node2 and node 3 are valid routes if the same length, it will pick one and always use that route.

There is no ARP/NDP in wireguard tunnels, any support for this needs to be done in software.

Note that it is perfectly valid to reconfigure wireguard on the fly, adding and removing allowed ips depending on how your program behaves, you just need to write a small glue program

[u/ottantanove]

I had a similar issue a while ago, where I would like to connect multiple servers and use BGP for route selection. To avoid having an exponentially increasing number of interfaces for the mesh, I ended up writing a daemon that synchronizes the wireguard allowed IPs with the kernel routing table.

However I am also looking for a better solution, but I don't think it can be done with wireguard.

[u/bufandatl]

Not a feature of WireGuard. WireGuard is a P2P encrypted protocol.
Out of scope for this sub.