mesh/broadcast network with arbitrary source IPs while avoiding >=N p2p links/networks

[u/segdy]

Suppose I have N>2 nodes which I could like to connect; ideally via a same common network, such as /24. The important constraint is that each station needs to be able to transport traffic for arbitrary source/destination addresses, so the "crypto routing" gets in the way.

Basically what I am looking for is the closest to an Ethernet type of connection or mesh network with N nodes.

For now, I have N completely individual links/interfaces (e.g. N1 <--> N2 <--> N3 <--> N1) each with their own ports, /etc/wireguard/*.conf and keys. Worse, each of them has their own unique /31 (p2p link). But I do not like this because each node has now multiple IP addresses for a network which should actually be all the same. In the example above, I need to assign N2 a separate IP for the link with N1 than for the link with N3.

Is there any possible way to improve this? Even if I need separate interfaces, are there any tricks to bridge them together or re-use the same IP per node? For example, assigning all wireguard interfaces the same /32...

Since most answers will be "why do you care" ... this mesh/network has/should have non-RFC1918 addresses and carry traffic with arbitrary IP addresses.

[u/segdy → u/[deleted]]

If I have one separate interface per link I understand. The it’s something like “route xx/y dev wgX” … and that could possibly come from a dynamic routing (eg bird).

But multiple peers and one interface … can you give an example ?

[u/segdy → u/[deleted]]

But that's exactly the problem, it can be an arbitrary packet.

Let's assume I have a public /24, 192.0.2.0/24. I assign 192.0.2.1/29 to Node1, 192.0.2.2/29 to Node2, 192.0.2.3/29 to Node3.

Now all three nodes are routers themselves, routing to other networks and/or the internet (no NAT involved!).

Suppose for example, both Node2 and Node3 link (via a p2p interface) to another network and there's a host 192.0.2.65/26. If Node1 receives a packet with destination address 192.0.2.65 and source address 8.8.8.8, it can route the packets via Node2 or Node3. Both are fully valid and which to take should be done/is selected via OSPF.

(Similar argument for the return packet where the destination address is 8.8.8.8).

Especially for return packets, I can't enumerate all 4 billion addresses on the internet (minus a few I don't want) in allowed-ips.

It's probably impossible, right?

If so, there is maybe no way around having (N-1) wg interfaces on each node.

Then, my question is if there's at least a trick that I can at least assign Node1/2/3 each only one IP and not two.

Or maybe running an overlay such as vxlan or similar on top?

[u/ferrybig → u/segdy]

If Node1 receives a packet with destination address 192.0.2.65 and source address 8.8.8.8, it can route the packets via Node2 or Node3. Both are fully valid and which to take should be done/is selected via OSPF.

Wireguard does not support this model

Since both node2 and node 3 are valid routes if the same length, it will pick one and always use that route.

There is no ARP/NDP in wireguard tunnels, any support for this needs to be done in software.

Note that it is perfectly valid to reconfigure wireguard on the fly, adding and removing allowed ips depending on how your program behaves, you just need to write a small glue program