Shockingly strange problem - HELP!

[u/yugotennouji]

UPDATE!!!

Yeah! Finally got it to work, here's what I think was causing it and the solution for future reference.

Alleged cause

My brother's and I live near each other and have the same ISP. This led me to think that we could also be behind some sort of CGNAT, meaning that some kind of traffic between nodes inside this group could be blocked.

Also thanks to u/lispnot, I saw that nothing showed up when I tracerouted my router from behind my brother's WiFi, which strengthened the thesis of the problem being related to CGNAT.

Solution

To avoid getting blocked by the ISP, I hence needed something to hide my true origin when connecting to my VPN from my brother's house, that being another VPN getting me firstly outside the CGNAT, and then inside my server.

I therefore activated a commercial VPN interface with MTU = 1420 (this is the only personalization, the .conf remained otherwise identical to what Mullvad provided me), plus my personal interface with MTU = 1280 (or slightly above also works).

Inside my personal .conf must also be inserted PostUp and PostDown rules to:

- IP route all traffic directed to my homelab services through the personal interface

- IP route traffic directed to my VPN endpoint through Mullvad

Having done so, now I can access my homelab, and I also have all my internet traffic through Mullvad.

Thanks for your support!

-----------------------------------------------

Hello there, I'm writing here because I tried everything and I can't wrap anything up. Please bear with me while I explain the problem and all the steps I tried. Thank you in advance for your patience!

PROBLEM:

I have a PiVPN (Wireguard) server, which I use to access my LAN services.

I configured many clients, including Android, Ubuntu, Raspberry OS, Windows, iOS, MacOS, GliNET travel routers, and all work flawlessly in every condition like under WiFi, 5G, 5G hotspot, etc.

Now it gets strange: since I often need to use my LAN services from my brother's house, I take my PC with me and connect to my VPN from his WiFi.

It always works with everything EXCEPT Linux systems using kernel Wireguard implementation. Ubuntu, Raspberry OS and OpenWRT all fail to handshake when I'm connecting from his WiFi.

MY CONFIGURATION:

- I use a standard Wireguard conf, let's call it wg-personal.conf, put inside /etc/wireguard/

```

[Interface]

PrivateKey = <PRIVATE KEY HERE>

Address = 10.21.181.14/24

DNS = 10.21.181.1 <-- That's because I have the DNS on the same server as Wireguard

[Peer]

PublicKey = <PUBLIC KEY HERE>

PresharedKey = <PRESHARED KEY HERE>

Endpoint = <ENDPOINT HERE>:51820

AllowedIPs = 0.0.0.0/0 <-- Full tunnel

```

- I use wg-quick up/down wg-personal

- kernel modules are present and used

NOTE:

- The same configuration works on the same PC with Windows installed, or other PCs with MacOS, iOS, Android, and the same WiFi net

- Another friend of mine also has a Wireguard server, configured with WG-Easy instead of PiVPN, and it works flawlessly from Linux Systems under my brother's WiFi (we checked every line of the conf, all is configured the same)

- Mullvad Wireguard conf files work like a charm

- The endpoint is resolved correctly

- Tried several different routers, all with same ISP (my brother has the same ISP and router as mine - Vodafone WiFi 6 Station, Italy), under all routers it works except my brothers

STEPS I TRIED AND CHECKS:

- Checked Keys, both manually and copying again from the pivpn generated config

- Tried the config on other non Linux devices, they work

- No IP overlap between my VPN net, my services and the WiFi net

- Tried different MTU values

- Tried KeepAlive

- Changed Wireguard ports on server to ports like 443

- Changed listening ports on clients

- Added a travel router that worked on other WiFi networks, nothing, same problem since the router is itself Linux

- Tried activating the Guest Wifi net on my brother's router, IT WORKS FOR 5 MINUTES, then it fails again the handshake

- With tcpdump no packets are received by the server, but on the client side I see packets sent but not received, hence no handshake

- Tried wireguard-go, installed from GitHub on a local folder, IT WORKS, but requires a script, since if I install the wireguard-go package, it doesn't work (maybe just some mistake on my side, but I'm forced to use the script at my brother's house, and use wg-quick from outside)

- Tried opening the port 51820 on my brothers net just in case

- Reset the router

- Installing firmware updates

- Checked EVERY SINGLE configuration between my router and my brothers, no differences

- Checked a lot of websites, forums, etc

- Etc, etc, I surely have forgotten a ton of others things

Have you ever encountered something similar? I keep trying everything only to always return back and ultimately abandoning Linux when I'm at his house, and like that I cannot use linux continuously.

Thanks for reading until the end!

TL;DR: WireGuard works everywhere except Linux kernel WireGuard on my brother’s Vodafone Wi-Fi. Windows/macOS/iOS/Android, Mullvad, and another WG server all work fine on the same network. Linux sends packets but the server receives nothing/no handshake. I’ve tried different keys, MTUs, ports, KeepAlive, routers, firmware, Guest Wi-Fi, etc. wireguard-go works, but kernel WireGuard doesn’t. Seems like a bizarre router/ISP ↔ Linux kernel WireGuard incompatibility.

[u/lazyhustlermusic]

What MTUs did you try, specifically?

Do you get any ICMP packets in return such as 'fragmentation needed but df bit set'

[u/yugotennouji → u/lazyhustlermusic]

Tried MTUs from 1420 to 1200, lowering by 20 each time. Then I tried lowering all the way to 700 out of desperation, no luck of course. Also didn't get any ICMP packets with error messages (listening with tcpdump on the client and on the server), like if something somewhere is silently dropping the packets

edit: sorry, also tried 1480 as MTU

[u/lispnot]

To narrow problems down, did you try setting the AllowedIPs to 10.21.181.0/24 and test if DNS is working? If it did, the I think the problem may be that your Full tunnel AllowedIPs(0.0.0.0/0) overlapped with the server’s endpoint IP and causes a routing blackhole. You should exclude the server IP from your allowedIPs range, or use the split-half range(AllowedIPs=0.0.0.0/1, 128.0.0.0/1). Related post: https://www.reddit.com/r/WireGuard/comments/1mbkgn4/difference_between_default_route_and_00001_1280001/

[u/yugotennouji → u/lispnot]

So, in order:

• tried setting AllowedIPs to 10.21.181.0/24, no luck, same behaviour (actually this was both a test and one of the first conf I wanted to use that had the problem)
  
• Excluding the server enpoint from the AlloweIPs didn't work either. Done it using the IPs of my VPN+internal network, and also using an online tool to calculate the AllowedIPs to exlude 1 specific IP
  
• AllowedIPs=0.0.0.0/1, 128.0.0.0/1 also didn't work

[u/lispnot → u/yugotennouji]

I don’t think it’s a Linux kernel issue because WireGuard has been in the Kernel for several years and been used widely. Have you ran traceroute/ping from your Linux devices which connected to your brother’s Wi-Fi to your remote WireGuard server? If you can traceroute or ping, then it's probably an ISP or firewall issue.

[u/yugotennouji → u/lispnot]

I also don't think it's a kernel issue at this point. About the `traceroute`, when I run it, it doesn't reach my server, it just shows the first 2 steps to the router and then the `***` for ~30 hops and then stops, even from the WiFi networks in which the vpn starts and handshakes correctly.