Title
b
big-wolf-4507
09/21/2022, 5:35 PMSo I'm seeing some serious weirdness after upgrading to 16.0. Specifically nodes that previously could talk to each other now no longer can. Looking at the wireguard configs I see the issue is that the ports populated in the config don't match the actual ports that wireguard is listening to on the other node. E.g., for node1 the wg interface is listening on 46780 but the config on node2 has the correct ip, but the wrong port 51938. Same issue from node2 to node1 but again with different ports. What's more if I watch the wg status the ports continuously change about every two minutes.
Oh and one of the nodes that is experiencing this issue has a static port set but other nodes have a completely different port number
I will note that the netmaker host does live close to where some of the nodes are located, but I was able to get it to work on 15.1 by ensuring that there were two public ips assigned to that machine with netmaker receiving the alternate ip and the other traffic going through the primary ip.
Any thought as to what to try or what might be going on?
b
bored-island-21407
09/21/2022, 5:40 PMcan you check the server node-- is dynamic port enabled or not?
b
big-wolf-4507
09/21/2022, 5:42 PMThe server node as in netmaker-1? No, that node does not have dynamic port enabled.
b
bored-island-21407
09/21/2022, 5:42 PMok good
how many nodes are on the network?
b
big-wolf-4507
09/21/2022, 5:43 PMI will also note that on one of the nodes that is on the same network as the netmaker container that the endpoint will occasionally swap from the public ip to a subnet ip
7 nodes
b
bored-island-21407
09/21/2022, 5:44 PMif it has the same pubic ip as the server, it should be using the subnet ip
would it be possible, on all nodes except netmaker-1 and one other, run netclient disconnect -n 'network name' and then monitor the one node to see if the port changes settle down
b
big-wolf-4507
09/21/2022, 5:52 PMYeah, though right now only one node (the only one outside the subnet where the other nodes live with netmaker) and that isn't going through a relay is changing ips constantly
b
bored-island-21407
09/21/2022, 5:54 PMare the ips all public?
need to step away for a bit ... will touch base when I am back.
b
big-wolf-4507
09/21/2022, 5:55 PMAnother addendum is that as I'm comparing the wg configs it looks like aside from the host that is cycling ports the others seem to have consistent but wrong ports to other nodes. E.g., node1 and node3 both have port 4959 associated with node2 but again that's not the port that node2 is listening on.
No, that was a detail I added in a ninja edit to the starting post. Aside from the netmaker-1 node and the node with the manually assigned static port (I'll call it the gateway node) all other nodes currently rely on udp holepunching or relaying via the gateway node.
I'm also noticing that in the wg config for the server node that all the endpoint for every node is 172.18.0.1 (the docker gateway?). Is that expected?
j
jolly-london-20127
09/22/2022, 3:07 PM@big-wolf-4507 does your network have an internet gateway? (0.0.0.0/0)
b
big-wolf-4507
09/22/2022, 3:16 PMYes, if I understand what you're asking @jolly-london-20127 . The netmaker server node is hosted on a machine with external ip and the netmaker ports are bound on that machine from the docker container for netmaker.
j
jolly-london-20127
09/22/2022, 3:31 PMI mean did you set netmaker as an egress gateway to 0.0.0.0/0
b
big-wolf-4507
09/22/2022, 3:35 PMI see. No, the mesh does not have an egress to 0.0.0.0/0 though it does have an egress to a subnet in 10.0.0.0/8
j
jolly-london-20127
09/22/2022, 3:37 PMdo you have any logs showing an issue connecting to the broker?
if you dont mind sharing journalctl logs, would be helpful
b
big-wolf-4507
09/22/2022, 3:55 PMFor sure
There's log exurpts for two of the nodes there
The others I looked at didn't really have anything out of the ordinary in their logs. E.g., just MQ chatter about sending updates. No errors that I saw
Can you verify if the behavior I'm seeing is expected? E.g., that the listening port for wg on a host doesn't match the port corresponding to the udp hole punched port? I can't imagine it would be, but I want to check my assumptions.
Likewise if I go to edit a node in the dashboard and turn off the dynamic port (e.g., so I can set up external port forwarding for that node) it wont let me edit the port number. Is that exepected?
j
jolly-london-20127
09/23/2022, 12:50 AMIt's definitely not expected, but we're having a hard time recreating
you should definitely be able to turn off dynamic port, but you need to first turn it off, click submit, and then you should be able to edit
b
big-wolf-4507
09/23/2022, 4:47 PMWeird! Going through all the nodes that were not working via a relay, turning off dynamic ports and giving them a port that is distinct for the external endpoint makes it all work even without doing port forwarding into the NAT