825071750290210916 #upgrades

handsome-lifeguard-70799

10/07/2022, 3:47 PM

MQ logs were all good, but netmaker logs began throwing errors, all related to an existing egress gateway on a network server node acting as a VPN hub (0.0.0.0/0). I removed the egress gateway and the errors stopped

handsome-lifeguard-70799

10/07/2022, 3:48 PM

Then I ungraded another node providing remote access via egress and ingress gateways. Same type of errors

handsome-lifeguard-70799

10/07/2022, 3:49 PM

`[netclient] 2022-10-07 15:29:27 error running command: wg-quick up /etc/netclient/config/nm-mat.conf [netclient] 2022-10-07 15:29:27 [#] ip link add nm-mat type wireguard [#] wg setconf nm-mat /dev/fd/63 [#] ip -4 address add 10.12.14.7 dev nm-mat [#] ip link set mtu 1280 up dev nm-mat [#] ip -4 route add 10.12.14.9/32 dev nm-mat [#] ip -4 route add 10.12.14.8/32 dev nm-mat [#] ip -4 route add 10.12.14.6/32 dev nm-mat [#] ip -4 route add 10.12.14.5/32 dev nm-mat [#] ip -4 route add 10.12.14.4/32 dev nm-mat [#] ip -4 route add 10.12.14.3/32 dev nm-mat [#] ip -4 route add 10.12.14.254/32 dev nm-mat [#] ip -4 route add 10.12.14.2/32 dev nm-mat [#] ip -4 route add 10.12.14.15/32 dev nm-mat [#] ip -4 route add 10.12.14.14/32 dev nm-mat [#] ip -4 route add 10.12.14.13/32 dev nm-mat [#] ip -4 route add 10.12.14.12/32 dev nm-mat [#] ip -4 route add 10.12.14.11/32 dev nm-mat [#] ip -4 route add 10.12.14.10/32 dev nm-mat [#] ip -4 route add 10.12.14.1/32 dev nm-mat [#] ip -4 route add 192.168.50.0/24 dev nm-mat [#]

iptables -A FORWARD -i nm-mat -j ACCEPT ; iptables -A FORWARD -o nm-mat -j ACCEPT ; iptables -t nat -A POSTROUTING -o nm-mat -j MASQUERADE ;  ; iptables -A FORWARD -i nm-mat -j ACCEPT ; iptables -A FORWARD -o nm-mat -j ACCEPT ; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE ;

/usr/bin/wg-quick: command substitution: line 295: syntax error near unexpected token `;' /usr/bin/wg-quick: command substitution: line 295:

iptables -A FORWARD -i nm-mat -j ACCEPT ; iptables -A FORWARD -o nm-mat -j ACCEPT ; iptables -t nat -A POSTROUTING -o nm-mat -j MASQUERADE ;  ; iptables -A FORWARD -i nm-mat -j ACCEPT ; iptables -A FORWARD -o nm-mat -j ACCEPT ; iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE ; '
[#] ip link delete dev nm-mat

handsome-lifeguard-70799

10/07/2022, 3:50 PM

These errors were thrown on pull BTW

handsome-lifeguard-70799

10/07/2022, 3:50 PM

I removed and readded the egress gateway and no errors on pull, connections working again, but...

handsome-lifeguard-70799

10/07/2022, 3:51 PM

The connection is no longer stable. For instance, remote desktops from full clients and external clients both disconnecting every minute. SSH to the gateway node also dropping out constantly

handsome-lifeguard-70799

10/07/2022, 4:02 PM

The egress gateway is working though, perhaps a bit more settled. I have upgraded a few linux and windows client nodes now, but neither are showing the updated version in the UI. All have gone to warning now, pull not seeming to go through or work as expected. netclient -v shw 16.1 on all upgraded nodes, services definitely running

handsome-lifeguard-70799

10/07/2022, 4:07 PM

[netclient] 2022-10-07 16:06:13 No network selected. Running Pull for all networks.
[netclient] 2022-10-07 16:06:16 waiting for interface...
[netclient] 2022-10-07 16:06:16 interface ready - netclient.. ENGAGE
[netclient] 2022-10-07 16:06:16 pulled network config for mat
[netclient] 2022-10-07 16:06:16 reset network all and peer configs

handsome-lifeguard-70799

10/07/2022, 4:07 PM

Pull -v seems happy on the client end

handsome-lifeguard-70799

10/07/2022, 4:14 PM

Running disconnect / connect in the UI has got them showing a healthy green now

handsome-lifeguard-70799

10/07/2022, 4:14 PM

Still showing 15.1 in the UI though

handsome-lifeguard-70799

10/07/2022, 4:20 PM

0.16.1 nodes are going orange once again. SSH connection drops also continuing

handsome-lifeguard-70799

10/07/2022, 4:20 PM

Let me know if there are any other logs I can send, that probably all I can report for now

handsome-lifeguard-70799

10/07/2022, 4:30 PM

On windows running netclient pull stops the service and doesn't restart it

handsome-lifeguard-70799

10/07/2022, 4:36 PM

That might have been coincidence actually. But the service is definitely dying after a while on windows and linux nodes

handsome-lifeguard-70799

10/07/2022, 4:37 PM

Right, better feed the children, hope some of this is useful

jolly-london-20127

10/07/2022, 5:33 PM

@bored-island-21407 seems like somehow, bad egress/ingress rules are still getting through ("; ; ")

jolly-london-20127

10/07/2022, 5:33 PM

are all clients windows?

jolly-london-20127

10/07/2022, 5:34 PM

nvm, i see the other comment that it's both windows and linux

jolly-london-20127

10/07/2022, 5:34 PM

I wonder if these issues are due to upgrading straight from 0.15.1 to 0.16.1. Not sure if our updater is handling that properly

jolly-london-20127

10/07/2022, 5:35 PM

thanks for this info in any case, we will investigate

bored-island-21407

10/07/2022, 5:36 PM

only thing I can think of is that the gateway was created with an old version of netmaker/netclient (that is the postup/postdown rules where generated before the upgrade).... I think the only way to fix is to remove the gateway and recreate it

handsome-lifeguard-70799

10/07/2022, 9:14 PM

I did a fresh install at 0.15.0 as I moved from Azure to AWS, so it's not that old. 0.15.1 to 16.1 issues maybe..? Removing and recreating the egress gateways has worked ok, but the errors seemed to be thrown by some syntax issues with apostrophes, perhaps. Bigger issue for me is the very unstable connection like I've never experienced before with Netmaker since 0.8. It's not currently usable and I will have to roll back before Monday like this. Netclient service is still up after these hours on the nodes, but I can't get any kind of check-in working. Using disconnect in the UI always makes a node show as healthy, even if it stays disconnected, so that's not really doing anything useful

handsome-lifeguard-70799

10/07/2022, 9:31 PM

Tried leaving and joining all the nodes which updates the UI version to 0.16.1, but they still go orange after some minutes, pull has no effect

handsome-lifeguard-70799

10/07/2022, 9:45 PM

mq doesn't seem happy, this on the client log:

handsome-lifeguard-70799

10/07/2022, 9:46 PM

[netclient.exe] 2022-10-07 22:38:46 [mqpublish.go-149] Hello(): Network: mat error publishing ping, connection timeout 
[netclient.exe] 2022-10-07 22:38:46 [mqpublish.go-150] Hello(): running pull on mat to reconnect 
[netclient.exe] 2022-10-07 22:38:46 [common.go-389] informPortChange(): network: mat UDP hole punching enabled for node SEEHUND 
[netclient.exe] 2022-10-07 22:38:48 [common.go-165] InitWireguard(): waiting for interface... 
[netclient.exe] 2022-10-07 22:38:51 [common.go-193] InitWireguard(): interface ready - netclient.. ENGAGE 
[netclient.exe] 2022-10-07 22:38:56 [mqpublish.go-52] checkin(): checkin with server(s) for all networks 
[netclient.exe] 2022-10-07 22:38:56 [localport.go-40] UpdateLocalListenPort(): network: mat local port has changed from  0  to  62993 
[netclient.exe] 2022-10-07 22:39:26 [mqpublish.go-252] publish(): could not connect to broker at broker.mydomain.org .org:443 
[netclient.exe] 2022-10-07 22:39:26 [localport.go-47] UpdateLocalListenPort(): could not publish local port change connection timeout 
[netclient.exe] 2022-10-07 22:39:40 [daemon.go-67] Daemon(): shutting down netclient daemon 
[netclient.exe] 2022-10-07 22:39:40 [daemon.go-202] messageQueue(): shutting down message queue for server broker.mydomain.org

handsome-lifeguard-70799

10/07/2022, 9:47 PM

Tried to uninstall the clients and start again when I found this

handsome-lifeguard-70799

10/07/2022, 9:59 PM

Oh, here we go. Looks like, after all that, problem may exist between brain and finger, Found a stray character in my broker line in the mq section of my compose. Funnily enough, everything's turned green!

handsome-lifeguard-70799

10/07/2022, 10:24 PM

I compared my compose following the guide with the compose on develop. I had a few things in a different order within each service section, as well as the broker typo. All seems to be working well except for a couple of grumpy nodes where I can't get the service to start, but I think that's from so many joins, leaves, uninstalls. Sorry for f-ups, hope I can let you know how things go over the weekend. Good night!

sticky-microphone-43760

10/08/2022, 4:32 PM

Hi, after upgrading to 0.16.1 and following the upgrade notes, my clients cannot connect via MQTT and I get

Client <unknown> disconnected due to protocol error.

in the MQ logs. Any ideas? This is an unproxied Mosquitto.