With the 0.16.0-2 clients and the changes to the dynamic port settings from my thread above, everything seems to be working with the exception of egress via one of the nodes. When I look at the wireguard config on the other nodes I don't see the egress subnet listed under the allowed ips for the egress node. Any thoughts as to what to try here?

when you create the egress are there any errors in netmaker or netclient on the egress node

Yeah, I didn't realize until just now, but yeah, that node crashes when I add the egress route. No errors on the server side but the last bit of the netclient log is this:

can you post the content of /etc/netclient/config/nm-.conf

Sure, the peers look normal so I'll omit tha part and just post the interface section with the firewall commands:

[Interface]
PrivateKey = <private-key>
ListenPort = 50821
Address    = 10.28.0.4
PostUp     = nft add table ip filter
PostUp     = nft add chain ip filter FORWARD
PostUp     = nft add rule ip filter FORWARD iifname <nm-interface-name> counter accept
PostUp     = nft add rule ip filter FORWARD oifname <nm-interface-name> counter accept
PostUp     = nft add table nat
PostUp     = nft add chain nat postrouting
PostUp     = nft add rule ip nat postrouting oifname <nm-interface-name> counter masquerade
PostUp     = nft add chain ip filter forward
PostUp     = nft add rule filter forward ct state related,established accept
PostUp     = nft add rule ip filter forward iifname <nm-interface-name> accept
PostUp     = nft add rule ip filter forward oifname <nm-interface-name> accept
PostUp     = `nft 'add chain ip nat prerouting { type nat hook prerouting priority 0 ;}'`
PostUp     = `nft 'add chain ip nat postrouting { type nat hook postrouting priority 0 ;}'`
PostUp     = nft add rule ip nat postrouting oifname enp3s0 counter masquerade
PostDown   = nft flush table filter
PostDown   = nft flush table nat

by any chance is that node both an ingress and an egress

It is

which order did you create them

Ingress first

try doing it in the other order -- we had a bug with iptables rules when the ingress was created first ; maybe the same thing with nftables

No dice taking down the ingress gw and just doing egress first before adding back ingress. Same error in the logs as before (needing

netclient pull

to reconnect to the mesh).

what happens if you do wg-quick down /etc/netclient/config/nm-.conf followed by wg-quick up /etc/netclient/config/nm-.conf

wg-quick up

fails with the same message that's in the logs:

Error: Could not process rule: File exists
add chain ip nat postrouting { type nat hook postrouting priority 0 ;}
                 ^^^^^^^^^^^

it comes up ok if you do a netclient pull?

Yeah

that does not make any sense to me (I believe you, just can't figure out what is going on)

0.16.0 server and 0.16.0-2 for all the clients. I assumed the -2 patch was just for the clients.

yes ---- the binary is the same but some of the other files in the deb package changed.

So I should try to upgrade the server?

and after you do a pull, other nodes still don't have 10.15.0.0/16 as an allowedip?

Oh and looking at some of the other nodes it looks like there is a route for the 10.25.0.0/16 subnet now. I think adding the egress first worked for the other nodes that aren't being relayed by that same node

server should be fine;;;;; but you could do a docker-compose pull; docker-compose up -d

When I dropped the ingress for the node all the ext clients went away and I haven't readded them. ace and apogee are relayed via orem-dc-gw

oh.. so you are using that one node as an ingress/egress and a relay

Right.

have you tried making one of the other nodes the relay for ace and apogee?

Orem-dc-gw has an external ip and static port. Most of the others are rely on udp hole punching

gotcha.... the other choice would be to use netmaker-1 as the relay

Yes, since I re-added the egress before the ingress on that node the other nodes get their routes set correctly for 10.25.0.0/16

Glad to help

Though, this technically should work with a node being both a relay and an egress gateway right? Is there any other testing that would be useful to ascertain if this is a bug? E.g., should I open a GH issue or anything? And I guess the ingress/egress creation order bug is already tracked somewhere?

An issue for the relay/egress combo would be good. PR already submitted for the ingress/egress issue