writing additional DB support is VERY easy if you need mysql

Good to know, but some time may have people like to use MySQL 🙂

why does MQ need public access?

Completely ignorant question, I have no idea what it's doing

it's a fair question. It's basically replacing gRPC. We previously had it going over WireGuard, but this created a lot of chicken and egg problems. You have to maintain the WG interface over the same channel where updates are being sent. We often had to send updates and hope that the interface would update correctly. If something goes wrong, there's no way to recover bc the wg config is off. In the new model, it's using certs generated by the server for encryption, and every machine gets its own pair of client certs, so is still be very secure.

ayy yall finally went nebula 😛 I'm a fan of the cert approach

out of curiosity, with the multiple wg tunnels between machines (and I thought I saw you mention something about a mangement plane), can you not just update over a constant management tunnel? Or does the update require completely shutting wireguard down

that's basically what we had, a wg management tunnel. But it would still need to be updated sometimes which caused issues

some updates require wg to be brought down and then back up

yeah, that's a tough cookie ...sort of in the same vein, I had the bonkers idea of tying this https://github.com/RITRedteam/vishnu to a rotating MFA code. So the main server is the rotating code source of truth, and then if you lose access the port knocker hits the server in the MFA order to re-open ports

Like I said tho.... bonkers

damn lol

Posting this to get some feedback. Based on @rich-agency-14084 's article, we're considering moving the default proxy to Traefik. By running MQ behind Traefik, we can make it reachable over 443, which will reduce firewall problems in some environments. It also reduces the number of exposed ports overall, and makes the architecture more simple (entire application is reachable over 443, except for WireGuard ports): https://medium.com/@panda1100/a-solution-to-use-port-443-for-client-certificate-authentication-of-netmaker-v0-14-0-455777589b12 Does anyone have objection to this? I know we've been with Caddy for quite a while, but since it's dockerized, sticking in Traefik should be relatively simple. There is also the option to use Caddy based on @rich-agency-14084 's latest guide, but it is much more complicated and based on an experimental module, which is not ideal: https://medium.com/@panda1100/how-to-setup-layer-4-reverse-proxy-to-multiplex-tls-traffic-with-sni-routing-a226c8168826

I prefer Traefik, dynamic configuration is quite handy.

Traefik would be a good move! Go for it!

I use traefik in a lot of places. It hasn't given me a reason not to. Would concur

I support this move... I have nothing against Caddy, but with tls passthrough for MQTT working out of the box with Traefik, it would be nice to provide it as the best defaults for easy setup. You'll then be free to remove my repo from your docs, too 😉

Yeah these are all good arguments. I'm gonna give it a go this weekend. OR, if you want, you can do the PR, then you still own all your hard-earned traefik work

lol, i'm indifferent... the only thing blocking this with the current docker build is the port number for the broker, as i was mentioning back here: https://discord.com/channels/825071750290210916/825071821594034196/979770745066516520 but sure, i'd be pleased to give you a PR with that port change and a default docker-compose.yml using traefik

Yup! the fix for that is already in develop if you feel like giving it a go

I'd be up for changing caddy to traefik

Architecture wise, wouldn't you rather use MQTT over websockets? The initial connect would still be over https to upgrade the connection to websockets. Both Mosquitto and Traefik support websockets.

I think it would depend on how scalable the developers imagine the application to be. Once you hit a large amount of subscribers, using mqtt or websockets makes a difference in memory, cpu and bandwidth usage. WebSocket works better when it comes to CPU and memory usage, However MQTT is far superior when it comes to network usage, since it uses five times less network than WebSockets. At a thousand subs you would be seeing MB/s vs KB/s.

@microscopic-appointment-2506 Interesting point. I actually don’t know much about pros and cons. I’m HPC engineer and new to mqtt. Could you give me an example when websocket shines in the context of mqtt client certificate authentication compared to direct mqtts. When you choose websockets over mqtt? and why?

Yup the fix for that is already in

@white-piano-73111 @rich-agency-14084 I did not mean websockets instead of MQTT, I meant MQTT over websockets, as explained here: http://www.steves-internet-guide.com/mqtt-websockets/. The inner data is still the MQTT data packet, with an added websocket enveloppe. The reason websockets exist is to use the HTTP handshake to establish a connection (to overcome the initial problem that started this discussion where firewalls only have port 443 open), and to allow bidirectional communication. Websocket support comes out of the box with Mosquitto (and many other MQ brokers). The netmaker team would need to make the netclient compatible with adding a websocket enveloppe. Example code is here: https://github.com/zhengjia/go-mqtt-websocket. The TLS side of the connection does not need to change: TLS is supported, also client certs are.

Traefik with tcp routing is pretty simple, adding a few lines of configuration for traefik and no code changes if you use upcoming release of Netmaker. I would be happy to know the use case of mqtt over websocket. I’m thinking if advanced firewall may block mqtts on port 443 but not block mqtt over websockets on port 443, if that’s the case mqtt over secure websockets becomes interesting solution. Paho-mqtt-golang, that netmaker uses, supports mqtt over websocket if remembered correctly.

Something like ALPN filtering on Firewall

Oeh, nice you guys followed my suggestion and made the jump to MQTT over websockets in 0.17 😄

@hereAre there any independent security researchers or pen testers in the community? If so, we'd like about half an hour of your time to answer a question about a proposed architecture change! We will also compensate you for the time 😃 ! Thanks in advance!