https://netmaker.org logo
Powered by
Title
l

lively-salesclerk-37295

08/20/2022, 2:48 PM
How should I configure the broker (mqtt) in regards to the reverse proxy (nginx)? I can see that this is not a simple http-based config. In traefik: 
yml
      - traefik.enable=true
      - traefik.tcp.routers.mqtts.rule=HostSNI(`broker.NETMAKER_BASE_DOMAIN`)
      - traefik.tcp.routers.mqtts.tls.passthrough=true
      - traefik.tcp.services.mqtts-svc.loadbalancer.server.port=8883
      - traefik.tcp.routers.mqtts.service=mqtts-svc
      - traefik.tcp.routers.mqtts.entrypoints=websecure
As far as I understand, I take the 
raw stream
on port 443 and pass it to mqtt port 
8883
without any form of TLS decapsulation? Should nginx apply TLS or leave that to mqtt as well?
b

bored-island-21407

08/20/2022, 2:51 PM
I am not 100% familiar with nginx proxying of non http traffic. I believe you just pass the raw stream.
l

lively-salesclerk-37295

08/20/2022, 2:51 PM
I tried that, but when I access the broker via browser it returns 
NET::ERR_CERT_INVALID
is this expected?
b

bored-island-21407

08/20/2022, 2:52 PM
yes, netmaker mqtt connections expect a client cert which your browser will not have. Best way to test would be with mosquitto_pub/mosquitto_sub.
see mq troubleshooting gist for instructions
l

lively-salesclerk-37295

08/20/2022, 2:53 PM
Ah. So just to confirm I got this right, reverse proxy will determine whether the traffic is destined for MQ or any other service based on TLS SNI field, and if it's for MQ, just pass the raw stream without any alteration?
b

bored-island-21407

08/20/2022, 2:53 PM
https://gist.github.com/mattkasun/face2a7c1f32031a2126ff7243caad12
I believe that is correct
l

lively-salesclerk-37295

08/20/2022, 8:49 PM
Just to follow up, it turns out that nginx cannot map raw streams and http streams on the same port, at least not without some significant refactoring of my existing configs... So I have exposed a custom port from MQTT directly and use that in the config.
Hmm, I'm pretty sure I did everything correctly, but now i keep getting this error whenever I connect my Windows client to the network 
1661029220: Client <unknown> disconnected: Protocol error.
1661029224: New connection from xxx:yyy on port zzz.
1661029224: OpenSSL Error[0]: error:14094412:SSL routines:ssl3_read_bytes:sslv3 alert bad certificate
I tried a full reinstall of all containers and volumes but nothing changed. I also tried the troubeshooting steps in above gist, but no help either.
b

bored-island-21407

08/20/2022, 9:21 PM
when you do the troubleshooting steps --- do you get errors with mosquitto_pub?
l

lively-salesclerk-37295

08/20/2022, 9:21 PM
I didn't really understand mosquitto_pub so I didn't test that, but I did try the sqlite delete part
b

bored-island-21407

08/20/2022, 9:22 PM
when you deleted the certs and after you restarted the netmaker container, did you also wait and then restart the mosquitto container
l

lively-salesclerk-37295

08/20/2022, 9:23 PM
define wait..? I waited 1-2 seconds
as fast as I could write the next restart command basically ^^
b

bored-island-21407

08/20/2022, 9:23 PM
need to wait until netmaker has completely started.... should wait 10-30 seconds
l

lively-salesclerk-37295

08/20/2022, 9:23 PM
sure, will try now and monitor logs
I changed the port from 8883 to a custom one and updated the mosquitto config as well as 
MQ_PORT
is that okay?
b

bored-island-21407

08/20/2022, 9:24 PM
yes, that should work
l

lively-salesclerk-37295

08/20/2022, 9:24 PM
[netmaker] 2022-08-20 21:23:54 checking keys and certificates
[netmaker] 2022-08-20 21:23:54 generating new root key
[netmaker] 2022-08-20 21:23:54 generating new root CA
[netmaker] 2022-08-20 21:23:54 generating new server key/certificate
[netmaker] 2022-08-20 21:23:54 generating new server client key/certificate
[netmaker] 2022-08-20 21:23:54 ensure the root.pem, root.key, server.pem, and server.key files are updated on your broker
[netmaker] 2022-08-20 21:23:54 REST Server successfully started on port  8081  (REST)
[netmaker] 2022-08-20 21:23:54 connecting to mq broker at netmaker-mq:1883 with TLS? false
[netmaker] 2022-08-20 21:23:55 successfully connected to mq broker
now I restart mqtt?
b

bored-island-21407

08/20/2022, 9:24 PM
yes
and then you need to do netclient pull -n on your client
l

lively-salesclerk-37295

08/20/2022, 9:25 PM
can I just not delete the client network and use a new one?
that's what I've been doing until now
b

bored-island-21407

08/20/2022, 9:25 PM
that works too
l

lively-salesclerk-37295

08/20/2022, 9:25 PM
testing
nope same error
it spams bad cert until the client times out and continues without mqtt connection
b

bored-island-21407

08/20/2022, 9:27 PM
do a netclient pull -n
l

lively-salesclerk-37295

08/20/2022, 9:27 PM
pull succeeds but mqtt logs only show bad certificate
I am using Windows GUI fwiw
from console it just says 
[netclient.exe] 2022-08-20 22:27:22 could not connect to broker at website:port
b

bored-island-21407

08/20/2022, 9:29 PM
dns or firewall
l

lively-salesclerk-37295

08/20/2022, 9:30 PM
wait I think linux works
linux client that is
though I am getting a different error here: 
2022/08/20 22:29:53 daemon restart failed  failed to find pid could not read pid file open /var/run/netclient.pid: no such file or directory
[netclient-arm64] 2022-08-20 22:29:54 error running command: systemctl restart netclient.service 
[netclient-arm64] 2022-08-20 22:29:54 Failed to restart netclient.service: Unit netclient.service not found.
b

bored-island-21407

08/20/2022, 9:31 PM
how did you install netclient on linux (distro?)
l

lively-salesclerk-37295

08/20/2022, 9:31 PM
wget netclient-arm64 on Ubuntu 22.04 arm64
I thought netclient was portable?
b

bored-island-21407

08/20/2022, 9:33 PM
run 
.netclient install
before you do the join or run 
./netclient join
with 
--deamon install
flag
l

lively-salesclerk-37295

08/20/2022, 9:33 PM
ah gotcha, testing
yep so I can confirm mqtt works on linux
I can clearly see in the logs incoming connection over public port and the user's username
has windows been tested if it works with latest version?
b

bored-island-21407

08/20/2022, 9:34 PM
yes it was
l

lively-salesclerk-37295

08/20/2022, 9:35 PM
FWIW this is Windows arm64 ran in Parallels under MacBook Pro M1 Max
So netclient likely runs under emulation
WireGuard is fully arm64 though, so as far as I understand, there should be no issues as you just invoke the already installed (and native) wireguard
actually nvm, mqtt happens before wireguard anyway
hmm I just tried a clean install of netclient on a clean Ubuntu and now I get this: 
[netclient-arm64] 2022-08-20 22:38:09 network: mesh error setting cfg.Node.Endpoint. 
[netclient-arm64] 2022-08-20 22:38:09 error installing:  public address not found
b

bored-island-21407

08/20/2022, 9:40 PM
that is output of which command? join? daemon?
l

lively-salesclerk-37295

08/20/2022, 9:40 PM
join
so I did install first, worked, then the join returns this
b

bored-island-21407

08/20/2022, 9:40 PM
run join with -vvvv flag
l

lively-salesclerk-37295

08/20/2022, 9:41 PM
[netclient-arm64] 2022-08-20 22:40:49 [join.go-91] JoinNetwork(): network: mesh error setting cfg.Node.Endpoint. 
[netclient-arm64] 2022-08-20 22:40:49 [commands.go-23] Join(): error installing:  public address not found 
[netclient-arm64] 2022-08-20 22:40:49 [systemd.go-124] RemoveSystemDServices(): removed systemd remnants if any existed
b

bored-island-21407

08/20/2022, 9:42 PM
what does 
curl ifconfig.me
return on that machine
l

lively-salesclerk-37295

08/20/2022, 9:44 PM
hmm actually it appears that the DNS somehow got broken on this machine
which would make sense, it can't resolve the IP for the server's hostname
I'll deal with that later, it worked once anyway
b

bored-island-21407

08/20/2022, 9:44 PM
systemctl restart systemd-resolved
l

lively-salesclerk-37295

08/20/2022, 9:45 PM
yeah fixed, interesting
joining also worked, no mqtt errors
so linux is sorted,. thanks! do you have any ideas why windows may be broken?
b

bored-island-21407

08/20/2022, 9:46 PM
no idea... did you try the msi installer or just downloaded the exe from the release page?
l

lively-salesclerk-37295

08/20/2022, 9:47 PM
so I tried exe first, but ran into: https://github.com/gravitl/netmaker/issues/1282
then I used MSI which didn't hit that error, but then cert issue arose
b

bored-island-21407

08/20/2022, 9:48 PM
did you say you were running on arm64 .... the only windows exe we build is for amd64
l

lively-salesclerk-37295

08/20/2022, 9:49 PM
yeah agreed, so netmaker is definitely running under emulation, but would that result in cert issues?
b

bored-island-21407

08/20/2022, 9:49 PM
it shouldn't run at all
l

lively-salesclerk-37295

08/20/2022, 9:49 PM
why not? the built-in Windows amd64 to arm64 emulation is incredibly good
I have yet to encounter any app that malfunctions or doesn't run
b

bored-island-21407

08/20/2022, 9:50 PM
ok, i will take your word for it
l

lively-salesclerk-37295

08/20/2022, 9:50 PM
the only issue I've encountered so far is with drivers, which for security and stability reasons will not be emulated
but netclient does not add any drivers, right? it only uses the existing wireguard install, and even then, through its CLI
b

bored-island-21407

08/20/2022, 9:51 PM
netclient does not add any drivers
l

lively-salesclerk-37295

08/20/2022, 9:52 PM
btw, it seems like 
netclient install
breaks my DNS on Ubuntu
I just tried again on a fresh Ubuntu 22.04 arm64 install, and before 
netclient install
, ping 
google.com
works, right after, it fails with 
Temporary failure in name resolution
b

bored-island-21407

08/20/2022, 9:54 PM
netclient install should not affect your dns... all it does is copy the binary to /usr/sbin install the netclient.service file, systemctl daemon-reload, systemclt enable netclient
l

lively-salesclerk-37295

08/20/2022, 9:55 PM
yeah, this is why it's strange
I can try again and record the commands, 1 sec
b

bored-island-21407

08/20/2022, 9:55 PM
try 
systemctl --failed
after the install
l

lively-salesclerk-37295

08/20/2022, 9:57 PM
root@user:~# systemctl --failed
  UNIT LOAD ACTIVE SUB DESCRIPTION
0 loaded units listed.
b

bored-island-21407

08/20/2022, 9:58 PM
systemctl status systemd-resolved
l

lively-salesclerk-37295

08/20/2022, 9:58 PM
here btw: https://asciinema.org/a/rJT8gLvKiiprDPsvhDfrPPVeR
(always wanted to do that)
I diffed 
systemctl status systemd-resolved
before and after, absolutely no difference
i.e. the service does not even get restarted
b

bored-island-21407

08/20/2022, 10:01 PM
but systemctl restart systemd-resolved get dns working again, correct
l

lively-salesclerk-37295

08/20/2022, 10:02 PM
yup
seems like systemd-networkd is restarted after netclient service installation though
here
b

bored-island-21407

08/20/2022, 10:03 PM
that is weird ... the only thing that could possible to that I think is systemctl daemon-reload
l

lively-salesclerk-37295

08/20/2022, 10:04 PM
I ran 
systemctl daemon-reload
and it did not break DNS
what does the netclient service depend on? it may be restarting something purely on dependency
b

bored-island-21407

08/20/2022, 10:05 PM
Description=Netclient Daemon Documentation=https://docs.netmaker.org https://k8s.netmaker.org After=network-online.target Wants=network-online.target systemd-networkd-wait-online.service [Service] User=root Type=simple ExecStart=/sbin/netclient daemon Restart=on-failure RestartSec=15s [Install] WantedBy=multi-user.target
l

lively-salesclerk-37295

08/20/2022, 10:05 PM
hmm nothing strange here
cannot reproduce on a cloud instance with Ubuntu amd64, so this is likely a quirk of my VM
btw, is the Windows standalone EXE supposed to work by itself? Or do you need to download 
winsw.exe
and put it next to it?
good news, I reset my Windows VM and now the mqtt certs work!
perhaps something broke during experimentation and netclient cached the "bad" certs on the Windows box
#netmakerJoin Discord
Powered by