grpc_pass grpc://*InternIP*:50051;

ooooh if he moved from caddy to nginx that would definitely be the issue

I have this. I have been using nginx from the beginning. As I said previously, I just moved the working installation to a new server. Config remains the same except public IP part. I have updated public IP where needed.

did you generate new tls certs for this server?

Used the same old let's encrypt cert.

I can browser API & GRC endpoint from browser. SSL works fine.

API root returns "404 page not found" page

GRPC page returns "502 Bad Gateway" page. I guess because of request is not a GRPC request?

@purple-restaurant-77209 is the UI still working?

Yes

I revert the server migration. I will try another day.

ok, let me know how that goes, I feel pretty confident it's a TLS cert issue

what is the node "warning" label typically indicative of?

it means the node hasn't "checked in" with the server in a few minutes

if it goes on for like 30 minutes it will turn to "error"

indicating either the machine is off or something is wrong client side

aight.... my netmaker server has crashed again for the hundredth time.... I'm gonna move it to a newer version of debian before I go poking too far into this Notable issues include: Completely locking up the VM ~ I cant even ssh into the box, have to go to AWS UI and clicky reset it. Starting / resetting? And not mounting volumes correctly. Right now it's in a state that looks like it's a brand new setup (asking to create an admin user), but if I try to create a user. "Could not reach server error" from the UI.

021-12-05T10:24:50.002766356Z 2021/12/05 10:24:50 [netmaker] connecting to sqlite
2021-12-05T10:24:50.013564870Z 2021/12/05 10:24:50 [netmaker] database successfully connected
2021-12-05T10:24:50.013586630Z 2021/12/05 10:24:50 [netmaker] no OAuth provider found or not configured, continuing without OAuth
2021-12-05T10:24:50.306467901Z 2021/12/05 10:24:50 [netmaker] Agent Server successfully started on port 50051 (gRPC)
2021-12-05T10:24:50.329134069Z 2021/12/05 10:24:50 [netmaker] failed to parse interface
2021-12-05T10:24:50.383131030Z 2021/12/05 10:24:50 [netmaker] failed to parse interface
2021-12-05T10:24:50.395657716Z 2021/12/05 10:24:50 [netmaker] REST Server successfully started on port 8081 (REST)
2021-12-05T10:24:50.440182873Z 2021/12/05 10:24:50 [netmaker] failed to parse interface

If I run docker-compose up -d --force-recreate everything is back to normal. Anything else yall would like me to check before I nuke this box?

your are running out of memory https://github.com/gravitl/netmaker/issues/533

I've experienced the same issue, I haven't found the cause and haven't had the time to look further into it. I too have netmaker running on an AWS EC2 instance Huh, running out of memory 😛 guess I have to scale up too

mystery solved

checks instance it's definitely tiny AF

workaround add

docker-compose stop netmaker; docker-compose up -d

to a cronjob

I can bring it on prem, I've been meaning to anyway, is there a recommended memory allocation? Or is the memory leak eventually going to kill any size server?

I think it will kill any sized server

on my DO droplet, it starts off at about 2% of mem and slowly increase to +50% at which time I get a memory usage alert and I restart the container

any idea what's causing it? Like is it something in the UI, say if I run the binary setup only without docker will the same thing happen, or is it a docker only problem?

not sure. It is the netmaker container, not the UI.

intradesting

I haven't tested with just the binary (no container)

@User has indicated that a refactor of the codebase is in the works, so that may take care of it