Our core mission is driving the future of distributed cloud computing. The infrastructure of the internet is siloed and byzantine. We can do better.

GRAVITL

if you click the egress on the graph you should get some info

the eggress gateway ranges are correct when i hit edit.

maybe on the opnsense I need to set the route to hit the netmaker hosts ip instead.

ideally netmaker would map out the ranges so that you can access internal ips through the egress

either that is not happening, or something else is blocking the connection

it might be worth a shot to just look at the raw wireguard files and see what netmaker is doing

what happens when you traceroute an internal IP from the vps? does it go through the egress?

for it to map it out wouldn't I need to test from the docker container that is running netmaker and not that actually host netmaker is installed on top of.

wireguard runs on the kernel so i don't think so

its going to the firwall first to find the host on 192.168.5.x network, but that would be true cause the actual host is not netmaker.

in that case the problem might be with the firewall

yeah i don't know how this is supposed to work fully.

i would think the docker container is what would have access to the network where the wireguard connection is made would be where i would really have to test from.

i tried connecting to the docker container with docker container attach and id and it never gave me a command prompt and when i hit ctrl-c it killed the whole docker container and i had to reboot.

wireguard has a kernel module so that it's faster

so while the container applies the configuration, it works system wide

ok then something is not right the connection shows its made but the route doesn't go through the tunnel.

it i do ip route get nvr ip it shows that the route is on the interface via the vlan11 that i am on

wonder where the routes are set so it knows the what path to take for networks on egress

Does anyone have a working example of server deployment to k8s for 0.17.1 ?

<@1044191462834847784> try using just the yaml configs from the Netmaker repo for now. We're going to get the helm charts updated this week but in the meantime, this should work: https://github.com/gravitl/netmaker/tree/master/k8s/server

have managed to hack together something that is starting to work

Ahhh ok, I'm guessing our switch to websockets is screwing with the implementation

yeah sounds like, cause we hacked around with the mosquito conf

apiVersion: v1
data:
  mosquitto.conf: |
    per_listener_settings true
    listener 8883
    allow_anonymous false
    listener 1883
    protocol websockets
    allow_anonymous true
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/instance: mosquitto
    app.kubernetes.io/name: mosquitto
  name: mosquitto-config
  namespace: netmaker

oooooh yes that would do it, looks like we didn't update the mosquitto config

which im not sure if that we have is secure or not but we are still early POC and just needed to unblock so

<@738151527344111666> here are the logs when running server on `image: gravitl/netmaker:v0.17.1`

Something else that generated some confusion is the mentions of grpc dns endpoints in the newer docs ... which we currently dont have configured

Thanks <@1044191462834847784> we'll have a look. Grpc should not be necessary in any newer versions, can you point to where in the docs this is mentioned?

in the kubernetes install docs here: https://docs.netmaker.org/server-installation.html#kubernetes-install

ahhh yeah, those docs are quite out of date

by the way we switched out the EKS nodes for latest bottlerocket AMI which ships with kernel wireguard and server is up and running nicely for now

having issues  with I guess connection to mosquitto after following the guides

<@1044191462834847784> wish i could help.

Anyone able to help me with a little bit of understanding 1) Ingress Host should it be able to reach the networks without accessing the docker container. So the hosts ip(ingress) is 10.x and the host I am trying to reach is on 192.168.x network(egress) should i be able to do it directly from the host.

Is your ingress running in Docker, and have you tried running directly on the host machine? Might be docker networking getting in the way.

My ingress is running in docker as it's quick install method.

When I do traceroute for ip that exists on the egress gw it takes the default gw on host with netmaker installed and docker containers.

right now just trying to see if that tunnel works.

I confirmed the egress gw address(192.168.5.x, etc) are typed correctly as well.

The host that has netclient on it can reach the 192.168.5.x netowrk and the 192.168.13.x network on the ports shown.

ok, usually I like to test from a few steps to make sure it's working correctly

so first would be to make sure the gateway range is reachable from the ingress machine. So I would exec into docker and make sure you can reach the 192.x egress from there

also need to make sure that the 192.x range is in the AllowedIPs of the Ext Client config that gets generated, so confirm that as well

nothing in allowed ip's just egress gateway ranges

I tried to attach to one of the containers before using the docker container attach id and never got a command prompt. What is the correct way to attach to these containers to test this connection?

ok inside the docker node the route it took was correct it used the eggress gw ip.

so it knows where to go. I just can't ping the host I am trying to reach. Which i can ping from the egress gateway where netclient is installed.

so i think the route is at least correct it must be something else.

https://netmaker.readthedocs.io/en/master/egress-gateway.html

"In the Netmaker UI, that node is set as an “egress gateway.” Range(s) are specified which this node has access to. Once created, all clients (and all new ext clients) in the network will be able to reach those ranges via the gateway."

i have those ranges in the egress gateway is this doc says.

i added the ip of host on eggress side im trying to reach to the allowed ip's as well.

so the ingress node can't ping the host you are trying to reach? if that doesn't work, definitely wont work from ext clients

you're certain the interface is correct? perhaps there are firewall rules blocking on the egress machine

is there way to test the from the egress machine the route the netclient would try and take to reach the hosts?

i did a package capture on the interface that the egress host is on and i see this form the tunnel -

do " ip route get 192.x" where 192.x is the destination machine past the egress

and confirm that this is the interface you set when creating the egress

oh so i have to set it to the vlan id of eth0

i didn't see a way to edit the egress to add that interface so I removed it.

and added it back and it works with eth0.14 set

so that was inside the netmaker docker container when i leave it and try direct on the netmaker host it doesn't know where to go.

it tries to go to the gateway address of the netmaker host and doesn't follow the tunnel.

inside the docker container this is the route.

yes, the netmaker host will be unable to reach the address. On the netmaker server, we contain all the networking inside of the container, so that it does not screw with anything else running on the machine and it will "play nice" with other hosted applications

if the host needs to reach the egress, then a netclient needs to run on it

im confused how you use the ingress point to make it back to the egress point to access services allowed that are on the egress side.

ingress should still work even if it's not reachable from the host

yes that should be correct, are ext clients still unable to reach egress?

I am trying to replicate what cloudflare zero trust allows which i also have tested on the netclient in the picture and I can access home assistant but it doesn't play well with tcp on nvr cause nvr can't have a client installed on it.

ok, but netmaker can now reach the egress range?

so i needed a way to access tcp ports and thought netmaker would work.

ok, one last thing, check to make sure the ext clients contain the egress range (in allowedips in the config) you may need to delete / recreate them. Or easiest thing might be to just delete/recreate the ingress gateway just to be sure

when you say ext clients where is that exactly?

those are the client configs you are generating from the UI, in the "ext clients" section

they are just wireguard config files which you run on your device without the netclient

that is the nodes pane; in the side bar you will see "ext clients"

yes, you need to generate clients to use with the ingress gateway

i generated a client with key that is what i put on the netclient side

I think i misinterpreted what you are trying to do

I don't want to put a client on the mobile phone

you want to reach a service over the public internet using Netmaker?

even with the ingress gateway, you still require wireguard

for what you're trying to do, you need a reverse proxy

i want seemless integration with home services so i don't have to open ports.

Ok, sorry for not understanding, but the ingress gateway will not help here

wouldn't that be what HA PRoxy on the opnsense box in red at top would do.

it sounds like this is what you are trying to do: https://www.youtube.com/watch?v=CGw4Kc424VE

I would follow this tutorial; this is the best you'll be able to do with Netmaker for your scenario

i kind of followed that but I am using HA proxy on the opnsense box.

gotcha, I think I understand the issue then

so HA proxy is on the same host as netmaker?

but its pointing to the host with netmaker cause that is where I thought it would gain access back to the egress point.

so maybe what you want to do is put the ext client on your opnsense firewall then

and have HA Proxy forward traffic over that interface

wonder how well that will play with opnsense.

one of the reasons I did it this way was I wanted to make it easier to control that only my ip's could hit the netmaker dashboard. cause if someone finds a potentially flaw in that code that is an access gw into things.

opnsense allows easy firewall rules to translate dynamic ip hosts for access the netmaker dashboard.

you could just use traefik for this: https://doc.traefik.io/traefik/middlewares/http/ipwhitelist/

Where would traefik go on the netmaker hosts itself?

traefik is already running if you installed netmaker using the installer script

so you can just modify the labels inside of the docker compose

i am somewhat new to docker so forgive me.

no worries, it can be a lot to handle at once, but if you follow that guide, modify the labels inside docker-compose.yml, and then run "docker compose up -d", it should apply the changes

i see the ip whitelist option but does it know how to handle a hostname instead of an ip?

Oh! Sorry I had a brain fart, we use Caddy now instead of Traefik: https://stackoverflow.com/questions/66357765/caddy-v2-ip-whitelist. Should be even easier, just need to modify the Caddyfile.

Yes. I think you want something like this for domains: https://caddy.community/t/implementing-cors-whitelist-in-caddy-v2/8590