FYI for anyone else who has a similar use case - I got internal services exposed rather smoothly on digital ocean following this guide: https://stackoverflow.com/questions/54119399/expose-port-80-on-digital-oceans-managed-kubernetes-without-a-load-balancer So I have two ingress controllers in my cluster now - ambassador handles public traffic through the load balancer and this seperate nginx-ingress controller handles the private "employee" traffic over wireguard for things that we want exposed outside the cluster but inside the VPC such as grafana or our self hosted ERP system etc. That way I don't need to front with a load balancer and I can also use ClusterIP on the services I want to expose. In this setup I have the Netmaker egress gateway configured on one k8's node - then all I have to do for DNS is to point the internal custom DNS in Netmaker UI to the IP of the egress gateway node. Then I can create ingresses for each one of my services on port 80/443 and they "just work" - e.g. I can create a dns entry called "grafana.internal" in netmaker UI, reference that in the grafana ingress, and then once i'm connected to the wireguard client just type in "grafana.internal" into my browser and have it take me to grafana

I am guessing servicex needs an update by the way - I tried deploying it as-is to my cluster and it does nothing currently - doesn't even log anything.

@User with the private dns, did you have it turned on in your PORT_FORWARD_SERVICES?

in the docker-compose, the line should look like this: PORT_FORWARD_SERVICES: "mq,dns"

Still using public DNS for this - the resolver runs on public IP and the dns lookups happen publicly. I still wasn't able to get private resolution with the internal IP to work. Above is more referencing how to use ingress to expose services with an egress gateway that's specified in netmaker but I assume it would work with either public or private DNS if private dns is configured properly. so currently PORT_FORWARD_SERVICES: only reads "mq" in my setup though ideally I would like to have it read "mq, dns"

ok, that was what i just wanted to check, if when you tested with private dns, that field was set

we just ran a test and were able to use the private address, so not sure where the issue lies for your setup at the moment

Ok, I can test it again - after we talked yesterday i tried it in both configurations "mq, dns" (with COREDNS_ADDR and coredns ports commented out). that did not work. Then i tried it with public and "mq" with COREDNS_ADDR specified as public IP and UDP/TCP ports specified in docker-compose, which did work. I did not try setting COREDNS_ADDR as the private IP though

to specify, you were unable to get dns working for the external clients, right?

Correct - but I also didn't test with peers. Here's my docker-compose yaml by the way

And here is the version that I attempted with port_forward_services DNS turned on

The top one (using public dns) works, bottom one does not

ok, and i am guessing that after this, you set "external client dns" at the network level to either the public or private IP?

yes, my external client config currently looks like this:

[Interface]
PrivateKey = <key>
Address = 10.11.12.3/32
DNS = <my_public_dns_server_ip>, 8.8.8.8
MTU = 1024

[Peer]
PublicKey = <key>
AllowedIPs = 10.11.12.0/24
Endpoint = <my_public_netmaker_ip (same as dns)>:51822
PersistentKeepalive = 20

And the config that I attempted when I was trying private DNS looked like this (but didn't work):

this looks very close to what I am doing, and more advanced- so I'm super interested. I have the following setup- Netmaker Server is cloud hosted by itself I have a Netmaker network called 'infra_access' this has an IP on 4 nodes- 1 Rancher and 3 nodes of k3s But I cannot work out how to add a netclient IP to a workload on k3s so road warrior clients can access their private Staff Manual on the k3s Wordpress instance? Ideally I'll want to add netclient IPs for networks 'client1' client2' etc. and then have Wordpress (Traefik???) decide who gets which articles... but that's the next problem... I don't currently need DNS, just want access- how do we do this please?

Yeah, that is pretty close to what I have. First, check if you can ping the nodes from your external client. If not, ensure that Netmaker Server is set up as the ingress gateway. From that you should be able to ping the nodes over your netmaker network. Then you need to have the egress gateway inside the cluster. It's another netclient that you install on one of your k8s nodes that's specifically designated as the gateway. It routes traffic inside the cluster from your ext clients. There is an example deployment in the repo. Then you have to label the node and mark it as the gateway using kubectl. Then you also have to mark it as the gateway in Netmaker UI. After all that is set up you need to somehow expose whatever service (Wordpress in your case). You can do it using a Nodeport but the proper way is to keep your wordpress service as a clusterIP and an ingress. Sounds like it could be traefik in your case. I actually have two ingresses in my cluster, one for public traffic and one for private traffic. Public is handled by ambassador and i have a dead simple nginx-ingress for the road warrior traffic. Helps me keep separation of concerns plus I don't need to pay for another loadbalancer. Either way once your ingress is set up to expose routes you can start routing. If you have DNS set up on Netmaker it's super simple - you just create the dns entry for a given route in Netmaker, say wordpress1.internal, and specify wordpress1.internal as the host in your traefik ingress config. The IP for the DNS is the cluster gateway IP (the IP of the cluster node that you labeled with kubectl earlier). Without DNS I suppose you would just use the IP of the gateway plus some sub url route e.g. 10.11.12.2/wordpress1. Not as clean but it would work

Hey @User thanks so much for the detailed instructions- that's the missing piece I needed. I owe you beers!

So I'm in the situation now where I'd like to have pods on my cluster that have netclient installed on the node be able to resolve netmaker internal DNS calls. I can do so perfectly fine from in the netclient pod but I want random pod xyz running on the node to also be able to do so. What's the recommended way to do this? Few thoughts, none of which seem super great 1. Specify my nameserver as an upstream nameserver in the kube-system coredns (https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/). The drawback here is that I still want to fall back on other dns for outside calls and it doesn't look like you can set a policy with this that tells the kube-system coredns that you want to try your own nameserver and then fall back to something like 8.8.8.8 when it fails (it looks like if you specify multiple upstream servers it just picks one at random). I guess I can set up that forwarding on the Netmaker server itself in the coredns config alternatively, which would mean I would specify my nameserver in the kubectl coredns, then when my nameserver (the netmaker server) failed to resolve it would forward the call to some public dns like 8.8.8.8 2. Somehow mount the /etc/hosts config from the node into the pod i'm using. Seems error prone and problematic but maybe I'm wrong. 3. Build the image I need this to work on top of the netclient docker image. Seems like a lot of extra work and bloated container to achieve what i'm trying to achieve. 4. Some other type of automation I haven't considered yet?

It depends on how you set dns up, I'm looking at the docs for Advanced Server Installation > DNS. If your CoreDNS is setup as a clusterIP it will only be reachable internally, and getting to it from the outside via ingress might be a little squirrely. What you really need to do is have coredns running somewhere where it's bound to the node IP + clusterIP. That way it's running in the same place, just with an outside and inside interface.... and I'm honestly not even familiar enough with kubernetes to know whether or not that's possible

what you do (from what I gather) is you run netclient join on the node first with --dns off --daemon off and once the wg interfaces are up on the node, you use the deployment to get your pod to take over control of the netclient/wg instance. If you look at the deployment https://raw.githubusercontent.com/gravitl/netmaker/develop/kube/netclient-template.yaml you can see the volume mounts on how it does it. That way the pod controls the netclient on the node and you can just point to the dns running on the netmaker server since the wg interfaces are natively routed. It's like a limited version of docker network mode=host.

Ok, so that makes a bit more sense. My setup I already have the DNS server running outside the cluster - so having to worry about node and cluster IP is not necessary since it's actually running on a public IP right now (ideally will be private later). What I'm reading from @white-piano-73111 is that if I take my deployment, let's say it's a deployment called "wordpress" for the sake of simplicity, and I want the pods for "wordpress" to be able to resolve my own DNS, what I need to do is have that netclient daemonset running already on the cluster nodes, then modify my wordpress deployment to also mount those volumes so that it does whatever magic with resolvectl to make the dns available?

Almost. Your "wordpress" doesn't need any modification at all. Once netclient is running on your node(or all nodes via daemonset in cluster) and the wg interface to the netmaker server is up on every node, you can change your k8s cluster coredns to forward "."

kubectl -n kube-system edit cm coredns

to the netmaker server (and a backup server if you like). Then everything in your cluster will automatically use it. Wordpress and whatever else you have running. You can chain it then to. Let's say your k8s cluster(coredns) is configured to use netmaker as a forwarder for "." then you can set your netmaker server to forward to pihole for example. You don't need to worry about resolvectl in this case at all.

Ah, I had actually looked at having cluster CoreDNS forward queries - I had wanted to do it like you suggest (point 1 of my original post), where k8s CoreDNS would first try to resolve netmaker server's DNS (let's say it's 10.11.12.254) and then if that fails to resolve the k8's coredns would attempt go out to somewhere like 8.8.8.8. I don't think you can actually do that though, CoreDNS lets you specify multiple "upstreamNameservers" but you can't tell it that you want to resolve and fallback like that (https://kubernetes.io/docs/tasks/administer-cluster/dns-custom-nameservers/). If you give it two nameservers it just picks one at random (the default for forward plugin https://coredns.io/plugins/forward/). I guess I could actually modify the corefile running on kube-dns and set that forward default to "sequential" to get the behavior I want, but I use managed DOKS and i'm not sure if DO will come back and overwrite that with some cron job. Your suggestion of sending everything through netmaker first would also work though - basically all upstream DNS going outbound from k8s coredns is routed through netmaker server first, and then that netmaker server in its own conf passes it on to someplace like 8.8.8.8 if it can't resolve it. The downside obviously is that I have to send all of my external dns lookups through an extra hop

Another downside off the top of my head of sending Everything through netmaker is if the netmaker node died I would have no outbound dns resolution whatsoever, even on external stuff. So I think maybe modifying the Corefile to set sequential is actually the more robust way to do it if it's possible in my case

Actually it looks like it can be done on DOKS in a pretty clean fashion - see this guide here: https://docs.digitalocean.com/products/kubernetes/how-to/customize-coredns/ In this case we would override forward plugin like this:

yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns-custom
  namespace: kube-system
data:
  log.override: |
    log
  custom.server: |
    mynetwork.internal {
      forward . 10.11.12.254
    }

It appears to work very cleanly - only requests to the domain myurl.internal are forwarded, everything else just happens as it did before. I suspect I might be able to put *.internal in there and have everything to the .internal tld go through that nameserver. I'm going to try this approach

I have not used doks before but cool that they let you mess around in there. I too like a clean and simple solution that fails predictably 😆