DNS Change Writeup:
In v0.11.0 many users noted that DNS was not working. Resolvectl seemed to be causeing major issues or not working at all on many systems. Resolvectl was our preferred way of setting DNS. If you look deep into how DNS is intended to be managed on Linux, resolvectl is correct, and can set nameservers for specific interfaces, allowing us to do proper "split" DNS which does not interfere with regular networking. To fix our DNS issues, we have gone with a much more brute force approach. (Thanks again to @User).
If you've worked with Linux, you know the easiest way to configure DNS is with /etc/hosts. Pretty much any entry you put in /etc/hosts will work automatically. This is not a recommended practice because /etc/hosts is not a source of truth (it is generated by other processes) and it is not persistent/will get overwritten. However, if you have a daemon running that sets it regularly, it's not as big of a deal.
Our working approach is to just set /etc/hosts with the private DNS entries. This is extremely reliable and works across many systems (Linux, Mac, FreeBSD, even Windows!). It is also very easy to implement. We don't even need a nameserver anymore. We just send the entries to the client and it plugs them into /etc/hosts. The docker-compose still contains CoreDNS, but it will largely sit idle and could be turned off.
Long term, we would like to implement a more "official" system for managing DNS that relies on the nameserver. But the short story is, DNS is very very hard, and very complicated on most systems. There's no clean approach, and nothing that is cross-compatible, so this will take time. In the meantime, if you have an issue with setting private DNS using /etc/hosts, you may want to turn it off.