feature: Support use of FQDN resolution with DNS and mDNS

[invario]

Resolves: #1753
Submitting as a DRAFT since it is still being worked on:

• Replaces the IP field in the Device Form with IP/FQDN. Updated the regex to validate as FQDN or valid IP address. Max. input length 255 characters.

• Added ResolveToIPAddr function that takes one string parameter - either FQDN or IP.
  • On Linux/FreeBSD mDNS lookup is implemented with dbus and go-avahi and if it times out, a call to the net.DefaultResolver is made.
  • On Windows/Darwin, a simple call to net.DefaultResolver will perform both a DNS and mNDS lookup.
  • In both cases, if the function is called with an actual IP, it simply returns the IP address.
• Wrapped all references to device IP to use ResolveToIPAddr and added valid actions if the IP string is ""
• wakeUDP no longer sends 6 packets for WOL.
  • This is because if a host is powered down, its .local name cannot be resolved via mDNS, and as a result there is no IP address. Without an IP address of the device, we cannot send a unicast (routed) packet, and we also cannot determine the broadcast address. Broadcasting to 255.255.255.255 does not work (reliably) because the OS will send the packet to either the default network interface, or the one with the lowest metric (behavior varies depending on the OS.)
  • As a result, when a device is powered down and the user has configured a .local device, wakeUDP will send (2) WOL packets (ports 7 and 9) to every interface (except loopback, non-broadcast capable, and inactive/down).
  • Refactored some of the existing code
  • Existing users that have IP addresses for their devices and proper netmasks should not have to change a thing

To Do/Need Feedback:

• I'm strongly considering adding an optional field on the Device Form for Interface that would allow specifying the local/host interface for broadcasting the WOL packet.
• mDNS lookups currently timeout after 5 seconds. That is less than the default of 3 seconds for UpSnap between pings. I need to resolve this in one of two ways
  • Reduce the timeout to 2 or 3 seconds. On most networks/setup, this shouldn't be a problem. If a device is up, it should respond within milliseconds. I am currently favoring this option
  • Increase the UpSnap ping cron.
• Update documentation
• Need testing and input!

[invario]

Need to add:

• If it is unable to connect to dbus/avahi when it encounters a .local FQDN, it should log a warning error and continue on to try to resolve it using a DNS server. Currently, it just errors out if dbus/avahi is not accessible. RFC 6762 says .local should be delegated to mDNS only and should not be passed onto a unicast DNS server. Google says that there may be some edge legacy cases of enterprise Windows networks that resolve .local via unicast DNS. I'm going to leave the logic in place: fail if a .local FQDN is provided and dbus/avahi are not useable.
• sleep.go needs to have GetString("ip") wrapped Actually, this looks like it was already done, my mistake.

[seriousm4x]

Thanks a lot for this pr. I really appreciate all the work you have done in the last days/weeks!

---

Some general issues i found accross this pr is the formatting for go files. All indentation is done with 2 spaces. You can use go's build in formattor called gofmt like so:

cd backend/
go fmt ./...

Which will format all files automatically. Depending on your editor you can enable format on save which will most likely already use gofmt. Before doing other changes to this pr, please make a commit to clean up the formatting.

Also using gopls in your editor gives you some warnings about code that will work but has minor issues.

---

I've left comments on all the parts where I'd make changes. I have not yet tested the implementation (just quickly changed the IP from my pi to pi.local which worked on my mac). Better testing the code across platforms still needs to be done.

Regarding the feedback you requested:

I'm strongly considering adding an optional field on the Device Form for Interface that would allow specifying the local/host interface for broadcasting the WOL packet.

How do we handle changes in interfaces? When we store nics in the database and those nics become unavailable (new host or other network card) it will cause issues.

mDNS lookups currently timeout after 5 seconds. That is less than the default of 3 seconds for UpSnap between pings. I need to resolve this in one of two ways

I've tested this a couple days back and resolution failed on timeouts below 100ms on my local network. I think setting it to 1-2 seconds should be plenty.

Again, thank you for your work :)

[invario]

I will take care of all the formatting and try gopls, thank you! It will take me a few days to go over it all but once I do, I'll reply and/or make the changes. Agreed on needing better testing on the other platforms. I have a MacOS VM and FreeBSD VM fired up inside my Windows PC and a separate Linux and LInux+Docker setup to test each new compile but I'm just one person testing lol.

How do we handle changes in interfaces? When we store nics in the database and those nics become unavailable (new host or other network card) it will cause issues.

We can identify the interface by the IP assigned to the host/server itself, so the optional input field on the form would take an IP. mdlayher/wol doesn't' allow specifying an interface, however, so this field would only be used in the case of a .local FQDN when the machine is powered down and the .local cannot resolve.

Or we can just leave it at broadcasting on all interfaces, but the OCD in me doesn't like the idea of sending so many packets out at once unnecessarily. BUT from what I can tell from Googling, there is 0 harm in keeping it this way. Packets are very small, and the occurrences of this (when a WOL is triggered) is rare.

I've tested this a couple days back and resolution failed on timeouts below 100ms on my local network. I think setting it to 1-2 seconds should be plenty.

Great! I'll set the context timeout to 2 then.

Again, thank you for your work :)

❤️

[invario]

Need to add:

• If it is unable to connect to dbus/avahi when it encounters a .local FQDN, it should log a warning error and continue on to try to resolve it using a DNS server. Currently, it just errors out if dbus/avahi is not accessible.

RFC 6762 says .local should be delegated to mDNS only and should not be passed onto a unicast DNS server. Google says that there may be some edge legacy cases of enterprise Windows networks that resolve .local via unicast DNS. I'm going to leave the logic in place: fail if a .local FQDN is provided and dbus/avahi are not useable.

[invario]

@seriousm4x Earlier in the other thread we spoke about mDNS queries flooding the network but I realized I made an oversight in my calculations. With 13 devices that are powered on, responses are going to be cached for the TTL (80% of 120 secs) which mitigates the spam factor a bit.

However, if all the devices are powered off, there is nothing to cache and UpSnap will send out 260 mDNS queries per minute, which is a pretty high number. Any ideas on how to better handle when devices are off?

[seriousm4x]

However, if all the devices are powered off, there is nothing to cache and UpSnap will send out 260 mDNS queries per minute, which is a pretty high number. Any ideas on how to better handle when devices are off?

We could save the last known ip address and ping that. When a device is offline, we could only do an mdns request every x ping. Ip leases don't change that often so it's quite likely that once the device comes back up it will get the same ip. So only quering mdns every x ping would reduce the amount of queries.

[invario]

However, if all the devices are powered off, there is nothing to cache and UpSnap will send out 260 mDNS queries per minute, which is a pretty high number. Any ideas on how to better handle when devices are off?

We could save the last known ip address and ping that. When a device is offline, we could only do an mdns request every x ping. Ip leases don't change that often so it's quite likely that once the device comes back up it will get the same ip. So only quering mdns every x ping would reduce the amount of queries.

In my experience, that varies a lot, depending on the network configuration and how many devices are joining/leaving the network. The longer the device is powered off, the greater the likelihood its IP will be handed out by the DHCP server to some other random device that needs an IP. In a busy network, it can happen very quickly. Example: I manage a family member's network in his coffee shop, and that is high traffic, high turnover for DHCP, so IPs get reassigned very, very quickly.

But even if I were to save the previous IP and continue using that when it goes offline, if/when the IP were to get reassigned to another device, then we're just stuck in the same boat again with flooding queries trying to get the new IP.

I'm going to keep thinking about a way to fix this. I've made the other changes discussed above, but am going to hold off on pushing the update.

[invario]

Still being worked on. Noticed some sort of memory leak and I'm not sure why it's happening. Checking to see if it's related to any changes from this PR. The test instance of Docker UpSnap starts with 1 device configured and using about 40-60MB but over the course of several days of idling it has ballooned to 928MB. My test device has no wake or shutdown crons configured either.

[invario]

I've rebased to the latest master and I believe I've resolved everything. Will push a new commit in a day or two!

[invario]

Latest push does the following:

• fixed cronjobs.go call to port.Set()
• refactored lots of the new code and fixed errors
• added validation functions into validate.go to handle unicode FQDN and punycode
• fixed memory leak from not closing avahi server connection by establishing permanent connection
• updated regex used in on device form and PB validation pattern to accept international unicode FQDN or IPv4 addresses
• updated regex used for netmask validation on form
• timeout mDNS queries after 2 seconds

I haven't done anything new to mitigate the potential network flooding problem that would occur if someone has a lot of devices all offline and using .local FQDN.

[seriousm4x]

Thanks! I will have a look in the next days when I have time 👍🏼

[invario]

Something I completely overlooked: I'm going to have to go through the newly added code from the IP tracking feature since the ip field takes fqdn/IP. Basically disable IP tracking if a FQDN is used.

edit: Completed on the frontend and pushed. But need to add a check somewhere in the backend in case a FQDN slips into the field somehow.

edit2: Good news, I tested it by forcing a FQDN into the field and IP tracking logs an error (as it should.)

[seriousm4x]

So, I've tested this PR on my local Mac. Seems to work so far. I just added 3 small fixes:

• The port scanning would not save if the deviceIP lookup failed. Also changed the behaviour of deviceIP so that it does not resolve the deviceIP for every port.
• send wol packets to 255.255.255.255 as well
• if host had uppercase .LOCAL it would not match (not sure if that will ever be the case though...)

Will merge and push a beta so I can test it on Windows and Linux as well.

Thank you :)

[invario]

Excellent! I look forward to hearing back from you (and others) with good or bad results! In the meantime, I'm going to draw up some notes to add to the readme or Wiki indicating that a static IP address is really the way to go and that FQDN resolution via mDNS should be used sparingly. FQDN using actual DNS resolution shouldn't have any problems at all.
Thanks for the merge and fixes!

[invario]

Just a heads up: I had removed 255.255.255.255 for a reason, but it's been so long I can't remember why anymore apart from my note above:

Broadcasting to 255.255.255.255 does not work (reliably) because the OS will send the packet to either the default network interface, or the one with the lowest metric (behavior varies depending on the OS.)

I don't remember if it results in an error and that's why I took it out.

[seriousm4x]

Just a heads up: I had removed 255.255.255.255 for a reason, but it's been so long I can't remember why anymore apart from my note above

I see. Removed it again. I thought is was gone by accident and couldn't remember any reason not to send.

So far I've tested the beta on Mac/arm64, Windows/amd64 and Linux/amd64 and found no issues. Hostnames resolved everywhere without issues. I think I will push a release in the next hours.