vlan stopped handing out IP addresses
WSM-12.9.3 / FSM-12.8.2
Firebox M370
After losing power to our facility during a storm, 8+ hours, the next day we had 6 IP phones not registering with our onsite server. As it turned out those 6 phones were not receiving an IP address from my Phone VLAN. My Firebox is handling the dhcp of my vlan.
My Phone VLAN has an address pool of .100-.199, with approx. 65 phones in use. All the phones have a PC attached and all Pc's had internet access while the phones weren't working, so my network was working.
What I found was the VLAN was handing out the last of the IP's, in the upper range, .150 through .199. The IP's below .150 were random and not all being used.
As a test I added to my address pool, I extended to .210. That's when 5 of my 6 phones received an IP address, .200 and above. Currently .200-205, 207,208 are being used.
I cleared my Arp cache and rebooted my firebox but that didn't seem to help.
My firebox reboots every morning @ 4am and my phone server(3cx) reprovisions every night.
Question: what would cause the phones or vlan not to use the lower IP's in the pool first, then act as if I ran out of IP's?
Thannks,
brad
Comments
No idea - quite odd.
A reboot of the firewall will clear the DHCP list and the ARP cache.
You can turn on Diagnostic logging on DHCP Server. Perhaps that will show something to help understand this.
Consider opening a support case on this.
Just curious, why is FSM at a lower version than WSM, since FSM is part of WSM
Thanks Bruce,
I opened a case with WG, I'll update when we have an answer.
I also bumped up my DHCP diagnostic logs to med.
Probably with the idea I was going to be upgrading, higher version of WSM when upgrading FSM and now I'm way behind.
Next on my list, update my Firebox!!
Hi @bford
There's a few things that could be causing the scope to start at the back end like that.
-Some clients will use DHCPINFORM to get a preferred IP address. If you've ever noticed a workstation sticking to a specific IP address via DHCP even if there isn't a DHCP reservation for it, it's probably this.
-There may be a lot of DHCPNAK activity, or other random MAC addressed that have fallen off the ARP table, but still have DHCP reservations. (By default DHCP leases are for 8 hours, even if they're only used for a few seconds.)
-There might be an issue with the DHCP server on the firewall.
If you're seeing any logs related to the DHCP server on that network that might suggest a problem, please copy them into the case -- it'll help the technician that's assigned to the case identify the issue.
-James Carson
WatchGuard Customer Support
@Bruce_Briggs and @james.carson
I did open a case with Watchguard; #02078563, which has been closed.
Worked mostly with Ryan in trying different things, he had me looking over my network for potential problems. I could not find anything new or any changes that might have occurred due to the power outage.
We didn't come up with any specific cause to our problem or any specific fix.
After several days of poking around and not finding anything specific, along with the phones working OK I decided to try to revert my changes I made to get the 6+ phone to work.
I removed my extended DHCP IP pool .100 - .210 back down to my original pool of .100 - .199.
I then rebooted each phone that had a .20x IP. Fortunately, and partially to my surprise they all got a lower IP address.
Part of my frustration or lack of knowledge, is they still grabbed the higher end IP addresses while leaving multiple lower IP's unused.
This is the order in which I rebooted the phones.
Phone 1- ip .208 went to .196 (ip's .151 - .195, .197-199 utilized)
Phone 2- ip .207 went to .138
Phone 3- ip .204 went to .139 (.137-.150, open)
Phone 4- ip .202 went to .140
Phone 5- ip .203 went to .141
Phone 6- ip .200 went to .142
Phone 7- ip .201 went to .143
Phone 8- ip .205 went to .144
IP's 103, 107-109, 111, 112, 114, 117-136, .145-.150 open
IP's .151 -.199 used
This comment from Ryan did make me wonder if he thoroughly understood my problem.
"The location of the IP that is handed out from within the scope set for the DHCP server does not indicate anything about the state of the DHCP server. IPs will be shuffled around all the time and the issue only comes into play if we are exhausting addresses without having more clients than there are addresses. If anything, I suggest reducing the DHCP scope to a value closer to the expected host count."
If I understand your configuration, your IP phone is acting as a switch for your PC.
Are the phones and the PC's on the same subnet, or are the phones on a different VLAN than the PC's?
If it's the second option I would look at the IP phones themselves and not the firewall.
Having your phones re-provision on a 3CX system every night isn't necessary as sometimes the phones don't take the provisioning and revert to default.
This comes from over a dozen years of experience running an on prem 3CX system myself.
Between the firewall reboot every night (why???) and the 3CX reboot every night (again, why???) it doesn't surprise me you are having this issue.
If a service on the 3CX needs restarted every night just create a script on the server to restart that service when you want. I have one running twice daily for a very specific purpose.
I think it's a 3CX and phone issue, not a firewall issue.
JMHO.
It's usually something simple.
Hi @bford
I took a look at the case.
In this case, it looks like your firewall is handing out IP addresses as expected, and you have quite a few more devices (specifically MAC addresses) making reservations.
Many devices use 'privacy' MAC addresses nowadays (this means a random MAC address to make tracking specific devices more difficult.) If you're seeing random MAC addresses filling up your DHCP table, I'd suggest looking into that.
Since you're using other equipment to bridge networks, there may be a component of this issue that is being caused by those devices as well.
If you'd like to explore this further, you can reply to your support case and let the technician know that your're still experiencing the issue. If there's a specific issue you think is being caused by the firewall, I'd suggest pointing that out.
-James Carson
WatchGuard Customer Support
Instead of ARP addrs list, make sure that you check the DHCP addrs list.
If there are unused entries in the DHCP addrs list, and devices are not getting IP addrs from the firewall DHCP server process, then that would be a concern.
While I mostly live in the WSM/FSM world, I find that the Web UI DHCP list is more user friendly. (System Status -> DHCP Leases)
As stated by support, it is not unexpected for a DHCP server to not only give out IP addrs from the bottom of a DHCP available list. They can be given out from higher range, and then go to a lower range later. It is up to the code in the DHCP server as to how it chooses to hand out available DHCP IP addrs.
@shaazaminator
I want to thank you your insight and comments about 3cx phones.
To answer some of your questions:
Our phones and Pc's are not on the same subnet.
Pc's are on a separate trusted network VLAN1-10.xxx.xxx.xxx; Phones are on Optional network VLAN50-xxx.xxx.50.100 - .199
Yes, the network cable runs to the phone and then the Pc connects to the phone.
During our phone down time all the Pc worked just fine.
We have a 3rd party that manages our on Prem 3cx system. I do not have admin rights to the operating side of the system. I don't believe the system reboots every night but unless I misunderstood, I was told it reprovisions every night. We have only had this 3cx for two years and I'm still learning about it.
One question that seems to have come up; why I reboot every night?
Should I not be rebooting every night? Should I reboot on a different schedule or reboot at all?
Somewhere along the line someone recommended I schedule a nightly reboot. Sorry, I don't remember at this time who or where that came from.
Thanks,
Brad
I don't reboot my firewall on a schedule.
It only gets rebooted when I do a firmware upgrade or have an infrequent power blip.
Some do a weekly scheduled reboot.