A Minor Change to a Data Network Caused Major Headaches for the Voice Network
December 26th, 2017 by admin
After experiencing phone issues for days, one customer learned how quickly TOTLCOM can identify a root cause.
Network outages and phone problems are always a business concern, but when they last for days, the cost to a company’s reputation and bottom line grows exponentially. This was the recent challenge facing our customer, a multi-site association, who experienced intermittent phone outages due to a "routine" hardware change.
See for yourself why TOTLCOM is one of the last truly full-service telephone companies.
While converging services onto a single physical network brings measurable cost savings to the company, deploying VoIP integrates the "Voice Network" with the critical "Data Network" and brings a new level of complexity into the mix. TOTLCOM "Total Network" solutions include an entire suite of telecom, internet, and networking products including life cycle device configuration management.
Companies make huge investments in procuring network infrastructure and employ highly skilled professionals to manage and administer the overall network infrastructure. But typically, few administrators totally understand the added complexities that VoIP adds to their network. This requires specialized knowledge, familiarity with all types of devices from different vendors, awareness of the impact of changes, precision, and accuracy.
At TOTLCOM, highly skilled network administrators carry out all configuration changes.
This customer’s data network vendor innocently made changes to their network equipment without giving consideration for the voice network. When errors in the configuration caused strange behavior in the phone system, they needed to move the configuration back to a properly working version. After troubleshooting for an inordinate amount of time, they called TOTLCOM to fix the problem.
80% of network failures are caused by human change control errors.
Misconfigured network devices can cause problems. For example, devices must be configured with correct IP addresses, subnet masks, and gateway information. If this information is not correct, then communication between voice devices and the data network won’t be possible.
Such was the case with the customer in question. By having a vendor that was strictly maintaining the data network, changes were made to a Gateway device which essentially eliminated communications between VoIP endpoints at a remote office, and the VoIP Server sitting at the headquarters (HQ) office.
As a result, all internal phone calls from the headquarters location to VoIP endpoints at the remote location were going directly to voicemail. But calls in the other direction were routing fine. Due to the length and severity of the error, TOTLCOM engineer Raul Fimbrez was dispatched to the headquarters site to troubleshoot.
Here it gets a little "techie"
Raul performed some simple network tests to ensure the voice network devices were communicating properly and found some clues that eventually led him directly to the root cause issue.
Checking traffic and settings one way he found all devices in the remote office were able to successfully ping all devices locally in the HQ office and was able to eliminate the Remote Office as a source of the error.
Going the other way, he discovered HQ phones could ping local switches and the local VoIP Server. However, as expected, the ping test from an HQ phone to a Remote Office phone failed. He moved on to check out the settings on the VoIP Server and noticed something slightly different.
Just one digit was the problem. Raul asked the IT Manager if they had made any changes to the HQ gateway device recently.
As it turns out, the vendor had recently swapped out an old HQ firewall for a new Dell SonicWALL with a slightly different IP address. By simply updating the DHCP server to assign a new Gateway IP to all DHCP Enabled devices they were able to quickly install the device, test the data network, and walk away.
Why isn't anybody answering their phone?
Because the Servers and Switches that make up the voice network do not use DHCP, they did not receive this critical IP update. Even without access to the Gateway the local server and switch were able to process most inbound and local extension calls but could not reach across the WAN to the remote office. When Raul discovered the problem, he quickly updated the Gateway IP on the Server and switch and was able to restore proper call handling.
This customer has TOTLCOM’s Gold PARTNER Support, so solving the problem was at no cost to the customer, even though the problem was caused by a third party. Under their Gold PARTNER Program there are no NO LABOR CHARGES for all Repairs, Moves, Adds, or Changes to their system. You get a proven vendor to increase efficiency, enhance productivity, help save time, cost and resources and minimize human errors and network downtime.
We suggest calling us for a free consultation before making any network changes.
With TOTLCOM, companies are making the best use of their network infrastructure, achieving increased network uptime and reducing degradation and performance issues.
See for yourself why TOTLCOM is one of the last truly full-service telephone companies.
Posted in: Success Stories
