Voice Notification Profile (VNP) Configuration and Registration

     Network is Unreachable: Datagram Send Failed

     Dealing with Multiple NIC Addresses

     Reviewing Network Traffic using Wireshark

Test Call Was Not Received By End User

     alarm.Notifiation.Voice.CallManager

     SIP Error Codes

Issues during a Received Call by the End User

     Using Telephony to Analyze VoIP Calls

     Media Debug Enabled

     Potential DTMF tone failure

     Poor TTS Quality

Disclaimer: This troubleshooting guide does not cover all potential scenarios. This includes both the information and diagrams. Use this as a tool but understand that it's not guaranteed to provide you a solution. 

This article will provide how to approach common issues related to the Voice Notification Module and its communication with VoIP Servers. It’s highly recommended to first familiarize yourself with basic VOIP terminology and concepts before reading this guide.

If you are attempting to understand the reason behind any thrown error during your test call’s execution, keep in mind that you are looking for where the test call’s execution process specifically fails based on the following steps:

1. A Voice Notification Profile (VNP) is created in Ignition
2. Ignition’s VNP will attempt to register to the VOIP Server
3. The VoIP Server will register with Ignition if configuration is correct.
   1. Note: Keep in mind that an Ignition VNP status on the Gateway Configuration page may state ‘Registered’ for invalid IP addresses. Confirm registration with your VoIP server directly. 
4. Test call executed by user in Ignition
5. Ignition sends out a Session Initiation Protocol (SIP) request carried by UDP into the network. 
   1. Here we are requesting the VoIP server to call users from a provided list of numbers.
6. The VoIP server is able to receive the SIP request by listening to its open UDP port.
7. The VoIP server notifies Ignition via SIP over UDP that it is going to dial out to users in the provided list.
8. Only once a call is picked up will it be initiated, which means Ignition will send a Real Time Protocol (RTP) message over UDP to the VoIP server. RTP is expected to be sent only after a call is initiated.
   1. This is where (Text-to-Speech) TTS scripts are passed with the appropriate audio conversions.
9. The VoIP server receives the RTP message and passes the data to the recipient of the call.
10. If acknowledgement is configured on the VNP, a caller will send DTMF tones back to the VoIP server.
    1. The VoIP server will then make a list of acknowledgements received by callers and send them back to Ignition.

Based on these steps, ignition loggers may or may not provide insight during failure of one or more of these steps. Since this process typically deals with UDP to transmit data, a Wireshark capture will provide the lowest level of detail for possible response or network issues. However it's important to keep in mind all possible sanity checks that can be observed during this entire execution process.

Voice Notification Profile (VNP) Configuration and Registration

Regardless of any details in your VoIP troubleshooting, it’s important to first confirm that initial registration between your Ignition and VoIP server. Refer to the following flowchart to ensure that this is the case:

1. Start - VoIP Doesn’t Work
2. Check - Is the VNP configured and Voice Module valid?
   1. Yes - Go to Step 3. 
   2. No - Go to Step 4.
3. Check - Did the VNP register successfully?
   1. Yes - End: Registration is successful. Begin a Test Call.
   2. No - Go to Step 5.
4. Action - Configure VNP and install the correct Voice Module, then: - Go back to Step 3.
5. Check - Are you experiencing a Network is Unreachable: Datagram Send Failed ?
   1. Yes - Go to Step 6.
   2. No - End: Review for Ignition errors and notify development for suggestions.
6. Check - Does the Gateway contain other available NICs?
   1. Yes - Go to Step 7.
   2. No - Go to Step 9.
7. Check - Is the Local Bind Address of the VNP configured to the correct NIC?
   1. Yes - Go to Step 9.
   2. No - Go to Step 8.
8. Action - Set the Local Bind Address to a different NIC, then: Go back to Step 3.
9. Action - Confirm if Server Endpoints are configured to be reached, then: Go to Step 10.
10. Action - Take a Wireshark Capture on Gateway Server, then: Go to Step 11.
11. End: Review settings and Wireshark capture.

Network is Unreachable: Datagram Send Failed

One of the most common errors in the Ignition Diagnostic logs, Network Unreachable: Datagram Send Failed, is a result of a misconfigured Local Bind address. The Local Bind address determines the network interface used for establishing communication between Ignition and your VoIP server. In order for a network interface to exist between the Ignition and your VoIP server, your overall network infrastructure must be configured to have an open UDP route between the VoIP server and at least one Network Interface Card (NIC) built into the Ignition server’s host machine. 

Dealing with Multiple NIC Addresses

On your Ignition Gateway’s Status Overview Page, the right hand side may present one or more Available NIC addresses that are installed on the host machine of your Ignition server. If more than one exists, you will need to find out if one of those addresses is capable of reaching your VoIP server since the VNP will choose one by default if it’s not specified in the Local Bind address. 

If for some reason, network information is not available, you may be able to get away with testing all available NICs or seeing what route you end up taking using the tool tracert (assuming ICMP is enabled for your network). These suggestions may work as a quick way to check if the notification profile is able to register with what is currently available in your Ignition server.

Confirm if Server Endpoints are Configured to be Reached

Depending on the scale of your network, you may need to review the infrastructure itself or consult a network administrator if you’re unsure about the reachability of the available NICs present in your host machine. If you have access, check all device and service configuration settings from both the Ignition and VoIP server endpoints. Ensure that the IP address and port of both servers can be reached both to and from the Gateway server endpoint by calling ping or telnet. 

Reviewing Network Traffic using Wireshark

The easiest thing to review from a wireshark capture is whether Inbound and Outbound traffic exists at all between your two endpoints.

• Outbound traffic is expected to be sent from the Ignition Gateway IP address.
• Inbound traffic is expected to be sent from the VOIP Server IP address.

The example above is a Wireshark capture of a successful test call where a single user was notified, the user answered, and acknowledged the call using a DTMF tone response before the call was ended. The capture filter for this example was specified as:

host {Public IP of Ignition Host Machine} and host {Public Hostname of VOIP Server}

 When observing this capture, take note of the following:

• Lines 1 - 6 shows the initial INVITE request being sent by Ignition to a VOIP server (Outbound) and the VOIP server’s acknowledgement of that request (Inbound).
• Line 7 is an outbound message where Ignition is providing information about the user who will be notified (i.e. the user’s phone number).
• Line 8 is a typical outbound UDP message for the purpose of periodically “pinging” the VOIP server, unrelated to the test call execution process.
• Line 9 is an inbound message where the VOIP server is notifying Ignition that the call was ended.

Confirming the packets mentioned above will help you resolve any potential confusion on  network interface or infrastructure misconfiguration. In the scenario of all monitored network interfaces containing little to no Inbound responses from your VOIP server, bring this information to your IT, network or VOIP server specialist to find out what else may be missing from your setup. Any issues with test calls after this confirmation will require a more in-depth look at your Wireshark capture.

Test Call Was Not Received By End User

Assuming your VNP was successfully registered to your VOIP server, you are ready to execute a test call. If your first test call is failing to reach end users, refer to following chart:

1. Start - Test Call Not Received
2. Action - Set alarm.Notification.Voice.CallManager loggers to TRACE, then: Go to Step 3.
3. Check - Do the Gateway loggers provide a clear reason for the failed test call?
   1. Yes - End: Resolve Against Error Produced. 
   2. No - Go to Step 4.
4. Action - Take a Wireshark capture on the Gateway Server during another test call, then, Go to Step 5.
5. Check - Do you see both inbound and outbound traffic between the Gateway and VoIP server?
   1. Yes - Go to Step 6.
   2. No - Go to Step 7.
6. Action - Analyze the incoming packets from the VoIP server and cross-reference with your Gateway logs, then: Go to Step 6a.
   1. End: Locate any SIP Error codes coming from the VoIP server.
7. Check - Did you specify the correct NIC for your Wireshark capture?
   1. Yes - Go to Step 8.
   2. No - Go to Step 9.
8. Action - Confirm if Server Endpoints are configured to be reached, then: Go to Step 8a.
   1. End: Review settings and Wireshark capture.
9. Action - Consider setting up a new Wireshark capture that monitors ALL NICs in order to locate all possible SIP traffic, then: Go back to Step 4. 

alarm.Notifiation.Voice.CallManager

In the event of an unsuccessful test call for any reason, check the Ignition logs for alarm.Notifiation.Voice.CallManager messages. Messages from this Ignition logger provide the most details of requests and responses between Ignition and the VoIP Server that are expected during a phone call event as well as the most descriptive errors of this subsystem. Set the following loggers to TRACE before making a test call if you require the lowest level of details that Ignition can provide during its execution:

• alarm.Notifiation.Voice.CallManager
• alarm.Notifiation.Voice.CallManager.Agent
• alarm.Notifiation.Voice.CallManager.PhoneCall

SIP Error Codes

Assuming your VNP is properly registered and a test call was attempted, any misconfigured properties from the notification profile are expected to result in SIP error messages that can be found in either Ignition Diagnostics logs or Wireshark captures. Please refer to this article for more information about response codes and their meaning.

Issues during a Received Call by the End User

Wireshark captures are required to assist you in narrowing down the details of an issue occurring during a call received by the end user. When inspecting a capture, keep in mind the three protocols mentioned earlier. They will contain information relevant to VOIP:

• UDP: Needed to confirm if transport layer messaging is successful.
• SIP: Needed for confirming initial VoIP registration and server requests.
• RTP: Needed for inspecting quality and transport of audio data and notification acknowledgement.

At this point in your troubleshooting, VNP should already be registered successfully and test calls should be making their way to end-users. If so, you will want to focus on any issues related to Text-To-Speech (TTS) or Real Time Protocol (RTP). This is because they are the fundamental pieces during phone call messages and end-user acknowledgements. Refer to the following chart if you are running into such issues:

1. Start - Test Call Not Behaving Properly?
2. Check - Do you hear the TTS speaking when the call is picked up?
   1. Yes - Go to Step 3.
   2. No - Go to Step 4.
3. Are Dial Tone acknowledgements configured and being recognized?
   1. Yes - End: Non-Issue.
   2. No - End: Potential DTMF Tone Failure. See KBA on DTMF Troubleshooting
4. Check - Is a Voice module installed and not in a faulted state?
   1. Yes - Go to Step 5.
   2. No - Go to Step 6.
5. Action - Take a Wireshark capture on the Gateway Server during another test call, then: Go to Step 7.
6. Action - Reinstall or fix the Voice module. The Gateway may also need to be restarted, then: Go back to Step 2.
7. Check - Do you see any RTP traffic being established between VoIP and Gateway servers?
   1. Yes - Go to Step 8.
   2. No - Go to Step 9.
8. Action - Listen to the RTP stream by utilizing the Teleophony tool in Wireshark, then: Go to Step 10.
9. Check - Did enabling media debug generate good quality audio files in the Temp directory?
   1. Yes -
   2. No - 
10. Check - Is there any jitter in the RTP stream of the packets? Is the TTS audio stuttering or skipping words?
    1. Yes - End: Investigate network health between VoIP to Gateway. Ensure both server hosts are healthy performance-wise.
    2. No - Go to Step 11.
11. Check - Are the RTP streams matching what was heard in the phone call?
    1. Yes - End: Non-issue.
    2. No - Go to Step 12.
12. Check - Does the user have an auto-attendant or any type of active call flow logic?
    1. Yes - End: Consider bypassing any auto-attendant call logic or implemented PIN confirmation requirements.
    2. No - Go to Step 13.
13. Check - Does the user have Single Directional Audio only?
    1. Yes - Go to Step
    2. No - End: Non-issue.
14. Action - If files were generated, the a Successful RTP exchanges occurred. Ensure you are inspected the correct NIC, then: Go to Step 16.
15. Action - Ensure Temp folder file permissions allow write access, then: Go to Step 16.
16. Action - Refer to section Considerations for Missing RTP Traffic, then: Go to Step 17.
17. Action - Consider setting up a new Wireshark capture that monitors ALL NICs in order to locate all possible SIP traffic, then: Go back to Step 5.

Using Telephony to Analyze VoIP Calls

With a Wireshark capture monitoring a failed notification execution, you can begin investigating the time-stamped packets that occurred during various steps of the test call procedure. Wireshark has a useful feature for further analyzing the audio data by selecting Telephony > VoIP Calls. By selecting this tool, you have the option to playback the RTP stream, which is a recording of the audio message that is executed during the user’s phone call notification. This will help confirm if details such as TTS scripts and dial-tone acknowledgements are being sent through your network.

If you are planning to take a Wireshark capture with the intent of listening to an RTP stream, ensure that no capture filters are present before starting. This will ensure that any SIP, RTP, and UDP packets using one or more potential IP endpoints are visible for inspection. However, this will obviously make the entire capture harder to read if the VOIP related packets are not in focus. Specifying a Packet Filter for SIP or RTP packets only should help.

Media Debug Enabled

If you’re unsure if RTP is making its way between endpoints during a test call, setting the VNP settings Media Debug Enabled to true may help as one final confirmation of either VOIP or Ignition configurations preventing a successful call.

Media debug audio files are expected to appear in the Ignition directory after a successful RTP response from the VoIP server. In terms of Ignition’s backend processes, MediaManager has an incomingRtpReader object which creates a file to hold the Media Debug recording. When incomingRtpReader starts, it should create the media file. This occurs once the MediaManager receives a successful response. If no media files are generated, you are more than likely failing to establish RTP communication. This can be verified with a Wireshark capture.

Potential DTMF tone failure

If you run into the scenario of potential DTMF tone failure on the flowchart for this section, refer to the KBA on DTMF Troubleshooting for more information. The next section also includes other providers that may also help in isolating where the DTMF failure is occurring.

Poor Call Quality (Jitter)

Call jitter refers to inconsistencies in the amount of time it takes for voice data packets to reach their receiver. It is often confused with latency, but they are not the same. Latency is the overall delay before a transfer of data begins, while jitter refers to the variation in packet arrival times. A few known causes in packet jitter would be the following:

• The performance of a Gateway/host machine itself can sometimes impact the delivery of packets.
• The network itself between the caller and the recipient. 

In either case, to isolate whatever node in the network topology is causing the problem, utilizing other providers or recipient alternatives may provide further insights:

• Calcentric is a free VOIP provider you can use (at the time of writing this article). This service is free when used with their own phone numbers.

• Microsip is an open-source portable SIP softphone you can use to determine if the recipient's phone network is causing the issue. This software allows you to directly connect to any local or cloud SIP server.

While most jitter scenarios are network-related, there are times when the performance of the gateway or the host itself can impact packet delivery time, which will increase jitter. So it's important to make sure the host and gateway are overall healthy from a performance perspective.

Considerations for Missing RTP Traffic

There are a multitude of scenarios that can cause missing RTP traffic in a Wireshark capture. Below are a few things to consider:

• Ensure that you have correct bind addresses configured.
• Ensure that proper RTP ports are configured.
• Remove/absolve any middle-men in your network infrastructure.

Note: This does not cover all potential scenarios.

After adjustments are made, perform another test call and capture once more in Wireshark.