Introduction 

The purpose of this article is to explain how an Ignition gateway handles telecommunication using VOIP technologies. This article's focus is not to provide you step-by-step instructions on troubleshooting but instead explain the concepts and call flow to make troubleshooting an easier process. In order to discuss this, there is needed context that exists outside of our normal subject matter. 

Ignition Gateway requirements needs:

• Text-To-Speech module
• Voice notification module
• Alarm notification module

Additional hardware:

• SIP solution OR
• VOIP to analog converter such as a Grandstream that requires a PTSN line

What is a SIP Server?

A SIP server is another name for a VOIP (voice-over-IP) server or call manager. (ref.1) SIP stands for Session Initiation Protocol and is not a voice specific protocol, it is also used for other multimedia implementations like video calls. For our purpose we only care how SIP operates with Voice. In the simplest form, SIP is the protocol that negotiates between Ignition and the Call Manager to set up and establish a call. This provides the Call Manager registration information, phone numbers, availability, and negotiates the voice codec used for transmission. By default this happens on port 5060.

From a IP/UPD standpoint, SIP is just a unique set of Headers wrapped around media information that instructs the VOIP server how to handle the call. It is a standardized structure. (ref.4) Note that audio does not go across SIP. Before discussing call flow in more detail, let's discuss the other kind of telecommunication, POTS/PTSN, and a generic term: PBX.

RTP:

For Ignitions purpose, RTP(real-time transport protocol) is the protocol that carries the information the end user receives or sends. This includes the actual Voice and DTMF(Dual tone multi-frequency a.k.a key presses) for Ignition.(ref.3)

What is POTS/PTSN?

The first acronym, POTS, stands for "plain old telephone service" and is interchangeable with the term PTSN, which is Public Switched Telephone Network. For Ignition, the only thing you need to know is that this is analog communication that uses TCP/IP. This phone system is circuit based, like switch board operators. 

What is PBX?

A private branch exchange isn't actually a single resource. It is an internal telephone network that handles inbound/outbound calls. This network can contain a VOIP phone system and be hosted in the cloud or it could locally sourced and use an analog telephone system(POTS/PTSN). (ref.2)

Call Flow

Ignition acts as a handset to the existing PBX telecommunications implementation. With that in mind we will talk about two different scenarios depicted in the image below. The third scenario depicted is how a standard call operates outside of Ignition.

Scenario 1:

In this scenario, Ignition is configured with a voice notification profile pointing at a VOIP server, also called a SIP server. This server could be in the cloud or local. The user could refer to this as a cloud PBX. As long as the TCP/IP we are communicating with is the VOIP server handling the call.

1. Ignition uses UDP on port 5060 to send hello's to the SIP server and to potentially register(most scenario's we don't actually register). This should be bidirectional communication.
2. When Ignition initiates a call it uses the SIP protocol to talk to the VOIP server. During this communication the VOIP server sees ignition as the handset defined by the username of the notification profile. During this time we pass the previously mentioned information and the VOIP server either gives us a response or dials out.
3. Once the call goes out, Ignition switches to the RTP(real-time protocol) to send audio and DTMF signals(dual tone frequency, also known as key presses). At this time the VOIP server is acting as a proxy so Ignition is sending the audio through the VOIP server to the end users phone.  

Scenario 2:

In this scenario, Ignition is talking to a VOIP to Analog Converter. The most common examples are Grandstream devices that tie into an analog line provided by a public carrier or local PBX.

1. Ignition sends ALL IP traffic to the converter in the same order. First UDP then SIP then RTP. The Grandstream documentation(ref.6) recommends the converter to be configured to expect the traffic on the LAN interface.
2. When SIP initiates a call the converter takes the session information and passes it across the FXO, provided there is a dial tone for the analog phone system to dial out. Any response we get is coming from the Converter as we do use analog communications.

FXO is the foreign exchange office, a physical port on the converter that is responsible for connecting to an existing PTSN line. The FXO is the outgoing PTSN line, and the FXS (foreign exchange subscriber) is the incoming line.

Troubleshooting:

This section will touch on how to approach voice issues and what tools are available. This section will also list some common issues and error responses.

Common Issues:

Faulted Voice Module:

If either TTS or voice notification fault, often the only solution is to restart the Ignition service. The reason for this lies in how the modules tie into the platform during the service startup sequence, the specific order of operations and references that are made. Sometimes you can fix the issue by restarting the module. Keep in mind that no healthy system should result in the voice modules faulting.

Call has no audio:

At this point in the article you should make the association that based on the call flow, the issue with no audio is primarily an issue with the lack of RTP traffic.

Unable to acknowledge Voice

This lies entirely with how DTMF is handled. For more information about this issue, check this preexisting KBA article: https://support.inductiveautomation.com/hc/en-us/articles/4413040434317-Troubleshooting-Voice-Notification-Module-issues-involving-unrecognized-keypresses-and-DTMF-tones 

Tools:

There are two tools at your disposal for troubleshooting voice loggers and Wireshark.

Loggers:

• alarm.Notification.Voice.CallManager
• alarm.Notification.Voice.CallManager.Agent
• alarm.Notification.Voice.CallManager.PhoneCall

The loggers listed above will walk you through the call flow previously discussed and show the send/recv IPs used by your configurations. If the call is successful, you will see this response:

INFO | jvm 1 | 2020/03/23 14:19:42 | T [a.N.V.C.Agent ] [20:19:42]: Read 960 bytes from 
sound manager, putting it in rawData buffer 960 bytes at a time.

Wireshark:

Wireshark is the best tool to use to troubleshoot voice. Most of the conversation and information needed for a successful call out happens at the IP level. SIP and RTP are both recognized protocols in Wireshark and are readable. In addition, if you open a Wireshark capture and select Telephony>VOIP calls it will automatically put together the packets. It will even allow you to play back the audio, knowing this capture is what Ignition sent out.

Common Error responses:

Most of the errors parallel http responses, but the responses do originate from the VOIP Server so the server response is what you will see in the Wireshark and our logs. 
Disclaimer:

The voice notification profile will ALWAYS display "Registered with VOIP Host" unless we are actively told otherwise. Keep in mind all this is based in UDP so unless something tells us otherwise we don't report any errors and will display 'registered'.

(ref.5)

Redundancy

In this article I've avoided specific configurations, since the intent is education rather then a step by step guide. However, there are two outlying scenarios that come up with Voice and redundancy described below. These are 2 edge case scenarios with users trying to implement redundant voice profiles. In the first scenario, the servers need different bind addresses to get to one SIP Server. The second scenario is you are trying to give master and backup different sip servers or need to use separate extensions. These setups don't come up often through tickets but IT departments typically implement redundancy wherever they can.

Ignition Redundancy with one voice notification profile

This may look straight forward but there is a trick involved in this configuration. Notification profiles are mirrored between the master and backup gateways so they can't differ. In this case, you have to define a local bind address on both Ignition servers or just one for connectivity to the VOIP Server so you have to modify that gateways 'ignition.conf' file by adding the following line to the additional parameters and restarting your gateway.

wrapper.java.additional.8=-Dignition.voice.localbindaddr=XXX.XXX.XXX.XXX

Redundant Pair talk to different SIP Servers

In this scenario there are two VOIP servers and you want your redundant pair to communicate respectively. The need for this arises when there isn't network communication for one Ignition gateway to communicate to both VOIP servers or you don't want them to. In this case, since the master and backup gateways have to have the same configuration you make two voice notification profiles so that each one can have their own unique username/extension. 

• Voice 1 profile will be valid on the master gateway and has connectivity to Voice 1 (VOIP server)
  • Voice 2 profile will be disconnected on the master due to connectivity
• On the Backup gateway, Voice 2 profile is valid and Voice 1 profile is disconnected.

This same setup can be used to talk to a single SIP server such as CUCM, if there is a need to use 2 phone numbers: one for master and one for backup. 

References:

1. https://www.networkworld.com/article/2332980/lan-wan-what-is-sip.html 

2. https://www.nextiva.com/blog/what-is-pbx.html 

3. https://www.3cx.com/pbx/rtp/ 

4. https://ieeexplore.ieee.org/document/1189367 

5. https://datatracker.ietf.org/doc/html/rfc2616#section-10.4.4 

6. https://support.inductiveautomation.com/hc/en-us/articles/360047175492-How-to-configure-the-Grandstream-HT-503-and-HT-813-for-use-with-Ignition-Alarming-voice- 

Active Bugs to be aware of:

1. IGN-1868, memory leak due to CereRenderer class
2. IGN-5273, memory leak due to the transaction Manager
3. IGN-4580,circles around the CNonce with the intent to get Twilio to work with us.(FB14893)