PDA

View Full Version : Quadro 2xi & Thomson ST2030



acsinove
08-26-2008, 09:49 AM
Hello.

I have a site using a Quadro 2xi and 12 Thomson ST2030 phones, plus one Aastra 57i phone.
Since day one, I have had issues with communications quality. Like, users report choppy sound and even from time to time barely understandable speech.
This has been happening both on incoming and outgoing calls, through either WAN interfaces (ISDN or SIP trunk, two different SIP providers), both on the ST2030s and the 57i. This has not been seen on internal calls, though.

1/ any ideas ?

2/ I have read somewhere that there's a guide for configuring ST2030s for use with Epygi PBX, but I can't seem to locate the document.

Thanks for your help,

R.

davrays
08-27-2008, 10:09 AM
Do you mean the installation have the same problems with both ISDN calls (done through onboard ISDN connection), and SIP calls to ITSP provider? Can you confirm that?
If so, the only suspected part should be the 2xi placement in the network (probably bad switches or something this kind), as if you have the same choppy audio when talking to ISDN, this means the packets are lost between phones and Quadro.
Did you look at the network statistics of those bad calls (especially to the ones going to ISDN)? Is there a packet loss or big jitter/delay?

Regarding the document on configuring the ST2030: there is no special document for those phones, but some guidance can be found in the documents called "Configuring Epygi Tested IP Phones" and "Quadro Features on Epygi's Tested IP Phones list". Just search the download area using the word "tested" and you'll find those documents.

Best regards,
David

acsinove
09-01-2008, 04:19 AM
Hello, and thanks for your answer.
Regarding your first question, yes we have the same issue with both ISDN and SIP calls, both inbound and outbound.
But not with entirely internal calls.
For you reply I guess I should look into the network. The fact that network connections are shared between the phones and the computers might be a problem. Are there some settings on the phones or on the Epygi to prioritize the voice traffic over data packets?

Thanks for your help,

R.

davrays
09-01-2008, 10:22 AM
The voice traffic is prioritised on the Quadro by default. But that relates only to the traffic, which has reached the Quadro. If the packets are lost in the LAN before reaching the Quadro (for example, because of glitchy switches, which drop packets when congested), Quadro can do nothing to help there. If the quality problems never appear during internal calls (when voice stream are passing directly from phone to phone, bypassing the Quadro), it means some network equipment (for example, a switch), which is located between Quadro and phones, could be the source of the problem (it may pass too big traffic, and so may be congested).

Best regards,
David

acsinove
09-07-2008, 08:54 AM
Hi again. Here's the rtp log for an occurrence of a bad quality call:

Journal RTP

qualité: 1 (excellent)

Local: 192.168.253.6:6076, Remote: 192.168.253.67:41000, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 7398
Taile des paquets reçus: 160
Paquets RX perdus: 0
RX Jitter: 15
Retard maximum RX: 21 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 6123
Taille des paquets transmis: 240

Local: 192.168.253.6:6078, Remote: 212.27.52.129:36014, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 6123
Taile des paquets reçus: 240
Paquets RX perdus: 0
RX Jitter: 15
Retard maximum RX: 54 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 7398
Taille des paquets transmis: 160
Configure Call Quality Event Notification

Does that tell you anything about what the problem might be? FYI, I have isolated the phone network so that all phones and the Epygi are connected directly to a 10/100 switch (entry level Netgear).

Thanks for your help,

R.

davrays
09-08-2008, 12:52 PM
Looking at the parameters of this call, you can see that is was really excellent quality, at least from Quadro perspective - nothing is lost in neither leg of the call, small jitter etc.
From the statistics I see only two strange things:
1. The "Local" IP addresses are the same in both legs of the call. That can mean only one thing: your phones are placed in the Quadro WAN side, and, but calls are proxied through the Quadro. Is that the case? If yes, why do you proxy streams through Quadro, but don't send them directly to ITSP?
2. The media is assymmetric. Phone sends one 160byte packet every 20msec, while the ITSP (212.27.52.129) sends one 240byte packet every 30msec. This ITSP behaviour is not standard and it may confuse the phone, if the RTP stack of the phone is not so good.

Some questions/things to test:
1. Do your users report problems only on their end? I mean does the bad audio happen only on their phones, or the remote users (at ITSP side) also hear choppy sound?
2. Do those problems appear on Aastra too, or it is happening with Thomson phones only?
3. Do other, good calls also have such asymmetric media (RX packet size = 160, TX packet size = 240), or the media is symmetric there. You can see that from statistics of good calls.
4. Do you have something placed behind the Quadro, which may cause a load on the unit, or do you use it as a router at all?
5. Can you force the media to go directly between phones and ITSP, without RTP proxy? Probably you just need to uncheck RTP proxy on routing rule and see if you still can make calls, and have normal audio. You can do that and see if choppy sound proble still persists on ITSP calls.

acsinove
09-10-2008, 11:28 AM
The phones indeed are connected to the WAN side of the Quadro ; as per the documentation, this is the proper setup when the Quadro is not used as a router (nothing is connected to the LAN side of the Quadro). The reason why calls would be proxyed through the Quadro is not clear : I have checked the routing rules, and the proxy box is not checked. The phones are not setup to proxy either. If have tried removing the proxy address from the profile setup on a ST2030, but then I couldn't place any outgoing call at all.
Answers:
1- I have had reports of bad sound on both sides. These include choppy sound but also occasional echo.
2- the Aastra has these problems as well
3- that's not going to help us : I have seen symetric calls with bad sound, as well as asymetric calls with good quality sound...
4- no, it's not used as a router. I can't imagine anything putting load on the Quadro, but is there any way to check? Does it have an SNMP agent (and a MIB!) that I could use to monitor it?
5- that's an interesting track I'd like to follow. I have verified the routing rule, and the "Use RTP proxy" box is unchecked. Could there be some specific setting on the ST2030 side as well?

For the record, here's another bad call record:

Journal RTP

qualité: 1 (excellent)

Local: 192.168.253.6:6094, Remote: 192.168.253.66:41000, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 2160
Taile des paquets reçus: 160
Paquets RX perdus: 0
RX Jitter: 15
Retard maximum RX: 12 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 2641
Taille des paquets transmis: 160

Local: 192.168.253.6:6096, Remote: 193.202.111.239:19362, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 2641
Taile des paquets reçus: 160
Paquets RX perdus: 0
RX Jitter: 2
Retard maximum RX: 3 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 2160
Taille des paquets transmis: 160

In this call, the local user had trouble understanding what the remote user was saying. He didn't ask whether it was the same the other way around.

For comparison's sake, here's a good call record:

Journal RTP

qualité: 1 (excellent)

Local: 192.168.253.6:6084, Remote: 192.168.253.68:41000, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 9749
Taile des paquets reçus: 160
Paquets RX perdus: 0
RX Jitter: 15
Retard maximum RX: 30 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 8903
Taille des paquets transmis: 240

Local: 192.168.253.6:6086, Remote: 212.27.52.129:35356, qualité: 1 (excellent)
RX Codec: PCMA
Paquets reçus: 8903
Taile des paquets reçus: 240
Paquets RX perdus: 1 (0,01%)
RX Jitter: 15
Retard maximum RX: 333 ms
Comptage de hausse du délai de réception 0
Comptage de baisse du délai de réception : 0

TX Codec: PCMA
Paquets transmis: 9749
Taille des paquets transmis: 160

Questions:
- would it help to enable the developer's log? could you make anything of it?
- on the phone side, I can elect to use either udp or tcp. One would think tcp would be more stable, what do you think?

davrays
09-10-2008, 12:43 PM
I can clearly see that RTP is proxied through the Quadro (oterwise you would not have any RTP statistics on Quadro). If you are absolutely sure RTP proxy is not set on the ITSP rules, there is only one option: you configured those phones as remote extensions and set the RTP Proxy on the "Remote Settings". If this is not the case, you would probably need to enable developer logs and send them to me (after making one-two calls), as this is pretty strange.

But I am afraid if you remove RTP proxy, you will have problerms calling out of your network. I am not sure your riuter will handle the voice streams correctly, if they go between phone and ITSP directly.

I s it possible to look at the statistics of the bad (with choppy sound call), going to ISDN?

acsinove
09-15-2008, 11:31 AM
I haven't pinned down any bad sounding call through the ISDN line in the last couple days.

I double-checked the proxy issue, and I can confirm the RTP proxy is not set on the ITSP rule, and the phones are not set up as remote extensions.
FWIW, the router is a Funkwerk Bintec that has a VoIP configuration option to enable SIP proxying and prioritizing.

davrays
09-15-2008, 02:23 PM
In this case I don't really understand why the streams are passing through Quadro.
If you are not afraid of me hovering about your Quadro, you can give me an access to your device (my external IP is 91.198.247.202), so I can understand what is going on there.

acsinove
09-15-2008, 03:16 PM
I'm not affraid, actually I'd be glad if you did... here's the URL :
http://espace-temps.dnsalias.org:8001/
(only your IP have access)

davrays
09-16-2008, 11:16 AM
I looked on your board and now I understand why the streams are proxied through Quadro - this is the "automatic" RTP proxy mode introduced lately: if the target and the destination are in the different subnets, Quadro opens RTP proxy automaticatty, regardless of the RTP Proxy setting on Routing rule or anywhere else. This is done to avoid problems with missing audio, and actually this works in your case (as you will not hear the audio, if there is no RTP proxy).

Apart of that, I am still not sure what can cause the problem on your device.
I can see there only two problems:
a) you could have accidental outages one or two times per day for several minutes up to one hour (we are working now to find the problem);
b) you have one call in the call stats with terrible call statistics (with length of 22 min and with about 30% of lost packets; also the codec is shown wrong on that call). Typically such stats mean that there are lost RTP streams in your network. Not sure which device is sending them.

Is it possible to have an RTP capture for such bad calls? As soon as you hear the conversation is going bad, you can open the "netcapture.cgi", and start the capture on the WAN interface. This could show what streams are passing by at that moment, and also can show some parameters of RTP stream. It seems that the RTP statistics is not enough to resolve this. Also (or alternatively) if you give an access to the port 645 on your Quadro, I can run remote logging, to see if something strange was happening on the device on the moment when bad call occured.

acsinove
09-16-2008, 02:54 PM
Thanks for your going that deep into this.
I have opened the port 645 (TCP) for you, that is for your public IP.
I will try to get a capture, though it might be tricky to have the customer call me when it happens and keep the call going at the same time... We'll see. I don't know what kind of logging you get through port 645, but if there's a long enough history it might be easier to use since the users usually report problems to me after they happen.

davrays
09-17-2008, 11:03 AM
I have started remote monitoring from the port 645. Lets try to catch the problem together.

Related to the outages you have on that device (I can clearly see tey are happening, though you didn't complained about them): this have started in September, probably something is changed in the confid at that time. The possible reason of that problem is those two settings:
1. One or more of the phones are using TCP
2. The SIP timers are set to "High Avalilability"

Is it possible to fix one of this factors to see if the second one still causes the issue? For example, change the settings on the phone to use UDP (this should be the default), or set the timers to "RFC3261"?

acsinove
09-17-2008, 04:37 PM
The specific outages you are referring to are probably failed outgoing calls to ITSP. They began appearing when I set the phones to use TCP, so I have now reverted back to UDP.
I will change the timer settings to RFC3261.
Can you keep the monitoring going until I get a timestamp of the problem from the user?
Thanks again,
R.

davrays
09-18-2008, 11:19 AM
Yep, the monitoring is still ON and we can keep it running for about one-two weeks, if necessary. Anyway, please try to get netcapture ("all packets" on WAN), even if the call is closed, as even if you get that in 20-30 minutes, it could containg the lost strem, if any).
BTW, I can see that some of your phones still use TCP. Please don't change this, so we can see if the "High Availability" timers were causing the problem (now you have RFC3261 timers set).

davrays
09-18-2008, 01:28 PM
One more suggestion: if it is hard or absolutely impossible to make a network capture of the bad call (this is the only thing actually, which will surely give an idea on what is happening), at least it worth to make a VM recording of such call.
This can be done the following way (you can tell users how to do that, so they can do thatt themselves):
1. Create a special extension for recording, with at least 1% of user space attached. Or use an existing extension with VMbox. Lets assume that is extension 11.
1. Create a routing rule, like 55555, NDS=5, Prefix=11, with a PBX-Voicemail type.
2. When problematic call occurs, user can hold, call to 55555, connect to VMbox of extension 11, then press Conference button. This will establish a conference call between those two parties and the VMbox of extension 11. So the conversation will be recorded to the VM box.
3. We can look later to this mail to: a) find that call in the statistics based on the caller/time info; ; b) listen to see which type of distortion is there.

acsinove
09-19-2008, 05:22 AM
Ok, I have had two bad calls:
Outgoing calls from ext 38 to 015369xx55 and 024067xx37, on the 18th of septembre, I don't have the exact time but it was before 1:08PM. Choppy sound on the inside end, maybe other end as well. For some reason outgoing call don't appear in the call log...?!

davrays
09-19-2008, 08:53 AM
I found both calls in the Call Stats: one of them is about 26 minutes long (started at 11:32:28) and marked as "Very bad", another is 2 min long (started at 12:04:16) and maked as "Bad".
It looks there are two (at least) completely different problems there:

1. The first call is not really "Very Bad" as it marked in the call stats. If it were so bad, as it shown, it woulnt last 26 minutes :) What really happened at that call is the following: you have "SIP session timer" enabled. After 15 minutes, the Quadro initialed session refresh, as a result of which (due to some specifics of Thomson phone operation), the connecteion was closed and reopened again with G.729 codec (the original codec was PCMU). So user may experience some click in the middle of conversation (at about 15 minutes), and probably the voice quality became worse after that (as it switched to from PCMA to G.729). But the quality was not very bad, as the person continued to talk for another 11 minutes.
To avoid that kind of problems you should either switch OFF the "Session Timer" at the "SIP Settings" page (this will help for sure), or change the codec preference on the Thomson phones to exactly match the prference list on the Quadro: it should have the same order (this should help, though I have no 100% guarantee, as this is up to Thomson phone behaviour).

2. The second bad call happened purely due to the newtork connection dropdown at the end of that call. If you look at the call stats, you will see another call on 12:07:49 (from ext.36 to the same ITSP), which had the "satisfactory" call rating, which means the call was also not good. It looks like either your ISP had some short period of bad connection in general, or the connection to the "freephonie" ITSP gone bad at that moment.
So this looks like a WAN network connect short outage, which is restored itself after some time.
Or the ITSP has network connection problems :)

acsinove
09-19-2008, 09:13 AM
HELP !
Right now users can't place outgoing calls nor receive incoming calls (not even call internal phones!)

davrays
09-19-2008, 09:44 AM
That is one of the outages which I was telling you about. System goes down for something about 5min-1hour (depending on the IP Phone registration time). You have such outages about 2-3 times a day. Just this time the situation happened at a wrong moment.
Please change all Thomson phones to use UDP for SIP transport, until we reproduce in our testlab and fix this problem in a future release. This will keep your system running stable.

Thanks and sorry for inconvenience..

acsinove
09-21-2008, 04:04 PM
Ok I have made some cleanup in the codecs so that all the calls should now use g729a. I have also double-checked that all phones are set to use UDP. Let's see how things turn out.
Are you still monitoring the system? Is the monitoring introducing a stability issue?

davrays
09-22-2008, 08:24 AM
I am still monitoring the unit.
The monitoring itself cannot be not the reason of stability issues (you had outages before I started that). And since you switched to UDP on all your phones, there was no outage anymore (the last one was at Sep 21 16:35, so I suppose you changed the setting on the phones last Sunday) and since that everything works OK.

acsinove
09-22-2008, 12:51 PM
Ok I have had some user feedback today (september 22nd):
- there seems to be a short delay when the users answer incoming calls, like if she says "Espace-Temps bonjour" after picking up the phone, the caller hears only "...jour".
- at 16h29, from ext 38 to 016904xx45, choppy sound from the start ;
- at 16h34, from ext 37 to 014355xx98, choppy sound?
- at 18h06, from ext 38 to 014461xx03, choppy sound.
On thing that bugs me: most calls use the PCMA codec, while I'm confident I set g729 as the preffered codec on both the Epygi and the phones.

KSComs
09-22-2008, 01:31 PM
The gap with communications is normal when you are using an ITSP. It is st the minute something that VoIP has to live with. You dont have the same on physical lines due to the immediate communication ie send and receive channels and the d channel for supervisory. You just dont get that type of signalling yet with sip.

I may be wrong but that is how I understand the technology used.

Regards

Kevin

acsinove
09-22-2008, 04:27 PM
The thing is, this delay is new, like it is associated with the monitoring feature.
About the PCMA codec: this is the codec used by the ITSP, so that's normal behaviour.

acsinove
09-23-2008, 03:47 PM
Davrays, do you have any hint what might have caused the choppy sound on the calls mentioned up there?
I have had another choppy-sound issue today the 23rd from ext 38 to 022820xx16. Interesting thing is that I have changed the ITSP so I now use g729a and symmetric packet size ; though the quality appears as excellent in the log, the user is still noticing choppy sound. Any clue?

davrays
09-23-2008, 04:55 PM
What refers to delay, i might be caused by enabled developer logs plus monitoring. If you want, we can switch that OFF and test.
Concerning the choppy sound - truly saying I am lost here :confused:
Those calls are reported as excelent, and the RTP statistics shows that they were really excellent - practically no jitter, no loss at all. How this call can have choppy sound..? Thinking further, we return to the same assumption as in the beginning: if the RTP stream has reached the Quadro from ITSP, and was sent to the phone without problems, those problems should be either in between of the Quadro and phone, or in the phone itself. Is that possible to replace ext 38 with normal analog phone for some time, and see if problems disappear? Or at least put the Aastra on ext 38. Also it could be very useful to put Snom phone instead (for experiment). Snom has a good habit: it sends the RTP statistics in the last SIP message of the call (either BYE or OK). This way we can see if the phone received all the packets.

acsinove
10-10-2008, 05:01 AM
Hello.
Could we (well, you ;-)) switch monitoring and developer logs OFF ?
At the moment I don't have a SNOM phone available. I guess I'll leave it as is for the time being. The customer has plans to have its cabling rebuilt, which might have an impact on the issue. We'll see...

davrays
10-14-2008, 11:00 AM
Sure :)
I switched OFF the developer logs, and disabled monitoring.
Lets wait a little until either cabling change will solve the problem, or you get a Snom :)
Hopefully, the delay you had, will disappear after switching OFF the monitoring.
Best regards,
David

MEDIAPCSYSTEM
05-24-2009, 06:47 PM
Any news how does it finish? I've got the same problem. with 11 st2030s. I change the codec latency in 30ms using g729 codec and disable firmware control.

davrays
05-27-2009, 05:08 PM
Well, it didn't finish :). Was just aborted, due to absence of Snom phone and time (as you can see reading the whole long story)... :D

This problem (bad voice quality) is very generic and extremely dependant on the network conditions at the specific site, so you cannot be sure this is the same problem just because there are the same phones used.

It doesn't make to continue this story here. If you are ready to debug the problem (ready to spend some time and make experiments), please open a new thread and describe your specific problem.

Best regards,
David