I was having the same issues with Welt the other day and I've done the same as you. For the most part, I have a decent connection with people but sometimes it's unbearable for the other person or for myself.
Just horrible playing online in general though, I feel for all you dudes with no local scene.
If I get the chance, I'll get a copy of the PC version and see if it fares better.