That happened to me once while I was soloing Kaiden Prisoner, but the next couple times, I just asked them not to touch the mob in /say and was lucky enough to have them leave me alone.
If you can sneak into the actual area following a group, you might be able to kite inside when there aren't as many bystanders. It's not as safe, but it's doable.
You could also try Eracus Temple--the mobs there are a couple levels higher and there isn't as much room to move around though.
XI Wiki




