If by "That's what I want to do." you mean separate FFXI (Speakers) and Vent (Headset), i'll explain it the best i can. It's been a long time since i fixed this in Vista, over a year now, so i'm a bit hazy on how i got it to work in XP.
Before i go any further, i'll let you know what i use on my system. Since i've only ever run it on my system and never really tried other configurations, your mileage may vary.
Originally i had it set up with:
OS: Windows XP
Soundcard: Creative Soundblaster Live! 5.1 Gamer
Onboard Sound: SigmaTel HD Audio
Speakers: Logitech 5.1's (With independent power/volume)
Currently it's set up with:
OS: Windows Vista
Soundcard: Creative Soundblaster X-Fi
Onboard Sound: SigmaTel HD Audio
Speakers: Logitech 5.1's (With independent power/volume)
In Windows XP:
Separating the sound output in XP works because of a glitch with how the OS handles multiple sound devices. Typically when you install a soundcard you disable the onboard sound. This glitch works by not disabling the onboard sound so you essentially have two sound devices installed. Officially XP won't support this, but as long as you have the SigmaTel codec's installed, it's there for use if a device can recognize it. Vent is one of those programs that will recognize it. In XP, with DirectSound enabled, you can speak to/hear Vent only though a headset while the game sound will come out crystal clear over the speakers plugged into the card. DirectSound allows Vent bypass the card when speaking so the game sound and Vent never get mixed.
In this configuration, you enable DirectSound with Vent (in Vents sound set-up options), and only Vent. Windows (and by extension, the rest of the system) recognizes your soundcard for everything (like FFXI), Vent recognizes your onboard sound for everything. The two never mix. With this configuration, unfortunatly, overall volume is controlled by Windows. This means if you want low System sound but high-volume Vent, you need to adjust System volume first and then Vent sound through Vent (inbound/outbound). Depending on who makes your soundcard you might be able to use a 3rd party volume control. Alternatively, you can also just power-off your speakers without affecting your Vent headset (or anything else plugged directly into your onboard sound).
Recap: This config.
Overall system sound is output on the soundcard and controlled by Windows. Powered speakers are plugged into the soundcard.
Headset sound is controlled by Ventrillo through the onboard SigmaTel audio. The headset is plugged into the mobo.
Volume is handled by Windows and the speakers. There is no true independent control. You can turn the speakers down/off via the actual volume knob on the speakers and Vent remains, of course, unaffected.
In Windows Vista:
Vista is much, much different. Microsoft disabled HAL in Vista, which in-turn totally removed traditional DirectSound.
So, first off, DirectSound has been essentially eliminated (runs in an emulation mode) which means you can't bypass the audio stack anymore with a headset. This is resolved by Vista's emulation modes though, and by the fact that it officially recognizes multiple sound devices and can utilize all of them at once. Vista can also control the volume for each application independently, regardless of which sound source they're using. This configuration takes more work to set up, but ultimately gives you much more control because Vista allows you to separately configure each sound device however you want. This means you can dedicate one jack to headphones, another to speakers, so on, so forth.
In this configuration i have my speakers plugged into my soundcard. All Windows sounds play over my speakers. By default FFXI and my media players play over my speakers too. Obviously that would be a mess if they were both going at the same time. My headset (Vent) is plugged in to my mobo on the Rear Channel of my onboard sound (SigmaTel), my headphones are plugged into my tower's front I/O plate on the Front Channel of my onboard sound (SigmaTel) or the headphone jack on my speakers (Vista allows independent control).
Recap: This config.
Overall system sound is output on the soundcard and controlled by Windows. Powered speakers are plugged into the soundcard.
Headset sound is controlled by Windows using the onboard SigmaTel audio. Vent is configured to use only SigmaTel HD Audio on the Rear Channel.
Headphone sound is controlled by Windows using the onboard SigmaTel audio or the soundcard depending on where you plug them in (i usually use the jack on my Speakers). When headphones are plugged in, all system sound is diverted to the headphones. Using Vista's independent app control, only what you want to hear comes out of the headphones (usually just music).
It took me a lot of trial and error to get this working how i wanted, especially after i moved to Vista. Now that i have it i wouldn't play any other way. I usually use one of four different configurations.
1. Full game sound through my speakers and Vent on my headset.
2. Game music muted, but game sound effects over my speakers, iTunes music over my speakers, Vent on my headset.
3. No sound over my speakers, all game music/sounds through my headphones.
4. No sound over my speakers, only game sounds through my headphones, iTunes through my headphones.
Once they're all set up, you never really have to change anything. By default i have Windows sound over my speakers and Vent over my headset. When i want to change that around all i have to do is move some volume sliders. Nothing else has to really be fiddled with. With Vista you can output as many sound sources to as many devices as you have plugged in and control all their volume independently. It's really pretty awesome.
XI Wiki

