Testing out fast songs, but I could use some help

[rog]

That pops up within 2 seconds. If you can get away with that on your computer, you're also capable of forcing R0 to ghetto pos hack you way through normally dangerous areas.

Put your modem on standby.

[Yinnyth]

Let's compare to something easier to grasp: You have a stopwatch, a racetrack of adjustable distance, and a runner who always runs the same speed (impossible but just pretend). You are not allowed to look at the runner or the racetrack, you cannot hear the runner's footsteps, but you can hear his voice. Our goal is to find out how long it takes the runner to go from start to finish. We do this by telling the runner "start running." Some random amount of time after we tell him to start, he starts running. Some random amount of time after he begins running, he tells us "I'm currently running." Finally, some random time after he crosses the finish line, the runner tells us "I'm done running."

Me, I time this by starting the watch when I tell the runner to start, and stopping when he tells me he's done.

You instead time this by starting your watch when he tells you he's currently running, and stopping when he tells you he's done.

Now let's say all delays range from 1-5 seconds. He can wait 1-5 seconds before running after we tell him to run. He can wait 1-5 seconds after he started running before telling us he's running. He can wait 1-5 seconds after he's finished before telling us he's done. Let's also assume both of us and the runner have a 1 second "processing time" (it takes us 1 second to go from hearing to understanding+taking necesary action).

For my timing: I start timer, tell runner to run (+1 second processing time). Runner decides when to run (+1-5 seconds random lag). Runner begins running, waits a random 1-5 seconds, then tells me he's running which I disregard because it has no impact on my timing. Runner crosses the finish (+1-5 seconds random lag) tells me he's done, I stop watch (+1 second processing time). My time is anywhere from 4 to 12 seconds longer than the actual time.

For your timing: you tell runner to run, he takes 1 second to understand, then takes a random 1-5 seconds to begin running which has no impact on your time because you haven't started your watch yet. He starts running but doesn't tell you yet. He waits a random amount of time, then tells you (-1-5 seconds random lag because you're starting your watch AFTER he already started running). Furthermore, it takes you an extra second to react to him telling you he's running so you start your watch even later (-1 second processing time). He crosses the finish (+1-5 seconds random lag) tells you he's done, you stop your watch (+1 second processing time). Your time ranges anywhere from 4 seconds short to 4 seconds longer than the actual time.

At first glance, your timing method seems more accurate, yes. However, our margin of error is exactly the same, and we can shorten the track the runner uses by known percentages. That means we get to calculate our margin of error. The fact that my initial measurements are further from the true measurement than yours means nothing after we plug these numbers into equations to determine the true measurement.

And don't think for a second that your method avoids these calculations. In the example I gave, every latency was equal. In truth, it's possible that the runner takes more or less time to process information than we do. Or it's possible he takes longer 2-12 seconds to start running, but only 1-2 seconds to tell us he's started running, but we would never know that. This randomness is why a consistant method with consistant results is more important than the points we choose to measure from. I could measure from exactly 30 frames after I give the command, but that doesn't mean I'm any closer to the actual time unless I account for the possibility of error.

Chances are good your initial recordings are closer to the actual casting time, BUT
1: that's not a guarantee you're closer, it's merely a statistical probability you are
2: equations will account for my margin of error
3: I would have to start over from scratch to change to your method
4: I would still have to use the equations on your method to ensure accuracy

So what is the advantage I gain from switching?

[Yinnyth]

Put your modem on standby.

Modem is up one flight of stairs, through the kitchen, up another flight of stairs, down a hallway, through a cubby hole, through the attic, through a door, in the network closet. It's possible, but impractical, and would probably annoy my family to have me switching off the internet whenever I damn well please.

[rog]

Well you only need to do it once.

[Yinnyth]

Alright, alright- going over all my tests, my margin of error seems way too large to be able to accurately test items like loq. earring. Even though I really don't want to, I'll start retesting tomorrow using Asmoranomar's technique (modified so I count at party box cursor appearance, not overhead cursor appearance). With any luck, it might help narrow my results down a bit~

[Asmoranomar]

Let's compare to something easier to grasp:

Lets not. This is a simple concept and doesn't require any unrealistic irrelevant examples to aid us. Furthermore, these things usually serve to confuse rather than help. I'll be making my replies toward the actual situation.

Me, I time this by starting the watch when I tell the runner to start, and stopping when he tells me he's done.

You time your casts based on the actions of your client, attempt to make sense of lag and server processing delay, and identify times based on the server times which can never be verified.

You instead time this by starting your watch when he tells you he's currently running, and stopping when he tells you he's done.

I time my casts off the client. I do not attempt to make sense of lag (lagged results are thrown out) and there is no server processing delay. All times are based on the client.

For my timing: I start timer, tell runner to run (+1 second processing time). Runner decides when to run (+1-5 seconds random lag). Runner begins running, waits a random 1-5 seconds, then tells me he's running which I disregard because it has no impact on my timing. Runner crosses the finish (+1-5 seconds random lag) tells me he's done, I stop watch (+1 second processing time). My time is anywhere from 4 to 12 seconds longer than the actual time.

In short, your times include: Client->Server REQ; Server Processing; Server->Client Start; Actual Casting Time and Server-Client Stop. Your times are longer simply because your recording more of the process.

For your timing: you tell runner to run, he takes 1 second to understand, then takes a random 1-5 seconds to begin running which has no impact on your time because you haven't started your watch yet. He starts running but doesn't tell you yet. He waits a random amount of time, then tells you (-1-5 seconds random lag because you're starting your watch AFTER he already started running). Furthermore, it takes you an extra second to react to him telling you he's running so you start your watch even later (-1 second processing time). He crosses the finish (+1-5 seconds random lag) tells you he's done, you stop your watch (+1 second processing time). Your time ranges anywhere from 4 seconds short to 4 seconds longer than the actual time.

In short, I monitor: Client->Server Start; Actual Casting Time and Client->Server Stop.

Your mistake is that I care about the start and stop lag times so that I can use them to calculate the times on the server. This is false - I am not calculating server casting times. If, during my testing, I lag at all - ALL of that sessions results are thrown away. As a result, my time ranges between 1-5 frames difference between casts - This is HIGHLY reproducible (it's not uncommon to see 20+ casts that all lie within the same 1-5 frame range). Lag is easily identifiable, as it usually results in frame differences around 10-15+ frames off standard.

At first glance, your timing method seems more accurate, yes. However, our margin of error is exactly the same, and we can shorten the track the runner uses by known percentages. That means we get to calculate our margin of error. The fact that my initial measurements are further from the true measurement than yours means nothing after we plug these numbers into equations to determine the true measurement.

Because we aren't testing the same thing, our margin of error is absolutely NOT the same. I've stated this before, and maybe I didn't make it clear, but the fact that your comfortable with a margin of error of 20+ frames is unfathomable to me. Your talking about having error margins higher than the values your attempting to measure.

And don't think for a second that your method avoids these calculations.

Actually they absolutely do. It might not be up to the standard of 'what the server says', but that IS EXACTLY why they avoid those calculations.

This randomness is why a consistant method with consistant results is more important than the points we choose to measure from. I could measure from exactly 30 frames after I give the command, but that doesn't mean I'm any closer to the actual time unless I account for the possibility of error.

Chances are good your initial recordings are closer to the actual casting time, BUT
1: that's not a guarantee you're closer, it's merely a statistical probability you are
2: equations will account for my margin of error
3: I would have to start over from scratch to change to your method
4: I would still have to use the equations on your method to ensure accuracy

These are some of the things I actually applaud you on. Just because I'm standing here telling you your wrong - should NOT be a reason to stop trying. Your absolutely correct in stating that as long as highly reproducible CORRECT results occur, then the means of the measurement should not matter. Obviously some measurements will be better than others, but as long as they are all the same it shouldn't be an issue. The bonus to this is that it offers a way of double checking results.

It wouldn't be hard to do both style measurements at the same time. However you should keep in mind that the method I use only cares about the client times (which if you have a solid setup, should be very close to the server times anyways). There is NO actual formula - at best you could probably throw up some standard deviation charts to identify any lag. But after you do it a few dozen times it becomes immediately noticeable.

[Asmoranomar]

Alright, alright- going over all my tests, my margin of error seems way too large to be able to accurately test items like loq. earring. Even though I really don't want to, I'll start retesting tomorrow using Asmoranomar's technique (modified so I count at party box cursor appearance, not overhead cursor appearance). With any luck, it might help narrow my results down a bit~

Actually i am GLAD you saw that when I posted my reply (I was writing it at the same time). Now that you see how bad your Margin of Error is it is restoring some faith I have in your future tests. However, don't feel bullied by my antics, if you truly feel that there is a means to test using your method you should continue looking for ways to narrow down your margins.

Like I said before, I really enjoy getting into the inner details of -SST and FC on BRD. Sometimes that might make me seem very pushy, but its because I get really excited over it and hope to uncover something new.

[Yinnyth]

In short, your times include: Client->Server REQ; Server Processing; Server->Client Start; Actual Casting Time and Server-Client Stop. Your times are longer simply because your recording more of the process.

Are you implying in the bolded section that the server waits until your computer has received confirmation before it begins counting down the timer on the song? There is still network latency involved here. Server begins counting your song, sends your computer signal that it has started. There is then network latency before your computer even receives the signal. The signal has to travel from the server across many router hops to reach your computer. After your computer receives the signal, it then processes it, loads the animation, plays the animation. The animation results in your character being targetted. You start your clock when your character is targetted. You are starting your clock late in comparison to when the server actually started counting the timer on your song and disregarding this fact.

[Asmoranomar]

I suppose in that aspect probably not. Similar to the scenario where pulling the plug prior to casting the spell results in your character doing nothing (whew that's long), IF you happen to lag so your client receives the 'Server->Client Start' late (but you receive your 'Server->Client Stop' normally) you'll just appear to cast faster. If the packet ends up being so late that the 'Server->Client Stop' somehow manages to get there before the 'Server->Client Start' (or it just ends up getting lost entirely) your spell will just fire immediately.

It would only affect you in situations where you were heavily lagged and usually results in the entire spell taking longer than normal to cast. I'll let you take that though.

However, my point still stands in the fact that your recording more of the process, and thus your times being longer.

[Yinnyth]

Just finished testing naked, nightingale, troubadour, and minstrel's ring using Asmo's method, and initial results are both promising and depressing. Promising because the variance from highest frame count to lowest frame count is generally smaller than in my previous tests. Depressing because there is still a 13-27 frame variation. I'm beginning to fear that 20 samples per set may not be enough to get the accuracy I seek from these tests...

Anyways, results. These are all 20 samples per set, recorded at a steady 60 fps:

Naked
Average 493.15 frames
Lowest 481 frames

Nightingale
Average 247.65 frames (-49.78% casting time)
Lowest 243 frames (-49.48% casting time)

Troubadour
Average 731.2 frames (+48.27% casting time)
Lowest 719 frames (+49.48% casting time)

Minstrel's Ring
Average 372.5 frames (-24.47% casting time)
Lowest 361 frames (-24.95% casting time)


I still need to test sha'ir and sheikh, then some combinations to see what order these calculations go in, and then I need to start calculating my margin of error so I can begin working on the items with an invisible amount of fast cast on them. I also have some central pieces of equip I should collect if I want my tables to be complete (full marduk is going to be a headache to collect just to run tests to prove that ebur tam + sheikh + rostrum is faster)...

Anyways, Asmo, I still don't understand what you're trying to explain about this method. In my mind, both methods still seem like they should be equally inaccurate, but it's hard for me to argue with the results. The variance in frame count is smaller in all 4 test samples which means your method should provide better accuracy.

[Kegsay]

the casting animation begins when you receive confirmation from the server that you have indeed begun to cast

Correct.

From my experience with FFXI's packets/coding, the way spells work is as such:

CLIENT: You select song and target, press enter to start casting.
CLIENT: Sends packet asking "can I cast x?"
SERVER: Responds "yes good chap you can. Here's the id of the animation to play."
CLIENT: "Jolly good." Begin animation.

Now this animation will continue for the max limit (30s iirc, easy to see by casting then pulling the plug, the cast animation will go on and on and on and on until there's no more left) until this happens:

SERVER: Sends packet: "okay, you've finished casting the spell. Here's your recast time."
SERVER: Sends CharUpdate packet: "here's your new effect."
CLIENT: Receives finished cast: Righto, starts the "end animation" where the song finishes.
CLIENT: Receives CharUpdate packet, adds effect icon.

The 2 server packets are sent almost immediately one after another.

EDIT: So time wise, the packets sent/received are (assuming 100ms to go one way, 10ms to "process" a packet, 10 seconds to cast spell):

Time       Who             Packet
0ms        Client          Sends RequestsCastSpell
100ms      Server          Gets RequestsCastSpell
110ms      Server          Sends AllowCastSpell -- Server thinks you start casting here
210ms      Client          Gets AllowCastSpell
220ms      Client          Starts Casting Animation
10220ms    Server          Sends FinishCastSpell -- Server sets lock on recast, timing down from now.
10320ms    Client          Gets FinishCastSpell
10330ms    Client          Finishes the animation.

Hope this helps.

[Asmoranomar]

13-27 frame variation

I have no idea why your seeing this (unless your seeing just one or two variations). As I said before, my system only showed 1-5 frames between casts. That is...unless your still trying to determine cast times from the server.

If you could somehow lower your settings down, disable animations, kill some unnecessary processes, obtain the best drivers for your gfx card, defrag your HDD, unload unnecessary plugins and third party software, etc etc etc, you might be able to lower your margins a bit.

I have a mirrored raid setup. I know it helps tremendously on area load times, but I don't know if that would have anything to do with speeding up load times for spells.

The fact that your seeing around the same margin of error does mean your first method of testing should be as accurate as this method, provided you can narrow down what is causing your frame difference issue.


Edit: I just took a stab at this myself, just to make sure I wasn't remembering incorrectly. Using Fraps to record and VirtualDub to scroll thru independant frames on an un-optimized system, I was able to obtain the following vaules:

Frames	Time	
480	8	-3.105263158
480	8	-3.105263158
482	8.033333333	-1.105263158
485	8.083333333	1.894736842
482	8.033333333	-1.105263158
482	8.033333333	-1.105263158
481	8.016666667	-2.105263158
482	8.033333333	-1.105263158
483	8.05	-0.105263158
480	8	-3.105263158
481	8.016666667	-2.105263158
481	8.016666667	-2.105263158
483	8.05	-0.105263158
485	8.083333333	1.894736842
482	8.033333333	-1.105263158
484	8.066666667	0.894736842
482	8.033333333	-1.105263158
482	8.033333333	-1.105263158
502	8.366666667	18.89473684 <-  System lagged when it was changing desktop backgrounds :D

[Yinnyth]

I'm running off a striping RAID for my main HDDs, 2.4GHz quad core, 4GB RAM, with 2 GeForce 8800 GT cards running in SLI mode and I use this computer for nothing but gaming. Everything that comes onto it is found, downloaded, and scanned on my laptop, then sent over across the LAN. So I don't think the problem is inside my computer, I think it's between my computer and the servers.

Don't get me wrong, my margin of error is smaller now with this method, it's just not as small as I would have hoped for. Casting a dozen songs and never having them vary by more than 5 frames is a bit hard for me to imagine though. I'll do tomorrow's testing in lower settings and see if that narrows it down any I suppose.

But I've gotta ask, what framerate were you recording at when you got a variation of no more than 5 frames? Also:

I repeated it about a dozen times each for each test, threw out any values that were obviously incorrect (easy to notice) and laid out my results.

How does one decide when a result is alright to throw out? I could get my results to be extremely consistant if I simply throw out anything that doesn't fit with the expected values, but then that sort of defeats the whole purpose of the test.


And Kegsay, thanks. That's somewhat how I imagined it working, but organized much better than I could ever word it myself.

[Asmoranomar]

Meh I updated my post right as you were doing yours. To answer your question, I bolded the part where I obviously lagged and got a large frame difference.

I've also ran into situations where I lagged on the initial cast time, in those instances, I get a lower frame count - lower than the expected casting time of the spell. So I also throw out any values that are lower than the value I'm supposed to have (in the case of naked songs, I obviously cant cast faster than 8 seconds - it gets more difficult when you actually put fast cast gear on).

striping RAID for my main HDDs, 2.4GHz quad core, 4GB RAM, with 2 GeForce 8800 GT cards running in SLI mode

Only thing I can imagine then is the SLI mode cards. FFXI doesn't play well with SLI and some of the SLI capable drivers were not optimized for FFXI even if you didn't run SLI (This I'm absolutely certain on back when I did my tests ages ago, as I ran into issues with this). I don't know if the issues have been cleared up since then, as I no longer run SLI anymore.

[Yinnyth]

I'll try killing the SLI then- just got it because it was new and shiny when I was building my computer. Your values look very consistant- unbelievably so. If I can't get this more consistant by disabling SLI and lowering all my settings, then all signs will pretty much point to my connection.

http://forums.trutlels.com/images/FastSongs.png
This is a pic of the spreadsheet I recorded my last tests on- highest values highlighted in purple, lowest in blue.

[Asmoranomar]

The tests aren't bad. Your naked tests have good values that are within expected results (though a few frames higher than mine for some reason). If you changed your conditional formatting to show standard deviation instead of min/max you'd see where your lag was. Your tests show you have higher lag rate, but you can clearly see it when it occurs.

I'll take a look at the rest of the data in a bit.

Edit:

Damn, your still showing about a 8-10 frame inaccuracy in the other tests.
This is better though - you should look at std dev on your other batch of tests too.

[Yinnyth]

Standard deviation on my old tests using my command to finish method:
Naked: 11.35
Nightingale: 7.33
Troubadour: 13.49
Minstrel's Ring: 6.68

On this newer batch of tests using your method:
Naked: 5.65
Nightingale: 3.88
Troubadour: 8.33
Minstrel's Ring: 5.10

Better, but still not a pretty list of values.

[Asmoranomar]

Your getting close enough to do some more accurate tests though - Just means you have to do more trials to to narrow things down.

[Yinnyth]

Lowered all my settings down to nearly nothing, disabled SLI, turned off all in-game effects, and retested naked songs.

20 samples naked
Average: 498
Low: 493, high: 503
Standard deviation: 3.76

Lower deviation, so I think this is about as good as I'm gonna get without building a computer with every component optimized for FFXI and moving next door to SE. From what I hear, the GeForce 8800 series are actually kinda bad for FFXI though ._.;

Anyways, I think my margin of error is low enough now that I can get started on some more interesting tests, starting with sha'ir vs. sheikh...

Edit: spoke too soon, my sha'ir manteel tests are ugly.

20 samples, sha'ir manteel:
Average 428.2 (-14.02%)
Lowest 421 (-14.60%), Highest 448 (-10.93%)
Deviation: 8.38 (eww)

Edit edit: sheikh is pretty ugly too

20 samples, sheihk manteel:
Average: 426.35 (14.39%)
Lowest: 421 (14.60%), Highest 438 (12.92%)
Deviation: 6.38


I'm starting to think maybe I should retest the number that's furthest outside of the standard deviation until I get a deviation of less than 4 on all of these... gonna see how that turns out.

[Yinnyth]

Ok, more testing. This time, I took 20 base samples, then I took extra samples. The value furthest from the average of all 20 values would get replaced by extra samples until the standard deviation of the 20 samples was lower than 3. From this method of testing, I received the following values:

naked average: 498.85
deviation: 2.60

Sha'ir average: 424.4 (14.92%)
deviation: 2.96

Sheikh average: 423.9 (15.02%)
deviation: 2.90

Minstrel's ring average: 377.3 (24.37%)
deviation: 2.72

Nightingale average: 249.4 (50.01%)
deviation 2.80