Testing out fast songs, but I could use some help

[Yinnyth]

Edit: I found a relatively reliable way to test these values, though it is not 100% accurate. As such, it is up to you whether or not you choose to believe the results of my testing.

How I tested for these values
I turned my game down to the lowest possible settings, filtered special effects, went to a corner of the chocobo circuit and recorded myself casting songs in various setups. Everytime the cursor appeared over my head while casting, I hit esc to remove it quickly (I use the cursor as a reference point for when the song starts and finishes).

I recorded using fraps at 1/2 quality (no sound) recorded at 60 fps for highest accuracy. When reviewing the videos, I used Virtual Dub to inspect everything frame by frame. To determine how long a song took to cast, I would find the first frame where the cursor appeared next to my name in the party box (lower right side of screen) and record that frame number in a spreadsheet. I would then scroll ahead in the video until the same cursor reappeared as I was finishing the song and record that frame number in the same spreadsheet. The spreadsheet was set up to subtract the two frame counts from one another to determine how many frames it took to go from start to finish. I repeated this process until I had 40 samples for each setup I tested.

From these samples, I had excel calculate various things for me. I plugged in the expected frame count for each setup under the assumption that a song normally takes 480 frames (8 seconds) to cast. I then compared the lowest frame count, average frame count, and highest frame count for each set to its expected frame count. The resulting differences I called lowdev, avdev, and hidev. On my baseline tests with known alteration %s (nightingale, troubadour, minstrel's ring), I took an average and standard deviation of the lowdev, avdev, and hidevs. The standard deviation showed me that my lowdev was the most accurate value I had for determining an item's song casting adjustment. The averages of the lowdev, avdev and hidev also helped me to adjust the expected frame count of items with unknown song modifiers to be in line with these averages.

How song casting time is calculated
The basic equation for song casting time is relatively simple. All sources of song spellcasting time -%(SST) and fast cast(FC) are added together and multiplied by 0.5 if nightingale is active, and multiplied by 1.5 if troubadour is active. The combination of SST and FC has an absolute cap at 50% reduction, but this cap is ignored by nightingale meaning one could reach -75% casting time on songs under ideal conditions.

Estimated casting time reduction of items I have currently tested
Minstrel's Ring: -25%
Sheikh Manteel: -12.5~13%
Sha'ir Manteel: -12~12.5%
Rostrum Pumps: -2~2.5%
Loquacious Earring: -1~1.5%
Veela Cape: -1~1.25%
Ebur Tam: -1~1.5%
Fast Cast II (from /rdm): -15%

Various thoughts
I also tested what I felt to be the ideal suit for fast songs minus the dweomer knife which I just can't get a good augment for... It consists of ebur tam, sheikh manteel, veela cape, loq earring, rostrum pumps, and minstrel's ring. My testing shows it grants somewhere around -45% casting time total. The tam also has the ability of being imbued with an evolith that reduces song casting time of a specific element by up to 3%. So with the absolute perfect set of gear, it should be possible to reach the cap of -50% SST without subbing RDM.

Like many percentage based bonuses, SST becomes exponentially more useful the more of it you have (up to the cap). It is possible that in the coming year as the level cap increases, more and more sources of FC and SST will become available, so it may even become possible to reach the cap without forcing latent on ring which is already enough of a pain seeing as how bard is usually expected to /whm.

[Tradition]

Do some math instead of depending on an external program's output?

[Yinnyth]

Elaborate

Edit:
Or perhaps I should. I have some specific goals in mind with these tests, but it begins with compiling the base song casting time duration of various items and abilities. As with haste, damage reduction, and every other value in the game, I would imagine the "Song spellcasting time -13%" is actually a hidden value that is slightly off from what is actually listed on the item. In addition to this, Ebur Tam merely claims "Enhances Fast Cast effect". So it is difficult to simply math my way through unknown values.

After I have these basic values, I wish to see how they add up with one another. Does 10% fast cast + 10% song casting time = 20% reduction, or 19% reduction? Further, is there a cap on fast cast, song casting time down, and/or total bonus? If there is, does nightingale break this cap?

[Robbinthehood]

I wouldn't use fraps. 25fps won't give you an accurate readng on time. If you want to go that route you would be better off using something with 29.97/30 fps and get as close to real time as you can. You could also use a stop watch with milliseconds.

[Yinnyth]

I wouldn't use fraps. 25fps won't give you an accurate readng on time. If you want to go that route you would be better off using something with 29.97/30 fps and get as close to real time as you can. You could also use a stop watch with milliseconds.

While you're right about the 25fps, unfortunately I would be even worse off with a stopwatch. Fraps records these frames up to 60 times per second and lets me scroll through the frames one by one to see exactly which frame a certain action occurs. Relying on a stopwatch would add an extra inaccuracy to the mix: human reflexes. I need to be accurate to within 0.05 seconds ideally, and my reflexes are not that quick.

I have made a few refinements to my system, and I'm beginning to understand exactly how much of an ordeal I am taking on. I've set the fps on fraps up to 60, and my system handles it without any loss in framerate. I've also increased my sample size from 5 to 20 for now. However, there is still a variation of 41 frames (nearly 0.7 seconds) from my fastest song to my slowest song without any changes in equipment.

So where are these variations coming from? Well, it turns out they're caused by internet latencies. I tell the server when I begin singing, and the server tells my computer when my song has finished. If my connection lags for 3 seconds just before my song ends, I view my song as taking 3 seconds longer than it should have, and no recording program or stop watch can remedy this problem.

So I have a feeling I'm going about this the wrong way by looking for an average. I should be using enormous sample sizes, and picking the fastest occurence. The server would never allow my song to finish faster than it should be capable of finishing, so in an infinitely large sample of castings, one could assume that the fastest one was the one with the least latency.

.....this is going to take me a lot of time....

[Masamune]

So I have a feeling I'm going about this the wrong way by looking for an average. I should be using enormous sample sizes, and picking the fastest occurence. The server would never allow my song to finish faster than it should be capable of finishing, so in an infinitely large sample of castings, one could assume that the fastest one was the one with the least latency.

Nice conclusion, and also nice explanations how you came to it and ways you used /clap

Too bad i can't be of much help on that topic, but i'm trying to find a way me too to test something else (Time To WS)..... a way avoiding server latencies exactly like you pointed.

Good side of such testing is to give us an approximate idea of server latencies under different circumstances (non lagging, medium lag, heavy lag) to be use for example in our spellcast .xmls Castdelay value
Example: most xml use value=0.5... you just proved under completely non lag environment, latency can be up to 0.7. Another example when i compare my WSs damage in Dynamis while blocking/unblocking my WSgears with a castdelay value=1: almost exact same average (~20 WSs for each test in same dyna tough :s), but pretty self explanatory test (ie WSgears useless in heavy lag situations lol... unless use higher Castdelay value)

[Byrthnoth]

Try PMing Kirschy. She's the one who has done the most of this in the past.

[Yinnyth]

I am actually fortunate enough to be in the same linkshell as Kirs, and she's been helping me design and reform my testing strategies. However, she's often too polite to call bullshit on my stupider ideas. BG has no shortage of people willing to do that in her stead, and some of them even know their head from their ass.

Anyways, last night just before I went to bed I noticed yet another imperfection in my testing strategies and was forced to scrap all my data again and start over from scratch. I was counting the frames between when the blue subtarget cursor disappears to when the bronze main target cursor appears over my head at the end of the song. It turns out that the cursor does not appear over your head when you gain the effect of the song, but rather it appears after the "target" box has completed its appearance animation which I just did some testing on, and it does vary by a few frames from test to test. So instead, I now do my counting from when the blue subtarget cursor disappears to when the bronze target cursor appears next to my name in the party list. Pic for teh proof.

http://forums.trutlels.com/images/Target.PNG

[Asmoranomar]

I've done tests like this in the past using fraps in a low latency area (chocobo circuit). Your basically doing the same things I've done except that I clocked my start/stop times upon when the in-game cursor appears. Specifically:

Upon casting (with a macro) and nothing selected, a cursor will appear over your head. This doesn't cross frames, and I used this as my START time.

After starting the song, clear target selection. A cursor will again appear over your head when the cast is complete. I used this as my STOP time.

I repeated it about a dozen times each for each test, threw out any values that were obviously incorrect (easy to notice) and laid out my results. Generally most equipment was within their stated values when compared to actual frames/seconds an unaltered song takes.

I think the only problem you have is your designated start time, there's been many times I've selected to cure or something on a target and there would be a delay from when I actually did it and when it actually started.

After I have these basic values, I wish to see how they add up with one another. Does 10% fast cast + 10% song casting time = 20% reduction, or 19% reduction? Further, is there a cap on fast cast, song casting time down, and/or total bonus? If there is, does nightingale break this cap?

There appears to be a cap of 50% for casting time for both Fast Cast and Song Spellcasting, combined. Nightingale will bypass this cap. It is assumed all casting reductions are based on 1/1024 scale similar to haste.

Edit: I've not seen a situation where the cursor isn't above your head and on the menu at the end of a cast, ever. That's interesting.

Edit #2: Might have been said before, but 25 FPS sounds pretty meh. I can get near 30 FPS without scaling stuff down in an unlagged area. Obviously for the tests you want to reduce everything and turn off any animations. If you want some serious results and accurate measurements, your going to want as many frames as possible.

[Yinnyth]

Yeah, I've turned fraps up to record at 60fps now. The game stays steady at 29.97 fps during all my tests, so fraps is getting about 2 frames per every 1 frame the game displays.

I just finished my first 2 rounds of testing (technically my 17th and 18th if you count previous failures -_-). 20 samples naked, 20 samples naked with nightingale.

Naked Average: 514.25 frames
Naked Lowest: 496 frames
Nightingale Average: 272.25 frames (257.13 was the expected value)
Nightingale Lowest: 259 (248 was the expected value)

Nightingale is supposed to be a 50% reduction, but is averaging around 47.5% reduction in these tests. This actually helps me to determine latencies between me and the server if we assume nightingale to be exactly 512/1024 reduction in casting time. According to these numbers, I am losing about 30.25 frames (half a second) on average due to latency, and about 22 frames (one third of a second) on my fastest songs due to latency.

I'm working on testing troubadour now, but since I can only get 4 songs per troubadour, it's on 10 minute cooldown, and I'm too lazy to run back to MH each time, it's gonna take me a little while longer. Hopefully I'll be able to plug these numbers in and see if my assumed latencies fit with these values too.

[Asmoranomar]

are you trying it based on the method I described? I still think your start time is poor.

I was able to get 49.66% on Nightingale back when i did my previous tests.

[Yinnyth]

From what I understand, casting spells goes something like this:
1: You input command
2: Computer sends command to server, begins animation
3: Server receives command, begins timer
4: Server's timer is up, sends signal to computer that song is in effect
5: Computer displays song finishing

My timing is from one frame after step 1 to one frame after step 5. Since you begin your timing when the bronze target cursor is above your character's head to when the bronze target cursor reappears over your character's head at the end of the song, you're inviting an extra variable which I explained 2 posts ago (with the picture). The forced target procedure goes like this:

1: Action begins on you, target box begins an animation, party box cursor appears, status effect appears if applicable
2: 20-25 frames (at 60fps) pass as the target box finishes its animation
3: Target box finishes animation, target name displayed in target box, target cursor appears over target's name

There are many reasons you could have gotten 49.66% on your nightingale. Assuming songs take 8 seconds to sing, assuming you mean this is your average over more than a dozen samples, then what you're implying is that your latency is so low that you are averaging 0.03 seconds for a signal from the server to reach your computer, be processed, and then displayed on your computer, which is amazing and also unlikely. If you got 49.66%, that was most likely luck.

Unfortunately due to the nature of these tests, you need to be able to account for the lag of the intarwebz. It's not so much about how closely your computer matches the expected timing, but rather how closely grouped your values are to one another. In 20 samples on naked songs, my highest and lowest values varied by 42 frames (at 60fps). If you had smaller deviation in your samples, I'll give it a try on my machine and see if it significantly lowers my margin of error.

[Asmoranomar]

The point being, you don't care about the timing you input the command to the time the spell ends. You care about the timing from between when the spell actually begins to the time the spell ends.

In a perfect world the two might be equal, but it isn't here. Your adding additional latency and processing time to your casting times that you shouldn't be. Your fast cast gear and song spellcasting gear will not reduce the times of latency and processing. Adding those times into your calculations is causing your results to be off.

you are averaging 0.03 seconds for a signal from the server to reach your computer, be processed, and then displayed on your computer, which is amazing and also unlikely. If you got 49.66%, that was most likely luck.

If you measured the way that I did, your not taking into account latency or server processing time. The 0.03 seconds is only the time it takes for my PC to display the results(animations) the moment my PC receives the signals for start/stop. If I lagged during the testing, it would be considerably larger and higly noticeable.

Think of it this way:
While we both know that the cast time includes latency and processing, the only way to reduce that is to either get a better connection or a better server - not better gear. It's therefore counter productive to start the measurement when you hit 'go', but instead wait till the game says 'i went' (ie: cursor appears).

Due note tho, its not entirely without latency - technically I still get some latency from when the server sends the 'start' and 'stop' to my PC, but its a lot better than measuring the entire process.

Edit: Looking at your steps above on how casting works, I noticed that you missed some key point between step 1 and step 2. The game doesn't start the animation the moment you send the command, but rather the computer sends the command, waits for an acknowledgment THEN starts the animation. This can be tested easily by removing your network cable from your PC before starting your test, then attempting to cast a spell. Your client wont animate anything. Basically your spell processing looks like this -
1) User inputs command. Client notifies server of action.
2) Server accepts Client action. Server sends ACK to Client. Sever begins action timer.
3) Client receives Server ACK. Client begins action.
4) Server ends action timer. Server signals Client action end/effect.
5) Client receives action end/effect. Client terminates action.

Your measuring step 1-5, I measured step 3-5

[rog]

I'm working on testing troubadour now, but since I can only get 4 songs per troubadour, it's on 10 minute cooldown, and I'm too lazy to run back to MH each time, it's gonna take me a little while longer. Hopefully I'll be able to plug these numbers in and see if my assumed latencies fit with these values too.

Do it in norg then? There should be no one around, and you don't even need to zone.

[Yinnyth]

You also get latency as your computer displays the target box animation unless you use a hybrid of our two techniques. However I do not see the advantage to your technique over mine. Let's theorize 2 ways casting works since I currently lack the means to prove which way it works in reality. Either the casting animation begins on your computer the moment you input the command to cast, or the casting animation begins when you receive confirmation from the server that you have indeed begun to cast. Let's assume "b" equals the number of frames it takes for a command to be sent either way between my computer and the server.

Animation begins during command
My method: I begin counting frames before signal is sent to the server, counting "b" too many frames before the server receives my signal and begins casting. Result: "b" too many frames.

Your method: You begin counting frames before the signal is sent to the server (because we're assuming the computer doesn't need to wait for confirmation before the animation begins), counting "b" too many frames before the server receives your signal and begins casting. Result: "b" too many frames

If this is the case, then our methods are essentially identical.

Animation begins after receiving confirmation from server
My method: I begin counting frames before signal is sent to the server, counting "b" too many frames before the server receives my signal and begins casting. Result: "b" too many frames.

Your method: The server begins casting, sends a signal to your computer that you have begun casting, and your computer receives this signal "b" frames later. Result: "b" too few frames.

If this is the case, then our methods have the exact same margin of error due to latency, it's just that the margin of error is in opposite directions.

I've started compiling my information thus far on the OP. Right now I'm attempting to pinpoint what my "b" value is so I can begin testing in earnest.

[rog]

Either the casting animation begins on your computer the moment you input the command to cast, or the casting animation begins when you receive confirmation from the server that you have indeed begun to cast. Let's assume "b" equals the number of frames it takes for a command to be sent either way between my computer and the server.

Just try to cast while dcing to find out.

[Asmoranomar]

However I do not see the advantage to your technique over mine.

The advantage is that in your case, your measuring extra values - namely, one round of latency between your PC and the server and server's processing time. Both of which could add significant times to your tests AND is unaffected by -SST and FC. Sure, you can try to account for that, but why? You don't have to.

Let's theorize 2 ways casting works since I currently lack the means to prove which way it works in reality.

You mean to tell me you can't unplug your network cable and attempt to cast? I can assure you the client will not do anything. Not only that, but it's pretty common knowledge if you've every lagged horribly bad or have been d/c'ed while playing. You can move around, but no matter how often you spam that cure macro nothing will happen.

As for your scenarios - in both cases, under ideal conditions, we'd be counting too many frames, not too few. Latency is always going to add extra time, even if its just the server telling my client to stop. The only time I'd get too few is when I'd hit lag triggering the start time - and it would be easily noticeable.

I think your too focused on trying to determine the exact time the server will start and stop your times. You'll never be able to determine that. Your best alternative is to measure when the server tells the client to start/stop, not a hybrid of the two.

You also get latency as your computer displays the target box animation

I never said you were wrong (clearly you have a pic), I just said it was interesting and it never occurred for me. This could easily be attributed to things like running in a higher/lower resolution setting, PC specs, 3rd party software, drivers, etc. The fact that it is *possible* makes question to my line of tests (and also yours but you mentioned that), which is why I thought it was "interesting". I still think it is reliable, but it does raise the question of whether something else would be more accurate.

I think your initiative is good and I really enjoy seeing more research into gear (especially the newer gear), but looking at your data the only thing that comes to my mind is:
"Holy crap, your testing shows it takes over 8.5 seconds to cast a song naked on average?"
If you can't narrow that down, then it makes it pretty hard to believe the rest of your results.

Edit: My second thought would be the difference in your average and your lowest (thus indicating you may have had similar high values, or overall higher value), those are some really FAR off values. Under ideal conditions you shouldn't have more than 1-5 frames difference, not 20-40 (reference to naked).

[Yinnyth]

You mean to tell me you can't unplug your network cable and attempt to cast? I can assure you the client will not do anything.

Pulled plug:

Error code: POL-2059
Connection terminated or not available.
Please try connecting again later or
check cables, networking devices, and
network settings.

That pops up within 2 seconds. If you can get away with that on your computer, you're also capable of forcing R0 to ghetto pos hack you way through normally dangerous areas.

I know which way I believe it works, but since I lack the means to test myself and have never seen extensive tests proving either way, I dare not make the assumption by citing "everyone knows" as proof.

As for your scenarios - in both cases, under ideal conditions, we'd be counting too many frames, not too few. Latency is always going to add extra time, even if its just the server telling my client to stop.

The actual casting time of any spell is determined only by the server. Assuming the generally accepted model of spellcasting, there is randomized latency at 3 points.
1: When you send signal to server, there is lag before the song actually starts.
2: After the song starts, there is lag before your computer receives confirmation.
3: As the song ends, there is lag before your computer receives confirmation.

My method suffers from latency types 1 and 3. Your method suffers from latency types 2 and 3. If all latencies were exactly equal, your method would give you the EXACT casting times of songs. But they are not all equal, they are all random.

Now as to why lag would make it appear as though a song takes less time to cast, it's because you don't begin measuring until after you receive confirmation. The song has already begun, but you haven't started counting yet.

I'm reluctant to start over from scratch yet again because this is so time consuming, and I don't see how it will improve the accuracy of the test. I'll consider it and possibly start over tomorrow.

I think your too focused on trying to determine the exact time the server will start and stop your times. You'll never be able to determine that.

That is indeed what I am focused on. Nothing else matters. The time the server starts and stops is the true casting time. Any variations to casting time will affect ONLY the server's stopping time, so if I want to know exactly how much effect any piece of equipment has on casting time, I need a way to account for the lag. Yes, it's impossible to be 100% accurate, but a reasonably consistant method of measurement with proper interpretations is the only way to get close.

Yes, by the points of measurement I'm using, my client takes 8.57 seconds on average to finish a song which is assumed to take 8 seconds to cast. But with nightingale active, my songs take 4.54 seconds to cast on average. That means I'm getting consistant results. Songs are taking somewhere around half a second longer thanks to lag, but it's consistantly longer, and that gives me the ability to make further measurements.

[Yinnyth]

Edit: mistake- delete if you have super powers please~

[Asmoranomar]

My method suffers from latency types 1 and 3. Your method suffers from latency types 2 and 3. If all latencies were exactly equal, your method would give you the EXACT casting times of songs. But they are not all equal, they are all random.

Incorrect, your method suffers from all 3 latency types. If you lag during type 2, it will have an impact on the results of the tests. Additionally, type 2 latency would only affect me if my type 3 latency was less than my type 2 latency (read: i would appear to cast faster than i should, or it would appear i was less lagged than I really am - both cases easily identifiable).

However, when using my style of test, If I lag during step 1, it has no impact whatsoever on the results of my tests.

In addition, you left out the server processing latency. Yours would suffer during that, mine would not.

Now as to why lag would make it appear as though a song takes less time to cast, it's because you don't begin measuring until after you receive confirmation. The song has already begun, but you haven't started counting yet.

That isn't the reason. Yes, technically my character on the server begins casting before my client does, but as long as the latency for both start and stop are the same, the times on the client should be identical to the server. Any lag I would get would also be reflected upon your setup as well. The reason it appears to cast faster is because my test is more accurate - you make it sound like I'm casting naked songs under 8 seconds, I'm not. Your testing says your casting naked songs over 8 seconds.