We've never observed any weird overflow effects other than capping at 255 seconds. That cap would only be the result of sending the information out as a single integer byte, which is what I think it is.
Agreed.
Pairing the values with their bases and treating everything as integers until the quotient is taken will work and has already been shown that it can work, there is no need to complicate the system beyond that.
While I disagree in terms of the technical aspects, this will probably be sufficient for most of our purposes. Tentative agreement.
assuming overflow is avoided (which I think it is because it's never been observed and would be a huge bug if it was,)
Well, certainly overflow is avoided. The question is -how- is it avoided? We see part of it in the manner in which recast times are stored. However if it's ever possible to increase recast above ~64 seconds before you reach the haste portion of the equation, multiplying by 1024 (0% haste) will always generate overflow.
This can be avoided if all other terms occur after haste, and none of them are stored in /1024. However it seems easier to convert all the haste values to floats, add those up, and then multiply the floats (haste and recast time) together.
Ultimately probably doesn't matter, though.
I wrote out some comparisons using binary math, but then fiddled with some possibilities in excel, and think I found something (correction: was an error due to Spellcast equipping my Loquacious Earring... However it's still useful in disproving certain approaches).
Haste can be considered in one of three ways:
1) n/1024 (standard current usage). 2% haste = 20/1024. This leads to our current issue about value representation, so won't go any further.
2) Accumulated units of /1024 fractions. If haste is stored as n/1024, and we know that float16 can represent a /1024 exactly, we can take, say, 2% and simply convert it into the closest increment of /1024. This is mostly identical to #1, but will vary depending on the order in which things are added.
3) A simple fractional representation stored as a float16. Of particular interest is that float16 can represent values to the nearest /2048 for fractions below 1.0 (not bothering to work out how far down that goes, but at least a moderate bit is sufficient).
So, I set up a check on recasts using three values:
Set a percent target (2%, 3%, etc)
Haste1: 1 - n/1024 [n is chosen to match current accepted values, such as 20 for 2%]
Haste2: 1 - floor(target, 1/1024)
Haste3: 1 - floor(target, 1/2048)
Most of the time it seems that Haste2 and Haste3 result in the same value, so their results are identical. However at 7% I find a discrepency: Haste1 and Haste2 predict a 10 second recast for Curaga IV (using either 70/1024 or 71/1024 for Haste1), while Haste3 predicts a 9 second recast.
(corrected) Observed recast: 10 seconds.
Conclusion: the haste resolution must be determined before converting to float16 (thereby: /1024 resolution, not x% haste that may end up with /2048 resolution).
Secondary confirmation of (presumably known) restriction:
9% haste would be 92/1024. If haste were the percent values added together before converting to /1024, Goading Belt + Goliard Saio would be 92/1024; otherwise 91/1024.
91/1024 recast of Aero III: 23 seconds
92/1024 recast of Aero III: 22 seconds
Observed: 23 seconds
So the haste value must already be in /1024 format before being added together, not given as their original integer number that's added together before converting to float. This allows for the possibility that they are already stored as float16's.
Rewrote things a few times, ending up with this in the spreadsheet:
Code:
BaseProduct = 4 * recast / (1024 - haste)
Where recast is base recast, and haste is haste as /1024 value (eg: 150 for the spell Haste)
Code:
Calculated recast = GetFloat16(BaseProduct)/(4*1024)
The extra multiplication/division by 4 was to ensure integral values for the bit manipulations.
GetFloat16 VBA:
Code:
Public Function GetRoundFloat16(x As Long)
Let exponent = 0
If Int(((Log(x) / Log(2)) + 1) - 11) > 0 Then
exponent = Int(((Log(x) / Log(2)) + 1.00000000000001) - 11)
End If
Let lowerBits = Int(2047 / 2 ^ (11 - exponent))
Let andLowerBits = BITAND(x, (lowerBits))
Let roundBit = 0
If andLowerBits > ((2 ^ lowerBits) / 2) Then
roundBit = 1
End If
Let mainBits = 2047 * 2 ^ exponent
Let andMainBits = BITAND(x, (mainBits)) + roundBit * (lowerBits + 1)
GetRoundFloat16 = andMainBits
End Function
Which depends on another VBA function, BITAND:
Code:
Public Function BITAND(x As Long, y As Long)
BITAND = x And y
End Function
GetRoundFloat16 uses simple rounding. It can be adjusted if/when we determine the actual rounding method in use.
There's another version which doesn't add the roundBit value called GetTruncFloat16, for comparison purposes, though I'm hard pressed to find anything where any difference shows up at all.
Also note that trying to do the rounding this way can break the function when combining terms (eg: haste + fast cast) due to excessive bit shifting. The truncation version doesn't have that problem.
Testing Haste first vs Fast Cast first, using these formulas:
Haste first:
Code:
Haste: GetTruncFloat16(4*recast*(1024-hasteVal))/(4*1024)
FC: GetTruncFloat16(GetTruncFloat16(2048*hastedRecast*(100-fcVal))/100)/2048
FC first:
Code:
FC: GetTruncFloat16(GetTruncFloat16(4*128*recast*(100-fcVal))/100)/(4*128)
Haste: GetTruncFloat16(4096*fcRecast*(1024-hasteVal))/(4096*1024)
Extra multipliers are in there to ensure the fractional portion is available for bit manipulation.
Several tests quickly showed haste to be calculated first, which matches current understanding.
Edit: Updated the Float16 formula; exponent needs a tiny addition to prevent issues with rounding of the 64-bit double value.