MACC Debate (From Random Thread)

[Kaeko]

Actually screw this, starting a thread for this b/c we're ruining the random thread.

Can you post there?

[Kaeko]

http://www.bluegartr.com/forum/78670...ml#post2918538

Please post this stuff here instead of randomer thread (since this may require more debate than the random thread would want).

To recap, we are talking about a data set based on nuking Qiqirn.

http://wiki.ffxiclopedia.org/wiki/Magic_Accuracy

http://images4.wikia.nocookie.net/ff.../Stone-QQP.jpg

The link and an associated chart can be found there.

In particular, we are debating THIS:

* When calculating the landing rate of a spell (without considering partial resists), the game first calculates your total amount of magic accuracy :

M=Magic skill + M1 + M2

where M2 is your total magic accuracy from gear/merits and M1 is the contribution of your stats to magic accuracy :

M1=(caster's stat) - (target's stat) if (caster's stat) - (target's stat)<10
M1=10+((caster's stat) - (target's stat)-10)/2 if (caster's stat) - (target's stat)>=10.

* This number M is then checked against the target's magic evasion (meva) and we have the following

landing rate(%) = 50 + X*(M-meva)

where X=0.5 if meva>M and X=1 if M> meva.

The value of meva is then easily determined by checking you current landing rate. The meva of level 77 (65 INT) Qiqirn Poulterer in Alzadaal Undersea Ruins has been determined to be comprised between 290 and 292 using this method. This means that cap accuracy will happen with ~340 magic accuracy. Using the same method, VT Heraldic Imp in Caedarva Mire have been determined to have a meva for the spell sleep of around 390.

Complete tests found here [Openoffice doc](http://rapidshare.de/files/47927000/..._bis_.ods.html) with the graphical summary here http://wiki.ffxiclopedia.org/wiki/File:Stone-QQP.jpg

exemple 1 : If my landing rate is 40%, the target's meva is 20 points above the value of my magic accuracy.

exemple 2 : (Useful for testing) If my landing rate is 47% with +/- 5% error, the target's meva is between M-2 and M+16 where M is my magic accuracy.

* If p is the landing rate calculated above, an elemental spell will also have chances of partial resists, but their probability will only depend on p in the following way :

no resist : p %
1/2 resist : (1-p/100)*p %
1/4 resist : (1-p/100)^2*p %
1/8 resist : (1-p/100)^3*100 %

in the case of spells that only have one form of partial resistance (such as duration resists for sleep or poison), the probabilities would be :
no resist : p%
1/2 resist : (1-p/100)*p%
full resist ("resist ! .. " message in the log) : (1-p/100)^2*100

[pchan]

Elemental magic received accuracy bonuses so that tier-II was harder to resist than tier-I, and tier-III harder to resist than tier-II.

[url=http://www.jpbutton.com/?p=157#more-157]JPButton

[pchan]

Found where I read the tier resist rates.

Elemental magic received accuracy bonuses so that tier-II was harder to resist than tier-I, and tier-III harder to resist than tier-II.

You can interpret this either as the player gets +mac or the mob get -meva. It's transparent, there is no way you will be able to differentiate both as the only stats that matters is (magic eva - magic accuracy).


"http://www.jpbutton.com/?p=157#more-157"

[Kaeko]

(MACC debate related)

I'm also extremely confused by this graph on Wiki you posted:

http://images4.wikia.nocookie.net/ff.../Stone-QQP.jpg

The "95% intervals" seem to be derived from your "theory" points. 95% confidence intervals are supposed to be derived from DATA POINTS, not your theory or "the line I'm looking for". Please forgive if I'm misunderstanding how you're deriving the idea that "my data fits", but I just don't understand how you can produce confidence intervals at points of data that do not actually exist (only "theory points").

The data are in blue on the graph, each data point has at least 1K sample size which gives around +/- 3% error, hence the confident interval.

But the "intervals" are coming out of your "theory" line, not the actual data point you collected (1k trials each). Confidence intervals come from your data points, not your theory.

This is very important because I could just make some huge ass squiggly line that perfectly fits your data and if I make the confidence intervals coming from my squiggly line, it will of course fit if I want it to. This is a hyperbole but I hope you get my point.

Basically, I think the graph needs to be adjusted and you really should manually calculate the confidence intervals and have them coming from the data. Seeing a chart where you have confidence intervals coming from data points that don't even exist just screams something is wrong.

You can interpret this either as the player gets +mac or the mob get -meva. It's transparent, there is no way you will be able to differentiate both as the only stats that matters is (magic eva - magic accuracy).

As I mentioned a few posts back, I agree so it's not really worth arguing further about semantics. However, I do think it would be humorous if at some point we say "we figured out MACC" when really all we did was dump all the complex variables we have no clue what to do with under "MEVA".

[Kaeko]

Sorry to spam this, but can everyone posting about this MACC debate post in the thread I made? I don't want to spam the random thread with this 1 debate. That goes for Pchan, myself, and the others.

[isladar]

Moved from RQT. Have fun here!

[Kaeko]

Moved from RQT. Have fun here!

Thanks Isladar!

[Cream Soda]

Pchan: "You Mad"ing bitches since 2007

[Kaeko]

To Pchan (or whoever else was involved in data collection):

(1) Qiqirn there vary from lvl75 to lvl77. Did you only nuke 1 level of these through check?
(2) Which nuke was it? I'm confused as to whether it was Stone I or Blizzard III.
(3) Overall, if you just summarized exactly what you did to collect the data I would greatly appreciate it.

[pchan]

But the "intervals" are coming out of your "theory" line, not the actual data point you collected (1k trials each). Confidence intervals come from your data points, not your theory.

This is very important because I could just make some huge ass squiggly line that perfectly fits your data and if I make the confidence intervals coming from my squiggly line, it will of course fit if I want it to. This is a hyperbole but I hope you get my point.

This is not important really. I checked my plot, and what I did is a translation of +/- 3% of the theorical landing rate. This is actually a wider confident interval that the one based on the data. The error for a sample size of 1k and a mean of p is sqrt(p*(1-p))*1.85/sqrt(1000) which is maximized for p=0.5 and gives 2.9%. Anything that has a different "mean" has lower error. Now you can argue that I used the empirical mean "p" in the error calculation instead of the empirical standard deviation/variance. In the end it's all about what you mean by "95% chance".

Basically, I think the graph needs to be adjusted and you really should manually calculate the confidence intervals and have them coming from the data. Seeing a chart where you have confidence intervals coming from data points that don't even exist just screams something is wrong.

It just means that the "true" confident interval is inside the interval of the picture.

As I mentioned a few posts back, I agree so it's not really worth arguing further about semantics. However, I do think it would be humorous if at some point we say "we figured out MACC" when really all we did was dump all the complex variables we have no clue what to do with under "MEVA".

I strongely believe that the only stats that appears in SE's algorithms is (meva-magic acc) anyway, that they do a spell by spell check, and that for every spell they add, they add + value if the mob is supposed to have darkness resistance (for instance) or remove some value if you merited ice accuracy. At the end of the check they have a final value for this number (which I interpret as meva - mac). If it is greater than 90 they give you 5% chance to land, if it's lower than - 45 they give you 95%. If it's between those values, apply the formula.

[Jeryhn]

To Pchan (or whoever else was involved in data collection):

(1) Qiqirn there vary from lvl75 to lvl77. Did you only nuke 1 level of these through check?
(2) Which nuke was it? I'm confused as to whether it was Stone I or Blizzard III.
(3) Overall, if you just summarized exactly what you did to collect the data I would greatly appreciate it.

Another question I'd like to see answered would include how much magic accuracy effects not just resists in general, but the different resist rates we encounter (1/2 effect, 1/4 effect, etc.).

Obviously, each spell is different from one another in terms of accuracy (see RDMs casting Stun on just about anything, of any level), and I'd suggest that the same elemental tiering concerning magic attack (Thunder is stronger than Ice, Ice is stronger than Fire, etc.) may have a parallel with elemental magic accuracy as well. This is why it might be important to test several different elements within the tiered black magic and their accuracy rates.

Could compare this data to AM2 spells, which, with equal merits, should have the same base attack power/accuracy.

[BRP]

No matter which party is right, that 'pedia page is a mess and should be cleaned up and made into an understandable piece of work.

[pchan]

To Pchan (or whoever else was involved in data collection):

(1) Qiqirn there vary from lvl75 to lvl77. Did you only nuke 1 level of these through check?
(2) Which nuke was it? I'm confused as to whether it was Stone I or Blizzard III.
(3) Overall, if you just summarized exactly what you did to collect the data I would greatly appreciate it.

(1) I've never seen an even match qiqrn in the ruins, they are all tough. They have 2 different levels, and 2 different INT.
(2) I use blizzard III every time I go there to add to the sample size. Of course I do this with the same gear every time, so that I'm sure that all have the same INT.
(3) After checking that they actually have the correct INT, that I do the test on non -thunder,wind, earth- day I spam stone I. I use K parser to register the values.

For each line, I note my base skill, my + mac from gear/merits, then the number of full damage spells, 1/2 resist etc. It's useless though because only the base rate matters, as the other resist rates are just a function of it. There is a column for the 95% error also and a column that calculates the total magic accuracy according to the model.

[Kaeko]

(1) I've never seen an even match qiqrn in the ruins, they are all tough. They have 2 different levels, and 2 different INT.
(2) I use blizzard III every time I go there to add to the sample size. Of course I do this with the same gear every time, so that I'm sure that all have the same INT.
(3) After checking that they actually have the correct INT, that I do the test on non -thunder,wind, earth- I spam stone I. I use K parser to register the values.

For each line, I note my base skill, my + mac from gear/merits, then the number of full damage spells, 1/2 resist etc. It's useless though because only the base rate matters, as the other resist rates are just a function of it. There is a column for the 95% error also and a column that calculates the total magic accuracy according to the model.

Hm I'm not seeing the confidence intervals on the excel file. Going from left to right, the columns I see are...

Skill, Macc, dINT, Full, Half, Fourth, More, BLANK, total, success%, "level(bliz3)nostaf", "wpn", and "critiques".

Also, regarding your graph and your arguments towards the validity of it, I'll try to get other people who know basic stats to see it, but to be honest, there's just something not right about someone giving me confidence intervals for data points that actually do not exist. I'm not actually that interested in your "conclusions" so much as the testing methodology though, so I'd rather not focus on that atm.

EDIT: I also saw that in your critique column, you note the EXP given by the mob. There are some that give 184XP and others that give 230XP, so you were nuking 2 different levels of the same type of mob. The INT may be the same, but the level isn't. How are you correcting or taking into account for this level difference.

[pchan]

Hm I'm not seeing the confidence intervals on the excel file. Going from left to right, the columns I see are...

Skill, Macc, dINT, Full, Half, Fourth, More, BLANK, total, success%, "level(bliz3)nostaf", "wpn", and "critiques".

Column S has the error
Column J has the landing rate

The confident interval at 95% is [S-J , S+J]

EDIT: I also saw that in your critique column, you note the EXP given by the mob. There are some that give 184XP and others that give 230XP, so you were nuking 2 different levels of the same type of mob. The INT may be the same, but the level isn't. How are you correcting or taking into account for this level difference.

Sanction, not sanction. Beseiged level 8 or not. The correct mobs give 230 xp once killed with beseiged level 8. I stopped checking the xp given once I was conviced that the qiqrns with the highest INT are of the highest level.

[Kaeko]

Column S has the error
Column J has the landing rate

The confident interval at 95% is [S-J , S+J]



Sanction, not sanction. Beseiged level 8 or not. The correct mobs give 230 xp once killed with beseiged level 8. I stopped checking the xp given once I was conviced that the qiqrns with the highest INT are of the highest level.

Ah sorry, didn't notice a bunch of columns after.

Your data collection seems pretty sound with all the things I can think of to try to poke holes at it, so I may try to crunch the data myself. I still disagree with you heavily about that graph and some other details so I may try to produce my own from your data set and compare.

Either way, from this point on, if we don't agree on a single other issue, I think this discussion was worth it because the data set seems sound. 40,000 trials of anything is significant (if done correctly).

[CDF]

I guess this is where I take a little responsibility for my statements and the fact that I did not even bother to make a grand summary of my posts concerning this topic, so here is my jar of pennies. Actually, this is a lot longer than I expected. I won't do TL;DR though. No, this won't be fodder for me to get on the soapbox that is my "lolblog"

1. Motivation for my magic accuracy modelette

As you saw already I described this "modelette" (again) in the "haste/accuracy" thread. I did not state my reasons for devising this though. Two reasons:

1) to reconcile the apparent phenomenon of having the effects of changes in skill/macc/INT be halved below the overall 50% "effective" magic accuracy (or magic hit rate, EMAC for now, apologies if you don't like my acronym) "threshold" and unaffected above 50% EMAC.

2) to explain the effect of HQ elemental staves crossing the 50% threshold. So this model has a basis, although I am not proving the model is true but forwarding an explanation for the data.

This threshold is meant to be independent of whatever spell you cast. I did not make any statement about every spell starting from a "base" EMAC, which seems to be a point of contention here.

The way I chose to explain this apparent phenomenon is to devise a hypothetical formula based on a difference of "magic accuracy" (MACC) and "magic evasion" (MEVA), both of which are meant to be catch-alls for positive and negative contributions to EMAC without worrying about specifics. 50% EMAC could then be a critical point where MACC - MEVA = 0. Where MACC - MEVA is negative, any changes in MACC (or MEVA) are cut in half.

Moreover, I continued to assume an additive model for EMAC, as opposed to a multiplicative one, where any distinct factor that affects EMAC has its own term and all the terms are added (or subtracted) from each other to obtain the final result. The additive model assumes that each factor is "independent" from one another and has no effect on the other factors. Of course, the additive model can break down pending demonstrable interaction among some of these factors. (Yet based on the data, the long-held idea of an interaction between staff effect and skill is not very credible.)

Examples of distinct factors, whether contrived or not, include level adjustment (correction), elemental resistance, resist traits, INT, etc. Moreover, for each factor, the rate of change of the effect of that factor (yes, unwiedly phrase) may be different depending on the range of delta(factor), as seems to be the case with INT and level adjustment. Note: this additiveness idea makes it easy to explain inherent differences in tiered spells, as I will describe later.

Also, I cite the melee analogue that is oriented about 75% hit rate, not as proof of the additive EMAC model but as an example where this threshold idea seems to be accepted.

2. Is a model for "effective" magic accuracy practical?

If the model(ette) is true, perhaps you can estimate an "effective" MEVA value for some mob of interest. This can be useful for hypothetical garden-variety testing of new equipment or abilities (uh, good luck finding more than a handful of people to do that with satisfactory sample sizes... credit to pchan) with unquantified EMAC (or MEVA) bonuses. If what we think is 391 total MACC (here, MACC is the catch-all) really corresponds to 50% EMAC for a specific mob that is somewhat convenient for testing (like a Qiqirn ranger), then we can avoid crossing this threshold and confounding our results.

But is the model true? Obviously, it is very tedious and difficult to validate any model of magic accuracy because we don't have convenient in-game tools like magic /check and SE is almost completely opaque about such things, not to mention any lurking variables unaccounted for in testing. So, I would not necessarily treat it even as provisionally true... yet. For the most part, most players are exclusively concerned with changes in so-called EMAC, so a comprehensive model has little interest for them anyway.

Then again, I never saw any validation of the melee hit rate "model," and pDIF information is/was treated as authoritative, too. But this may be based on comprehensive data or official information that I am not familiar with. (For example, whether pDIF came from a Japanese guide or was experimentally determined by the grassroots, I don't know.)

2(a). So what about the FFXIclopedia article?

First, this is an article about the magic accuracy trait, not EMAC. As such the article should be modified to focus on what players are most interested in, comparing the effect of magic accuracy to other attributes.

Unfortunately, I imagine very few people would be all that interested in a comprehensive description of an EMAC model, although pchan seems to be putting substantial effort into validation, so anything about modeling should not be included in this "summary" page.

Note: there is a separate article for melee hit rate, so you could put the more speculative stuff in the analogous "magic hit rate" article (or whatever terminology is eventually accepted).

------------------

Now, for some issues raised recently in this thread.

3. Accounting for differences in spell accuracy by tier

This seems to be another point of contention.

Assume differences by tier as true (based on comparison of Burst II and Thunder IV from a JP guide). Suppose we build on my my model(ette) without paying heed to the (lack of) soundness of the foundation. An easy way to reconcile spell tiers with the model is to conceive of an intrinsic MACC bonus, or MEVA penalty, based on tier level.

This does not mean there is a fixed EMAC oriented about spell type. So, different tiers of a "type" of spell can have independent MACC bonuses/penalties and there can still be an absolute difference in EMAC between tiers. You don't need to talk about a "base land rate" here.

Does my explanation not address "innate accuracy bonuses" among tiers? I mean, we see that AM II's magic accuracy increases with additional merits.

4. Use regression

This is regarding the generated graph. Of course, confidence intervals are based on the data. But, you don't need to generate confidence intervals for each point. Just use linear regression based on the assumption of piece-wise linearity where appropriate. Using this reasonable assumption will make the interval estimates of EMAC, given input level, more efficient than interval estimates that ignore the assumption. Then you can also check for violations of regression assumptions and lack of fit to the data.

5. Not controlling for Qiqirn level

Technically speaking, this is not very important assuming that the samples are balanced with respect to level differences (and assuming no other potentially confounding factors) because we are concerned with the change in EMAC with skill level. If that seems unpersuasive, think of randomization in clinical trials.

However, unbalanced samples will yield biased estimates due to level adjustment. For example, if you compare two samples, a biased estimate of difference will occur if you are comparing a sample heavily weighted with one level qiqirn to another sample heavily weighted with a different-level qiqirn, assuming level adjustment is in play. Something to keep in mind.

[Thorny]

sanction with capped besieged rank will turn a 184 xp mob into 211, just sayin'

[Kaeko]

I was hoping I could draw a response from you at some point. TY for the reply.

This threshold is meant to be independent of whatever spell you cast. I did not make any statement about every spell starting from a "base" EMAC, which seems to be a point of contention here.

The way I chose to explain this apparent phenomenon is to devise a hypothetical formula based on a difference of "magic accuracy" (MACC) and "magic evasion" (MEVA), both of which are meant to be catch-alls for positive and negative contributions to EMAC without worrying about specifics. 50% EMAC could then be a critical point where MACC - MEVA = 0. Where MACC - MEVA is negative, any changes in MACC (or MEVA) are cut in half.

I think we all agree at this point that "spells having innately different accuracies" and "mobs have different MEVA for different spells" really amounts to the same thing if we assume an additive model like you mention. You can either account for it in MEVA or in MACC. Since you're adding/subtracting it can be done at either step (assuming no multiplication is involved).

I think you address the same point in a different way here...

3. Accounting for differences in spell accuracy by tier

This seems to be another point of contention.

Assume differences by tier as true (based on comparison of Burst II and Thunder IV from a JP guide). Suppose we build on my my model(ette) without paying heed to the (lack of) soundness of the foundation. An easy way to reconcile spell tiers with the model is to conceive of an intrinsic MACC bonus, or MEVA penalty, based on tier level.

This does not mean there is a fixed EMAC oriented about spell type. So, different tiers of a "type" of spell can have independent MACC bonuses/penalties and there can still be an absolute difference in EMAC between tiers. You don't need to talk about a "base land rate" here.

Does my explanation not address "innate accuracy bonuses" among tiers? I mean, we see that AM II's magic accuracy increases with additional merits.

About the graph...

4. Use regression

This is regarding the generated graph. Of course, confidence intervals are based on the data. But, you don't need to generate confidence intervals for each point. Just use linear regression based on the assumption of piece-wise linearity where appropriate. Using this reasonable assumption will make the interval estimates of EMAC, given input level, more efficient than interval estimates that ignore the assumption. Then you can also check for violations of regression assumptions and lack of fit to the data.

I think the main sticking point for me is the "theory" line. Is the "theory" line the regression line, Pchan? As you mention, a linear regression would have been the way most people would have analyzed this I think, which was what I was going for. I just don't understand where all the confidence intervals coming from the "theory line" are coming from.

5. Not controlling for Qiqirn level

Technically speaking, this is not very important assuming that the samples are balanced with respect to level differences (and assuming no other potentially confounding factors) because we are concerned with the change in EMAC with skill level. If that seems unpersuasive, think of randomization in clinical trials.

However, unbalanced samples will yield biased estimates due to level adjustment. For example, if you compare two samples, a biased estimate of difference will occur if you are comparing a sample heavily weighted with one level qiqirn to another sample heavily weighted with a different-level qiqirn, assuming level adjustment is in play. Something to keep in mind.

I think based on his description of his collection methods, he was able to get mobs of the same level. Check for T (to eliminate 75 mobs), then nuke to get the correct amount of INT (higher INT is 77), then test on LVL77. So his set should be lvl77 mobs if I'm not mistaken. So he has this variable controlled well assuming this is accurate.

EDIT: Based on what Thorny says, I don't know now. As Robonosto mentioned, the difference should be small given that Pchan nuked relatively equal numbers of 76 vs 77 mobs, but would be nice to have a 'perfectly executed' data set of 40,000.