All right, so I checked some of the "Stone" sheet today just to check a few things.
1. Appropriateness of the additive EMAC model below 50%
I used rows 3-17 to check for a linear relationship between the computed MACC score (comprised of delta(INT), elemental magic skill, magic accuracy, and staff bonus) and EMAC below 50% and got the following results.
Code:
Data Set WORK.QIQIRN
Distribution Binomial
Link Function Identity
Response Variable (Events) success
Response Variable (Trials) total
Number of Observations Read 28
Number of Observations Used 15
Number of Events 4579
Number of Trials 12227
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 13 11.9312 0.9178
Pearson Chi-Square 13 11.9303 0.9177
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -0.9692 0.1050 -1.1750 -0.7633 85.12 <.0001
macc_score 1 0.0051 0.0004 0.0043 0.0058 162.95 <.0001
For this data, delta(INT) was less than or equal to 10. The estimate for the MACC score is consistent with the idea that any bonuses (or penalties) are cut in half below 50% EMAC.
The linearity assumption appears valid (as we would hope). There could be problems if level adjustment came into play.
2. Appropriateness of the additive model, broken out by specific factors
Since we are interested in model checking, we can break down the MACC score by its constituent parts and see how EMAC varies with changes in each.
Code:
Data Set WORK.QIQIRN2
Distribution Binomial
Link Function Identity
Response Variable (Events) success
Response Variable (Trials) total
Number of Observations Read 15
Number of Observations Used 15
Number of Events 4579
Number of Trials 12227
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 11 11.0852 1.0077
Pearson Chi-Square 11 11.0933 1.0085
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -0.0879 0.0869 -0.2581 0.0823 1.02 0.3115
macc 1 0.0049 0.0005 0.0039 0.0058 108.91 <.0001
int 1 0.0057 0.0013 0.0031 0.0083 18.05 <.0001
staff 1 1 0.1600 0.0164 0.1278 0.1922 94.70 <.0001
The parameter estimates are consistent with previous work based on EMAC being below 50%. The HQ staff bonus is around +15% EMAC and the magic accuracy effect is about +0.05% per 1-point increase.
Recall that lodeguy did not check for the effect of INT below 50% EMAC. Here, the effect of INT appears to be halved as well.
3. Estimating observed magic evasion
You could either obtain individual MEVA estimates for each value of MACC score or base MEVA estimates on the assumption of a linear relationship between MACC score and MEVA (based on the regression model and the estimates of slope and intercept).
In either case, this involves using the estimate of the true EMAC value for a given MACC score. (These estimates will be different depending on which approach you take.)
I noticed that the interval estimates for MEVA (pchan's approach) are 93.57% confidence intervals (z-score of 1.85 used). Why did you use that? At any rate, the 95% confidence intervals are slightly wider.
The other approach uses the information from all other samples (summarized in the slope and intercept estimates) to provide narrower interval estimates, so they are generally more efficient than estimates treated in "isolation." Here are the results using this approach:
Code:
Score E^[Y|X] E^[MEVA|X] CI lower CI upper
---------------------------------------------------------
240 0.24777 290.446 286.308 294.584
246 0.27819 290.362 287.054 293.67
251 0.30354 290.292 287.622 292.962
256 0.3289 290.22 288.09 292.35
261 0.35425 290.15 288.376 291.924
265 0.37453 290.094 288.39 291.798
266 0.3796 290.08 288.358 291.802
268 0.38974 290.052 288.254 291.85
270 0.39988 290.024 288.102 291.946
271 0.40495 290.01 288.01 292.01
274 0.42017 289.966 287.684 292.248
276 0.43031 289.938 287.436 292.44
277 0.43538 289.924 287.306 292.542
This is what I meant when I said "use regression."
4. Appropriateness of the additive EMAC model above 50%, broken out by specific factors
I am focusing on rows 75-86 here. This data is more problematic because delta(INT) exceeds 10 for some samples. But, if I recall correctly, INT remains the same regardless of Qiqirn level. Also, there appear to be two different levels of qiqirn involved. In particular, rows 75-78 are based exclusively on a single qiqirn that gave 230 EXP. I assume the other qiqirn were of a different level.
We think that the effect of INT is halved for delta(INT) >= 10, so let's see if that's the case here by excluding the two samples with INT = 65.
Code:
Number of Observations Read 10
Number of Observations Used 10
Number of Events 6571
Number of Trials 9790
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 5 7.5070 1.5014
Pearson Chi-Square 5 7.4390 1.4878
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 0.2456 0.0683 0.1118 0.3793 12.94 0.0003
macc 1 0.0112 0.0007 0.0098 0.0127 231.28 <.0001
int 1 0.0034 0.0007 0.0021 0.0048 24.76 <.0001
staff 1 1 0.2972 0.0217 0.2546 0.3398 186.92 <.0001
level 1 1 0.0063 0.0168 -0.0266 0.0392 0.14 0.7067
There are some borderline cases here from the standpoint of statistical significance, such as macc and INT. Checking the residuals, the Alkalurops data (row 84) seem "extreme." Interestingly, the level adjustment effect is not close to significance (are rows 76-78 for a different level qiqirn than the other rows?).
Of course, the identification of delta(INT) = 10 as a critical point was done informally. Let's try applying the model for samples where delta(INT) <= 10. (I removed the level effect from the model.)
Code:
Number of Observations Read 5
Number of Observations Used 5
Number of Events 2725
Number of Trials 4424
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 1 0.4115 0.4115
Pearson Chi-Square 1 0.4118 0.4118
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -0.4354 0.1674 -0.7635 -0.1074 6.77 0.0093
macc 1 0.0088 0.0019 0.0051 0.0125 21.97 <.0001
int 1 0.0127 0.0022 0.0085 0.0170 34.09 <.0001
staff 1 1 0.2833 0.0190 0.2460 0.3205 222.08 <.0001
INT is indeed more efficient below delta(INT) 10 than above.
5. Appropriateness of the additive EMAC model above 50% based on MACC score
Using the MACC scores, there is a statistically significant lack of fit to all twelve samples (p-value .0026... compared to the saturated model) even accounting for level difference (p-value .0701).
So, it may be worth checking model fit to the data where delta(INT) <= 10 and where delta(INT) >= 10. Start with the first case where delta(INT) <= 10:
Code:
Number of Observations Read 5
Number of Observations Used 5
Number of Events 2725
Number of Trials 4424
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 2 4.5021 2.2510
Pearson Chi-Square 2 4.5427 2.2713
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -2.5469 0.1985 -2.9360 -2.1578 164.60 <.0001
macc_score 1 0.0104 0.0006 0.0092 0.0117 268.48 <.0001
level 1 1 0.0136 0.0162 -0.0181 0.0454 0.71 0.3995
And for the second case with delta(INT) >= 10:
Code:
Number of Observations Read 10
Number of Observations Used 10
Number of Events 6571
Number of Trials 9790
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 7 20.7036 2.9577
Pearson Chi-Square 7 20.4193 2.9170
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -2.7372 0.2020 -3.1331 -2.3413 183.64 <.0001
macc_score 1 0.0110 0.0006 0.0098 0.0123 294.82 <.0001
level 1 1 0.0335 0.0148 0.0044 0.0626 5.10 0.0240
While superficially the MACC score is not really different in either case, there is a lack of fit to the data given delta(INT) >= 10. Checking the residuals, row 84 (Alkalurops) stands out like a sore thumb (std. residual above 3). Removing the Alkalurops data gives the following result:
Code:
Number of Observations Read 9
Number of Observations Used 9
Number of Events 5760
Number of Trials 8785
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 6 8.7710 1.4618
Pearson Chi-Square 6 8.7827 1.4638
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter DF Estimate Error Limits Square Pr > ChiSq
Intercept 1 -2.6206 0.2091 -3.0303 -2.2109 157.14 <.0001
macc_score 1 0.0106 0.0007 0.0093 0.0119 254.43 <.0001
level 1 1 0.0349 0.0149 0.0056 0.0642 5.46 0.0194
6. Conclusion
The so-called additive EMAC model looks pretty good below 50% EMAC. It seems ok above 50% EMAC with some caveats about level correction, INT, and Alkalurops. At any rate, the data support this sharp dichotomy oriented about 50% EMAC.
Here, I didn't provide an explanation for why the Alkalurops data seem unusual relative to the rest of the data (above 50% EMAC). I mean, the MEVA estimates seem consistent (around 290). I would have to look at it later.
If you think I am looking at the data incorrectly, please provide an explanation. I already checked to see if I made any input errors.