Kaeko demonstrated it works in the way described as "Yutuyu's equation" above, in the 4th installment on his blog: http://kanican.livejournal.com/14209.html
Code:
Results
Enmity Change CE Change
-10 -32
-5 -16
-2 -7
-1 -4
0 0
+1 +3
+2 +6
+5 +16
+10 +32
Were it calculated as Ichthyos suggested, it would have been a CE change of 29 in the enmity-10 case.
Also, to what maxx was asking before. The general rule for equations used in FFXI is that you get accelerating returns when the formula to determine the bonus is "total = 1 - change" (haste, -enmity, damage taken/shell, etc.), and diminishing returns when the formula is of the form "total = 1 + change" (MAB, MDB, +enmity, DA, etc.)
The interesting part is that calculations of the first sort usually have some kind of cap on the individual terms, so if the game actually lets you get to -100 or -99 enmity, that would be interesting indeed. I know the cap on +enmity is +100, but has it been confirmed that the cap on -enmity isn't -50 or thereabouts?