DAT Mining DISCUSSION- May 15, 2012

[Septimus]

This patch seems to have broken the PoLUtils Magic/Ability DAT. (ROM/118/114.DAT) Getting a "There are no views available for this file" error when I try to open it. Help!

[Radec]

The first 1-3 padding bytes now appear to have information in them, which is causing polutils to pitch a fit, since after the expected data size, there should only be 0x00's until the 0xFF that signifies the end of the entry - and now there are at least 3 bytes of new data.

I'm not sure if the information is meaningful or not yet, but at the very least I can say the new and old 1-118-114.dat files have the exact same number of lines, and the spells names, ability names files didn't get an update today. Working on a fix but it'll probably be sometime tomorrow at the *very* earliest, blame diablo.

EDIT: yeah, nothing coming tomorrow, see below, more complicated than I had expected.

[Khamsin]

For reference, here's what the first five spell data lines (Cure I-V) were:

Format: [Field: Byte Size]

[Index: 2] [Magic Type: 2] [Element: 2] [Valid Targets: 2] [Skill: 2] [MP Cost: 2] [Cast Time: 1] [Recast Time: 1] [Job Levels: 24] [ID: 2] [Unknown: 2]

01 00 01 00 06 00 3F 00 21 00 08 00 08 14 FF FF FF 01 FF 03 FF 05 FF FF FF FF FF FF FF FF FF FF FF FF 05 FF FF FF 01 00 06 01 
02 00 01 00 06 00 3F 00 21 00 18 00 09 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 01 
03 00 01 00 06 00 3F 00 21 00 2E 00 0A 18 FF FF FF 15 FF 1A FF 1E FF FF FF FF FF FF FF FF FF FF FF FF 1E FF FF FF 03 00 06 01 
04 00 01 00 06 00 3F 00 21 00 58 00 0A 20 FF FF FF 29 FF 30 FF 37 FF FF FF FF FF FF FF FF FF FF FF FF 37 FF FF FF 04 00 06 01 
05 00 01 00 06 00 3F 00 21 00 87 00 0A 28 FF FF FF 3D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 05 00 06 00

This is what it is now:

01 00 04 00 06 00 3F 00 21 00 08 00 20 14 FF FF FF 01 FF 03 FF 05 FF FF FF FF FF FF FF FF FF FF FF FF 05 FF FF FF 01 00 06 00 06 00 01 00 
02 00 40 00 06 00 3F 00 21 00 18 00 42 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 00 FF FF 01 00 
03 00 40 00 06 00 3F 00 21 00 2E 00 82 18 FF FF FF 15 FF 1A FF 1E FF FF FF FF FF FF FF FF FF FF FF FF 1E FF FF FF 03 00 06 00 FF FF 01 00 
04 00 40 00 06 00 3F 00 21 00 58 00 82 20 FF FF FF 29 FF 30 FF 37 FF FF FF FF FF FF FF FF FF FF FF FF 37 FF FF FF 04 00 06 00 FF FF 01 00 
05 00 40 00 06 00 3F 00 21 00 87 00 82 28 FF FF FF 3D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 05 00 06 00 FF FF 00 00

The new version has an extra 3-4 bytes at the end, which I included, that aren't in the old version.

So it appears they changed the Magic Type and Cast Time fields somehow, or at least, rearranged them or changed how they interpreted the data. Magic Type used to be:

1 White Magic
2 Black Magic
3 Summoning
4 Ninjutsu
5 Songs
6 Blue Magic

And Cast Time was the casting time in 1/4 seconds (i.e. 4 = 1 second).

Now the Cast Time looks like it might be some other conversion factor.

2 seconds: 8 -> 32
2.25 seconds: 9 -> 66
2.5 seconds: 10 -> 130

I don't know what to say about Magic Type yet. I'll keep looking into it.

[Radec]

What program are you using to view the hex? The one I grabbed doesn't like the endianness of the ffxidats and it's a hassle to convert them by hand. Figuring out the new codes won't be awful if I can get a way to read them all as

01 00 04 00 06 00 3F 00 21 00 08 00 20 14 FF FF FF 01 FF 03 FF 05 FF FF FF FF FF FF FF FF FF FF FF FF 05 FF FF FF 01 00 06 00 06 00 01 00 
02 00 40 00 06 00 3F 00 21 00 18 00 42 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 00 FF FF 01 00 
03 00 40 00 06 00 3F 00 21 00 2E 00 82 18 FF FF FF 15 FF 1A FF 1E FF FF FF FF FF FF FF FF FF FF FF FF 1E FF FF FF 03 00 06 00 FF FF 01 00 
04 00 40 00 06 00 3F 00 21 00 58 00 82 20 FF FF FF 29 FF 30 FF 37 FF FF FF FF FF FF FF FF FF FF FF FF 37 FF FF FF 04 00 06 00 FF FF 01 00 
05 00 40 00 06 00 3F 00 21 00 87 00 82 28 FF FF FF 3D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 05 00 06 00 FF FF 00 00

[Khamsin]

What program are you using to view the hex? The one I grabbed doesn't like the endianness of the ffxidats and it's a hassle to convert them by hand. Figuring out the new codes won't be awful if I can get a way to read them all as

01 00 04 00 06 00 3F 00 21 00 08 00 20 14 FF FF FF 01 FF 03 FF 05 FF FF FF FF FF FF FF FF FF FF FF FF 05 FF FF FF 01 00 06 00 06 00 01 00 
02 00 40 00 06 00 3F 00 21 00 18 00 42 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 00 FF FF 01 00 
03 00 40 00 06 00 3F 00 21 00 2E 00 82 18 FF FF FF 15 FF 1A FF 1E FF FF FF FF FF FF FF FF FF FF FF FF 1E FF FF FF 03 00 06 00 FF FF 01 00 
04 00 40 00 06 00 3F 00 21 00 58 00 82 20 FF FF FF 29 FF 30 FF 37 FF FF FF FF FF FF FF FF FF FF FF FF 37 FF FF FF 04 00 06 00 FF FF 01 00 
05 00 40 00 06 00 3F 00 21 00 87 00 82 28 FF FF FF 3D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 05 00 06 00 FF FF 00 00

One I wrote myself back when Aikar was asking people to help find the new spell/abils dat. Speaking of endianness, that gave me an idea to examine those new cast time codes in binary:

00100000 = 32
01000010 = 66
10000010 = 130

However, if you swap them around:

00000010 = 2
00100100 = 36
00101000 = 40

36 = the proper cast time in 1/16 seconds, and 40 is the proper cast time in 1/16 seconds. Of course, the 2 is an anomaly, however, the non-flipped 32 is the proper cast time in 1/16 seconds.

Or, more simply, swapping the hex digits... 42 become 24, and 82 become 28.

[Khamsin]

So I pulled another sample from the file, these correspond to Phalanx, Phalanx II, Regen, Refresh, Regen II:

Old:

6A 00 01 00 06 00 01 00 22 00 15 00 0C 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 01 00 00 00 00 
6B 00 01 00 06 00 05 00 22 00 2A 00 0C 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 00 00 00 00 
6C 00 01 00 06 00 05 00 22 00 0F 00 06 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 01 00 00 00 00 
6D 00 01 00 06 00 05 00 22 00 28 00 14 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 01 00 00 00 00 
6E 00 01 00 06 00 05 00 22 00 24 00 07 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 01 00 00 00 00

New:

6A 00 40 00 06 00 01 00 22 00 15 00 03 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 00 FF FF 01 00 
6B 00 40 00 06 00 05 00 22 00 2A 00 03 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 FF FF 00 00 
6C 00 40 00 06 00 05 00 22 00 0F 00 81 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 00 FF FF 01 00 
6D 00 40 00 06 00 05 00 22 00 28 00 05 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 00 FF FF 01 00 
6E 00 08 00 06 00 05 00 22 00 24 00 38 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 00 FF FF 01 00

The cast times according to the old version should be 3, 3, 1.5, 5, 1.75... corresponding to the new cast time codes of 03, 03, 81, 05, 38.

The 03s work out, when swapped, to be the proper cast time in 1/16th seconds, as does the 05. The 81, when swapped, works out to the cast time in 1/16th seconds. The 38, when not swapped, works out to the cast time in 1/32nd seconds.

So something weird seems to be going on, it can't be changing it's rules all the time. Maybe there's some bitwise operations going on, or shifting of some sort, or something else entirely.

[Selamis]

Possible wrong section, not sure since coding for cure stuff discussed above could be related to macro coding? Can move if need be.

Is anyone else having a lag now when hitting macros after the adjustment to the macro bars? I had to spam provoke and my utsusemi ni macro a few times because hitting it once now, doesn't make it fire right away. It wouldn't have been a pc lag issue either I don't think, happened with two other people around me, in the middle of La Theine and Tahrongi where I've never had lag spikes before. Or am I just imagining things with the macros?

[Septimus]

This sounds horrifically confusing. Thank you for looking into it.

[Radec]

So I pulled another sample from the file, these correspond to Phalanx, Phalanx II, Regen, Refresh, Regen II:

Old:

6A 00 01 00 06 00 01 00 22 00 15 00 0C 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 01 00 00 00 00 
6B 00 01 00 06 00 05 00 22 00 2A 00 0C 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 00 00 00 00 
6C 00 01 00 06 00 05 00 22 00 0F 00 06 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 01 00 00 00 00 
6D 00 01 00 06 00 05 00 22 00 28 00 14 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 01 00 00 00 00 
6E 00 01 00 06 00 05 00 22 00 24 00 07 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 01 00 00 00 00

New:

6A 00 40 00 06 00 01 00 22 00 15 00 03 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 00 FF FF 01 00 
6B 00 40 00 06 00 05 00 22 00 2A 00 03 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 FF FF 00 00 
6C 00 40 00 06 00 05 00 22 00 0F 00 81 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 00 FF FF 01 00 
6D 00 40 00 06 00 05 00 22 00 28 00 05 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 00 FF FF 01 00 
6E 00 08 00 06 00 05 00 22 00 24 00 38 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 00 FF FF 01 00

The cast times according to the old version should be 3, 3, 1.5, 5, 1.75... corresponding to the new cast time codes of 03, 03, 81, 05, 38.

The 03s work out, when swapped, to be the proper cast time in 1/16th seconds, as does the 05. The 81, when swapped, works out to the cast time in 1/16th seconds. The 38, when not swapped, works out to the cast time in 1/32nd seconds.

So something weird seems to be going on, it can't be changing it's rules all the time. Maybe there's some bitwise operations going on, or shifting of some sort, or something else entirely.

Maybe something with the new bytes can show it? I don't see anything really in that snip though that would be a flag except maybe the 3rd byte being reversed as well.

[Churchill]

Possible wrong section, not sure since coding for cure stuff discussed above could be related to macro coding? Can move if need be.

Is anyone else having a lag now when hitting macros after the adjustment to the macro bars? I had to spam provoke and my utsusemi ni macro a few times because hitting it once now, doesn't make it fire right away. It wouldn't have been a pc lag issue either I don't think, happened with two other people around me, in the middle of La Theine and Tahrongi where I've never had lag spikes before. Or am I just imagining things with the macros?

Yes, the menu comes up instantly, the macro takes 2-4 presses to actually fire.

[Pandorablue]

Anyone else having an issue with crashing? Both of my two boxes seem to be crashing a lot after the use of job abilities (really annoying, I JUST entered dynamis to find this out). I'm testing out a bunch of different plugins to see if it's any, but so far no luck. I'm running a file check right now, but while I wait, anyone else getting this?

[Khamsin]

Maybe something with the new bytes can show it? I don't see anything really in that snip though that would be a flag except maybe the 3rd byte being reversed as well.

I have a feeling it's simpler than that. Some bit operation we're overlooking, maybe.

1.50 seconds = 81 = 10000001
1.75 seconds = 38 = 00111000
2.00 seconds = 20 = 00100000
2.25 seconds = 42 = 01000010
2.50 seconds = 82 = 10000010
3.00 seconds = 03 = 00000011
5.00 seconds = 05 = 00000101

There's a pattern in there, somewhere!

[bungiefan]

Might be an issue with using XI-View. They changed the icon file for status effects, to use 32x32 icons instead of the old 16x16, and it jumbles the icons around if you apply that UI mod. I'm trying to dig up the old file right now to go back. Apparently it crashes some people when certain JA icons load. I haven't had it happen yet, but I'm expecting it.

[Khamsin]

So further examination of the dat makes it seem like some of the bytes are not meant to be bit-shifted per the algorithm that decodes the rest of the data.

For example:

Cure II

Pre-Update
Raw data:
08 00 04 00 18 00 FC 00 84 00 60 00 24 58 FF FF FF 2C FF 38 FF 44 FF FF FF FF FF FF FF FF FF FF FF FF 44 FF FF FF 08 00 18 04 00

Decoded data:
02 00 01 00 06 00 3F 00 21 00 18 00 09 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 01 00

Post-Update
Raw data:
08 00 01 00 18 00 FC 00 84 00 60 00 09 58 FF FF FF 2C FF 38 FF 44 FF FF FF FF FF FF FF FF FF FF FF FF 44 FF FF FF 08 00 18 00 FF FF 04 00

Decoded data (incorrect):
02 00 40 00 06 00 3F 00 21 00 18 00 42 16 FF FF FF 0B FF 0E FF 11 FF FF FF FF FF FF FF FF FF FF FF FF 11 FF FF FF 02 00 06 00 FF FF 01 00

Note the red numbers. Pre-update, the red numbers were decoded to get the correct values. e.g. 04 got decoded to 01, and 24 was decoded to 09. 01 and 09 are the correct values.

Now look at the post-update raw data. 01 and 09 are already there, no bit-shifting required. Those values in the post-update decoded data are incorrect, because they've been bit-shifted when they shouldn't have been. Those bytes should be left alone.

Perhaps not-so-coincidentally, those are the same bytes (along with the 12th byte) that are used to determine the bit-shift used to decode that line. So, in summary, all we need to do is update our decoding algorithm to not bit-shift the 3rd, 12th and 13th bytes. Then our data should be correct.

[Byrthnoth]

Why would SE have chosen to change this?

[Radec]

Probably has to do with how the encryption is handled, see http://code.google.com/p/polutils/wi...FileEncryption - if you don't rotate the 2 12 13's, it's easier to reverse it later.

[Khamsin]

So now, with a modified algorithm, we get:

Old:

6A 00 01 00 06 00 01 00 22 00 15 00 0C 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 01 00 00 00 00 
6B 00 01 00 06 00 05 00 22 00 2A 00 0C 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 00 00 00 00 
6C 00 01 00 06 00 05 00 22 00 0F 00 06 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 01 00 00 00 00 
6D 00 01 00 06 00 05 00 22 00 28 00 14 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 01 00 00 00 00 
6E 00 01 00 06 00 05 00 22 00 24 00 07 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 01 00 00 00 00

New:

6A 00 01 00 06 00 01 00 22 00 15 00 0C 28 FF FF FF FF FF 21 FF 4D FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 69 00 06 00 FF FF 01 00 
6B 00 01 00 06 00 05 00 22 00 2A 00 0C 28 FF FF FF FF FF 4B FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6A 00 06 00 FF FF 00 00 
6C 00 01 00 06 00 05 00 22 00 0F 00 06 30 FF FF FF 15 FF 15 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 12 FF FF FF 6B 00 06 00 FF FF 01 00 
6D 00 01 00 06 00 05 00 22 00 28 00 14 48 FF FF FF FF FF 29 FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF FF 6C 00 06 00 FF FF 01 00 
6E 00 01 00 06 00 05 00 22 00 24 00 07 40 FF FF FF 2C FF 4C FF FF FF FF FF FF FF FF FF FF FF FF FF FF 25 FF FF FF 6D 00 06 00 FF FF 01 00

Everything matches up, and we're all good.

Except for a little rearranging at the end (they inserted a "00 FF FF" in the middle of the "06 01" part toward the end), but none of that data is data that we currently use for anything, anyway.

Now to see if they changed anything in the abilities section...

Probably has to do with how the encryption is handled, see http://code.google.com/p/polutils/wi...FileEncryption - if you don't rotate the 2 12 13's, it's easier to reverse it later.

Makes sense, but before this update, they bit-shifted the bytes used to determine the bit-shift. So why change it now? It was working fine before.

Besides, given how the algorithm works, it doesn't matter how much you bit shift those bytes, the number of bits they have will never change, meaning that it's just as easy to reverse it later as if they hadn't bit-shifted at all.

[Khamsin]

So lastly, the same change needs to be made for the abilities (do not shift 3rd, 12th and 13th bytes), and they also inserted two bytes between the icon ID and the MP Cost.

So...

[Field: Byte Size]

Old:

[ID: 2] [Type: 1] [Icon ID: 1] [MP Cost: 2] [Unknown: 2] [Valid Targets: 2]

New:

[ID: 2] [Type: 1] [Icon ID: 1] [Unknown: 2] [MP Cost: 2] [Unknown: 2] [Valid Targets: 2]

Just make that change, and the bit-shifting thing, and you're set.

[TheDangerousDave]

The new Limbus gear and augments to Homam wasn't this patch?

[Selamis]

Yes, the menu comes up instantly, the macro takes 2-4 presses to actually fire.

I'd rather it go back to the old way if this is the case lol, it's rather annoying. Glad it's not just me having that happen though.