Calling all Programmers/Hackers/Modders... (spells and abilities found)

[Aikar]

As you all may know, even after the resources update spellcast and recast are STILL having issues.

It's because ALL data from resources files are generated through the program POLUtils which exports .dat files to raw XML, which I then process into a much cleaner files as we have them today.

However, sometime in year 2005 or 2006 SE switched to a new DAT file that holds "Common" data for Spells and Abilities between the 4 languages.

So the files are setup like this:

Spells_Common
Spells_English
Spells_Japanese
Spells_French
Spells_German

same for abilities.

The 4 languages files never changed, and still work, however, the common file changed to a different #.

These 2 files (1 for abils, 1 for spells) has never been found since the change in 2005/2006, and we have been using the old one.

Well what this means is all new spells/abils since then have been missing information/had incorrect information.

We remedied this by creating the "fixes.xml" as noted on the Resources page on wiki. This let me correct data on the generation script to be the right data.
This method was fine until now because new spells and abilities were rare and only in small numbers when they did come, so we had people find the new ID's, we added em, game done.

But now with level cap rise we are getting so much new things... The "junk data" in the old common file is not matching up at all with the new stuff, so we are getting /song prefixes for spells, 0mp cost, etc. Stuff is breaking

We desperately need to find these new common files...

So I call on anyone who is able to please, download POLUtils and whatever you know how to, and crack at the .dat's and try to find the new common files...

Look at http://resources.windower.net/rawdata/Spells_common.xml and http://resources.windower.net/rawdat...ies_common.xml
for an example on what kind of data we need to find...

The new files should contain very similiar data, but would contain all the new data.

For example, Stoneja is ID 499, and the current common spells file lists Singing skill and /song for prefix.
The new one should list /magic and BlackMagic for ID 499.

I encourage people to work together on it, maybe start a thread in the DAT Forums and post findings/thoughts.

Even though I no longer play FFXI, I want to try to keep everything running for Windower 3.4. So please help me with this huge task that everyone will benefit from.

Thanks guys.

[Klaatu]

Just wanted to say thank you for your effort Aikar. Been a long time user and your work is greatly appreciated. Thanks.

[rog]

Is there any way you can narrow down when it was changed?

[rog]

The .dats you're using now were both updated last in 2007, are you positive they haven't been used since 2005-2006?

[rog]

Eh, the last spell file was 768 kb, searching all of the files in rom\ updated this last update that are >700 kb shows 410 results, so i'm just going to go through them all until i find it...

This should be fun.

[rog]

It's either not between 700kb-1 mb, or it's somewhere in rom22-24, which has a number of files that polutils can't open. I also checked every single file between 1.3.-2mb, in case they combined the files, and nada (32\58 being unopenable.).

99% of the files >500 kb are dialog tables, and basically all of the rest can't be opened by polutils.

I'm out of ideas.

Or it wasn't updated this update at all, in which case we're going to need to locate it by when it was originally added to the dats. Though the 2 year range you gave isn't very helpful for that...

[Akairyu]

I've unfortunately had no great luck searching through the dat files in ROM/ for some of the text strings that should be there if the format of the file matches the old ones we know about. The XSearch program I was trying to use just isn't turning anything up, even when I tried putting in search parameters that should have brought up the old files, so that seems to be a bit of a dead end. The only alternative I can think of is brute force checking every dat file manually which is going to be a nightmare. I suppose we could also consider the possibility that the spell information files have been split up into various smaller files as well, and possibly buried in the expansion ROM folders but that doesn't seem to fit with SE's file structure. In any case, I'll keep picking at this problem when I get another brainwave.

[rog]

Well we know it should be in rom\, and it should have been updated in the last update, which narrows it down to at very most 1700 files, which of course is still a huge amount, but we can rule more than half of them out, because we know like 177-180, etc is all dialog tables, etc

Unfortunately polutils can't read a ton of these files, so it's possible the ones we're looking for can't either.

[rog]

181\72 has all the spells/abilities listed in it, but not the other info.

[SnickySnacks]

Obvious questions:
What DAT was the 2005/2006 file?
Is there a copy of the 2005/2006 file somewhere we can look at?
Is there a spec sheet of the .dat formats that we are able to read?

Spent 10 friggin' years doing customer data translation, often without solid specs, .dats can't possibly be that hard to figure out.

[Eole]

If you use POLUtils to visualize DAT file, you have to know it uses a method taken from ffxitool to decrypt the DATs.

DAT Encryption - POLUtils Forum

This encryption might have changed for some reason for this file, and the file might have been moved. POLUtils might be unable to decrypt the DAT because of this.

[Sekundes]

If you need an old version of the dats for referencing changes, wouldn't installing from the various physical discs work? ie, the original install discs? I have a decent amount of them in various forms. Ps2 and pc both and such. If someone knows what they need I could upload a copy of the disc which might help.

[rog]

Nah, the old versions are still there. It'd help if we knew exactly when the new dat was first used, because they just put the dats in order of when they're made, so if we knew an exact date of when it was first used, we could find which folder(s) it'd be in, and then search those folders for files updated on september 8th, and then manually look through the remaining files, which would likely only be 1-3 dats at most, and definitely no more than 20-30.

Though i've already looked through almost all of the updated dats that it could possibly be, and didn't find it, so i'm thinking polutils can't read it.

[Shalafi]

Seems too simple, but afaik they introduced entirely new spells (didn't provide access to existing spells); wouldn't searching for those be the best starting point?

I guess the answer would be that the new spells could've been added to these new files at anytime, we just didn't know it?

Therefore, the number of files that need to be searched is not reduced?

[rog]

Seems too simple, but afaik they introduced entirely new spells (didn't provide access to existing spells); wouldn't searching for those be the best starting point?

I guess the answer would be that the new spells could've been added to these new files at anytime, we just didn't know it?

Therefore, the number of files that need to be searched is not reduced?

The job requirements were changed, so the file had to have been to.

[Khamsin]

Starting from scratch here, so kinda figuring things out as I go regarding the structure of the old spell dat, but so far as I can tell, it appears to be:

Index (2 bytes)
Magic Type (2 bytes)
Element (2 bytes)
Valid Targets (2 bytes, probably based on which bits are set)
Skill (2 bytes)
MP Cost (2 bytes)
Cast Time (1 byte) (in quarter-seconds, e.g. 9 = 9/4 = 2.25s)
Recast Time (1 byte)
Job Levels (1 byte each, 24 bytes worth altogether, so some must be unused. Order appears to be: ??? ??? ??? WHM BLM RDM ??? PLD DRK ??? BRD ??? ??? NIN ??? SMN BLU ??? ??? ??? ??? ??? ??? ???)
ID (2 bytes)
the Unknown column from POLUtils (2 bytes)

...with a bunch of white space between records. Each record starts at a multiple of 1024.

By the way, for anyone trying to write their own decoding routine, it appears the shifted bits need to wrap around. Was trying it myself last night, and some of the data blocks didn't decode correctly until I realized that. Plus, even in the 2 byte length values, each individual byte needs to be bit shifted separately, however you'll still need to string them back together again to form the complete 2 byte value.

My intention is, once I can comfortably decode the old spell dat, to then write something that'll scan each .dat and rotate the bits of each data block 1 through 7 times and search the result for something we know.

e.g. pick a dat, go through each data block, rotate its bits once, see if it matches the values for a new spell, rotate its bits again, check again, up to 7 times, then move onto the next data block and repeat.

It's a bit intensive, but if all SE did for the new dat was to use a different algorithm for determining how many bits to rotate, this should find it. If, however, they did something completely different, then I give up. lol

[SnickySnacks]

I'm curious if these are still in the 1024 byte record blocks that they used to. I did a pretty simple scan that should have found anything with the old ability or spell format regardless of any bit rotations and wasn't able to find it in any file updated after August 1st.

Made a dumb mistake, will have to rescan these again. Will keep you posted.

I'm thinking my next step will be to just log every dat file that gets opened by FFXI as it starts up and then see if any of those are new and unreadable.

[Eole]

It's a bit intensive, but if all SE did for the new dat was to use a different algorithm for determining how many bits to rotate, this should find it. If, however, they did something completely different, then I give up. lol

Yes, that is exactly the right thing to do!
Let us know how it goes!

[Khamsin]

Yes, that is exactly the right thing to do!
Let us know how it goes!

Well, I wrote a scanner that goes through each dat, skipping files it identifies as image files or dialog tables or files above 5kb, reading the first 42 bytes of every 1024 byte block, comparing it to known spell values, and then shifting the bits once and checking again, up to 7 times.

It found matches in 0/11.dat and 119/56.dat, which we knew about already. But nothing else. I also opened all the dats that have been updated since March 2009 (last time the old spell dats were updated) in UltraEdit32 about 20 at a time, just to quickly glance through at the way the data was formatted. Most looked like dialog tables and image files, with a few that didn't look like anything I knew and that POLUtils didn't recogize. However, the data in these files appeared either too dense, or too inconsistently distributed within the file to be good candidates.

Either the new dats have changed the arrangement of the values, are using another encryption method than bit shifting, or have been appended to something else of a different structure. Or possibly even included as a resource in a dll or something.

Only thing I can do now is scan everything, any size, regardless if it thinks it's an image or dialog, and see. Beyond that, there's not much to do other than maybe pattern searching. Looking for 600-800ish blocks of data at regular intervals with some repeating values, sequential values, and zero padding. Combined with encryption, that'll be a tough search. Might as well kiss my laptop goodbye for the day if I run that one.

[rog]

If you want to share whatever tool you're using to scan them, i'd be happy to run it while i sleep.