Parser please.

[TsingTao]

Yeah, I always hated the basement in Garlagie for that very reason, the Attack down is just evilly strong. God forbid you actually invite a WHM that has Erase by 47 to fix it.

[Cylo]

But, just because some people use it for that purpose doesn't mean everyone does, just like all Americans don't put ketchup on everything they eat (in other words, stereotypes aren't accurate).

yummmm, someone say ketchup?

[genome]

Hey divisortheory or anybody writing parsers, I am writing my own just for myself right now, and I pretty much guessed what all the fields in each line are for except for the one right after the line number basically, my guess is that it is the time since the previous line since there isn't any other "time" info field. Is that correct?

Also, what's all the extra bytes at the very beginning of each .log file? anything useful/interesting?

I thought I would just look at spyle's source and get a quick idea on those but his VB6 coding is very messy and I actually found it easier to guess everything from the logs themselves...

[Delenn]

I pretty much guessed what all the fields in each line are for except for the one right after the line number basically, my guess is that it is the time since the previous line since there isn't any other "time" info field. Is that correct?

Also, what's all the extra bytes at the very beginning of each .log file? anything useful/interesting?

What generated the log file that you're analyzing?

I'm personally not familiar with approach Spyle used, but I know divisortheory uses LogMon to generate XML logs. Here's an example chatline structure:

<chatline>
     <date>25/10/06</date>
     <time>23:36:41</time>
     <code>121</code>
     <line>2049</line>
     <multiline>1</multiline>
     <conversationthread>1921</conversationthread>
     <color>c0c050</color>
     <text><![CDATA[Equipment changed.]]></text>
 </chatline>

The XML is extremely easy to parse in Perl or Python 8)

Hope that helps.

[Delenn]

As for some details that I noticed:

• Date = day, month, year[/*:m:01824]
• Time = hour, minute, second[/*:m:01824]
• Code = tag, akin to chat filters[/*:m:01824]
• Line = line number since LogMon starting recording, not really useful[/*:m:01824]
• Multiline = at first I thought this would be great for determining weapon skills / critical hits / other instances that span lines, however, multiline seems to always be set to 1, so it's not useful [/*:m:01824]
• Conversationthread = denotes an atomic unit, like a critical hit, this IS useful[/*:m:01824]
• Color = RGB color code in hex, tied to your game settings[/*:m:01824]
• Text = the actual text string[/*:m:01824]

Please feel free to correct me if I'm wrong.

[divisortheory]

I actually implemented hundreds of lines of code in my parser to piece together multi-line chat log messages before this conversationThread field was introduced into LogMon. Wish that had been around when I first started lol, it took me days to get that right.

Edit: Delenn, do you happen to be familiar with memlocs of the chat log structures in FFXI? I would do anything to be able to remove my dependency on LogMon.

To genome, probably the chat code is the field you are wondering about, it classifies the message into broad categories, and this is also most likely how chat filters work, by enabling / disabling entire chat codes.

[Delenn]

Unfortunately, I know little about FFXI memlocs due to lack of interest.

I originally played with LogMon to keep track of funny LS quotes.

Admittedly, it is fun writing perl to generate webpages, then modifying content via CSS, especially for small Limbus runs

[divisortheory]

Unfortunately, I know little about FFXI memlocs since I never was really interested.

I originally played with LogMon to keep track of funny LS quotes.

I'll admit writing perl to generate webpages then modifying content via CSS has been fun though, especially for small Limbus runs

That sucks I found this thing called ffxialarm recently which also reads the chat log and is real time. Unfortunately it's written in Ada, which I've never even seen a line of before lol, but I might try looking at that source and seeing if I can figure out how it gets memlocs. It really pisses me off to no end that LogMon forces you to be a "plugin", so if all people want to do is parse something they have to start up this other program, click a bunch of buttons, and finally they've loaded your parser.

[genome]

I pretty much guessed what all the fields in each line are for except for the one right after the line number basically, my guess is that it is the time since the previous line since there isn't any other "time" info field. Is that correct?

Also, what's all the extra bytes at the very beginning of each .log file? anything useful/interesting?

What generated the log file that you're analyzing?

I'm personally not familiar with approach Spyle used, but I know divisortheory uses LogMon to generate XML logs. Here's an example chatline structure:

<chatline>
     <date>25/10/06</date>
     <time>23:36:41</time>
     <code>121</code>
     <line>2049</line>
     <multiline>1</multiline>
     <conversationthread>1921</conversationthread>
     <color>c0c050</color>
     <text><![CDATA[Equipment changed.]]></text>
 </chatline>

The XML is extremely easy to parse in Perl or Python 8)

Hope that helps.

Oh, that explains a lot but doesn't help me much
I am not using anything to generate log files. FFXI writes down temporary log files in its own binary format, and keeps them in the the TEMP folder in FFXI. One slight disadvantage is that you can't immediately see all the data in real time, until it dumps it to the files (usually each few screens), but I don't really care much about that plus it's nice to be able to parse anytime independently of any programs running or not.

So far I had to guess and interpret the meaning of each byte, it's not that hard but it's a bit tedious so I was hoping to save some time by asking.. I guess I'll have to keep doing it though since I don't want to use memlocs, ffximon etc, mine right now is very short and compact and completely independent of everything.

Divisortheory, maybe you want to use my same approach to become independent of ffximon (or keep both options). I already mentioned some of the binary encoding I deduced in this post and in the PM i sent you which could help you too.

[Devek]

Unfortunately, I know little about FFXI memlocs since I never was really interested.

I originally played with LogMon to keep track of funny LS quotes.

I'll admit writing perl to generate webpages then modifying content via CSS has been fun though, especially for small Limbus runs

That sucks I found this thing called ffxialarm recently which also reads the chat log and is real time. Unfortunately it's written in Ada, which I've never even seen a line of before lol, but I might try looking at that source and seeing if I can figure out how it gets memlocs. It really pisses me off to no end that LogMon forces you to be a "plugin", so if all people want to do is parse something they have to start up this other program, click a bunch of buttons, and finally they've loaded your parser.

Reading the chat log real time poorly and reading the chat log in real time reliably are two different things. Gimme a link to the source and i'll look at it, I know Ada.

[divisortheory]

If all I wanted was a parser, FFXIP is sufficient. I want to take parsing to the next level by having it be real time. There are so many cool features that are possible to implement if you have real time parsing. Imagine, a single column called DPS that calculated your actual, hands down damage per second. This would fucking destroy everything else in the world, and it's very possible with real time parsing. Guy got killed during party and was out for 5 minutes skewing results? DPS would account for that. Switched party members halfway in skewing results? DPS would account for that. In fact, with a DPS field, you almost wouldn't even need other fields at all, and it would be a far superior way of comparing performance than with current parsers, all you would have to do is change gear and check if your DPS is higher or lower than it was before.

Devek: http://files.filefront.com//;6057090;;/ is a link to the latest version of the source code of ffxialarm. You can find out more info on this program here (Note: I've never used this program, I only learned of its existence today)

[genome]

Definitely a real time DPS would be really cool.
But aside for seeing a number instantaneously instead of with a little delay, the other examples you mention are all things you can do in offline parsing too.
For swapping gear you just have to add some text in your macro, and for people dead you have the log message when they die and when they get raised.

I guess having memloc info directly though would be even cooler and less gimmicky, but all I am saying is that you still can do all that except display it without a delay with offline parsing.

You have to take and average some minimal number of lines to get meaningful DPS values anyways so even seeing it with a 2-3 minutes delay wouldn't be the end of the world, and it's still nice to be able to not have to run memory-dependant programs or 3rd PT tools for me, so I might stick to offline for mine for the moment (and use yours for online stuff when it's ready!)

[divisortheory]

I talked to the creator of ffxi alarm. Nice guy, and said he was gonna try to explain to me how to get memlocs for chat logs. If that works I'll be a happy man

[LinktheDeme]

slightly related. . . anyone know if all parsers make that annoying computer beep or just ffxip?

[Aodh]

Another thought... If there is the ability to timestamp logs, would it also be possible to create a parser which also acts as a chat filter so that you don't have to see all the damage in the log, but it would still be recorded? Or does something like that already exist?

[divisortheory]

Another thought... If there is the ability to timestamp logs, would it also be possible to create a parser which also acts as a chat filter so that you don't have to see all the damage in the log, but it would still be recorded? Or does something like that already exist?

The only person who knows how to actually timestamp the logs you see on your screen is Cliff from Windower, and it's never been released. But yes, I think what you're saying is possible. Youd' have to leave all your filters off in game, then the 3rd party program would delete lines from your chat log after they were received by the client but before they were displayed to your screen.

[TsingTao]

I don't even care if it timestamps the logs. A parser that also acted as a chat filter would be kick enormous amounts of ass.

[divisortheory]

I don't even care if it timestamps the logs. A parser that also acted as a chat filter would be kick enormous amounts of ass.

Well they go hand in hand. If you know how to do one, you probably know how to do the other.

[TsingTao]

While knowing how to do one might go hand in hand with the other actually caring to implement it might not. People have different priorities....mine would be the code to filter the chat and not time stamp it.

[TazzyLynn]

slightly related. . . anyone know if all parsers make that annoying computer beep or just ffxip?

Go to Options > Enable Timer Sounds > Unclick it. The beep will stop.