Ogg Vorbis

[Leo Davidson]

Hello,

I've created a Structure Viewer definition for Ogg Vorbis audio files and thought I'd share it here in case someone else finds it useful.

I'm writing code to extract Ogg Vorbis meta-data (album, artist, cover-art, etc.) and basic information (sample rate, bitrate, channels) so that's what I have focused on.

This is my first time creating a Structure Viewer definition and I'd welcome any advice.

EDIT: The question in the quote box below was part of my original post. Barfy's reply solved my problem and I've updated my definition. I've left the question, and the old defintion, here in case it helps someone else (and so the rest of the thread still makes sense):

[Barfy]

Hello Leo,

First of all, we would like to thank you very much for contibuting the OGG file format!

Certainly it would be great to have a for operator (and we plan to add support for it in the future), but there currently is a way to simplify the definition using the intrinsic array_index variable. When used inside an array, it returns a current array index, thus allowing us to reference data from other arrays, for example:

[Leo Davidson]

Thanks for the tips!

array_index does exactly what I needed. I overlooked the Field Access page in the manual and I see there are some other useful things in there.

The stuff I did with $bind was for convenience, but not really needed anymore so I'll switch to using your version. Originally I had it parsing the entire Ogg container, which could have thousands of pages/segments, and it was a bit of a pain to see just the metadata stuff in the top few segments. So I moved those structures into separate top-level binds and collapsed all the other stuff. But now that it stops parsing the file after the end of the metadata, and also now that I've discovered that Hex Editor Neo will save which branches of the tree are expanded/collapsed, there's no need for that.

I'll update my copy of the file in a little while, once I've had a chance to check out the changes against a few examples. (Taking a short break from this project at the moment!)

I really appreciate your quick & helpful support, as always! Hex Editor Neo has made doing this kind of work soooo much better.

Sometime later this week I expect I'll have a FLAC (free lossless audio codec) structure viewer example to put up as well. Edit: Done. FLAC sample is up too.

[Leo Davidson]

I checked out your changes against some test files and incorporated most of them into my version, which I've updated in the root post. Thanks again! Using array_length makes things much more elegant. I also noticed the improvement that using an enum for the bitfields gives (where the structure viewer uses less space to say which bits are set).

I've kept my old bind logic though, and remembered the most important reason it's needed.

The Ogg container is split into pages and segments. Each page can contain up to 255 segments and each segment can contain up to 255 bytes.

Chunks of Vorbis data within the Ogg container can be longer than 255 bytes, and also longer than 255*255 bytes. So a Vorbis data structure can span multiple segments or even multiple pages.

If any structure spans segments (e.g. a VorbisComments block that is longer than 255 chars then I don't think you can parse the Ogg and Vorbis parts at once, because the Vorbis structures are not always contained within the Ogg ones. However, within a given page there is no data between each segment -- the length array comes first and there is no padding -- so you can treat contiguous segments as a Vorbis structure. Using bind does just that and solves that problem well.

Here's an animated gif I made to show what I mean:



(Essentially Ogg Vorbis is a format within a format.)

It gets more complex with page spanning:

If any structure spans pages (which could be due to a very long structure, or just a >255 byte structure that happens to appear at the end of a page after several other segments) then I don't think we can parse that structure at all, at least not in full. That's because there will be bits of Ogg page headers interwoven with the Vorbis data.

With Ogg Vorbis metadata parsing, you're most likely to encounter this problem when there's cover-art in the VorbisComments block. (It's base64 encoded and inserted like any other comment string. Since a cover image could be a couple of meg, in an extreme case, it's possible for several pages, and lots and logs of segments, to be spanned.)

Perhaps page-spanning Vorbis data could be parsed by a script which actually extracted and combined the data into a memory buffer and then parsed that, but that's overkill for my purposes. So long as the structure viewer shows me the start of the structure (and doesn't give up completely) I'm happy, and that's what I've got right now.

[Barfy]

Decided to post here again to mention that we have implemented a while, for and do... while loops (as discusses above), as well as improved the scripting support.

For a good starting example, you may consult the updated PE header file, which now uses a Javascript file to keep some state and to perform address translation. This allows it to parse almost as much information as contained in PE header, including export directory, import directory, resources table and so on. It even displays the type of individual resource.

In addition, javascript code may now be specified directly inside Structure Viewer's definition files, there is no need to use external files anymore (pe.h still uses one, though).