ENIGMA Game Format

[Josh @ Dreamland]

ENIGMA needs its own format for game storage. GM takes far too long to load and store some executables; I was thinking it would be best to use a format allowing for quick addition, subtraction, and modification of contained resources.

With that in mind, and with the interests of picking a format that works well with Java, I believe our best bet is to use ZIP.

Essentially, .ogf or .egm files would be a compressed directory containing a subdirectory for each resource, which would be in its own format. I was thinking that the format could be largely eYAML; for instance, a sprite would be given a file name reflecting its ID (sprite0000.ey), and would contain a name: and file: attribute indicating, of course, the actual name of the resource and the image file it contains. This way, the format would basically document itself.

Just as easily, however, individual resources could be kept in separate binary files with their own unique formats.

As an option, a manifest file could be kept, but I think it'd be nicer if users could open and manipulate the resources in any zip-capable file browser and text editor. ZIP also reduces the likelihood for total corruption, and there are many tools around which can repair damaged zip files. Plus, since ZIP is he medium of choice for Java, it is unlikely we will encounter such an issue if we are using Java's own ZIP functions for the same purpose.

Any other settings would be stored in the trunk of the Zip, along with the game information and other important metrics.

This concludes my proposal for the egm (ENIGMA) or ogf (Open Game-) format. If you believe you have a better idea, post it here, and I'll call you a trollwe'll have a reasonable debate.

[IsmAvatar]

I agree with the zip directory idea.
I also think textual resource information is a good idea especially earning it the "ogf" extension.

I'm concerned:
1) About using eYaml for the textual resource information. This is basically just a growing distaste for Yaml. I was hesitant about it at first, but went with it because it was basically the best format I could find for the job (which isn't saying much). Now that I've seen it get hacked to bits in the eYaml spec and the parsers that parse it in such a hacked up way that I wonder how it even sticks together... I can't help but look at eYaml with the same kind of feeling I get when I look at a clunker car that was assembled with odds and ends in my garage. Fortunately at this point in the project eYaml is a module that serves as a translation between the yaml file and the areas that use it. It could very easily be replaced and probably even interchanged with a XML module or any other module.

2) Binary support? Text is all well and grand, but some projects get massive, even after being zipped, and having everything in a text format... Just tell me what your plans are for binary support, and I'll be happy.

[Josh @ Dreamland]

Same as my plans for image formats. Each resource has a text file with the name and any other information that can be left text, and a reference to a binary file in the same directory. I don't have a problem with resources that would use only a binary file and a name simply having such info glued onto the beginning (we could standardize a format for that).

I won't defend YAML. Some of it is glued together. But it's much easier to parse than XML, and it's much easier to read and much harder to fuck up. Assuming INI is the format used in GNOME .desktop files, that is my secondary proposal.

Besides, I think that if the format is to earn the OGF extension, it should utilize a format that every language has a parser for, not just Python. So yeah, how's INI sound?

[luiscubal]

Possible formats:
1. XML
2. JSON
3. INI
4. YAML

I personally prefer JSON, even if Wikipedia says JSON is a subset of YAML.
I don't know about Java, but I've used JSON in C++ and it's pretty simple. Not to mention that being a subset of JavaScript means it's on its way to becoming a major language of the web.
It's also incredibly popular.

One other possibility is not mandating any format at all. Just demand a format for sprites, objects, rooms, etc. but not for custom resources. Then anything(plain text, XML, MP3) would be acceptable for custom resources.

[RetroX]

I prefer YAML because it has the least amount of wasted space.

XML:
<category><subcategory><list><of><stuff><x>0</x><y>0</y></stuff></of></list></subcategory></category>

YAML:
x: 0
y: 0

Not really like that but you get the idea.

[Rusky]

YAML looks like a nicer format, but I haven't had to deal with parsing it so it may not be the best. JSON is growing in popularity and might be a better choice than YAML, depending on what you want to store.

XML has much better parser support, but it is rather verbose and doesn't make much sense unless you're actually using it as an eXtended Markup Language rather than for configuration or whatever of the billions of other misuses there are.

INI seems like an old, tired format from the 90s. Not that that's a bad thing - it makes it relatively easy to parse - but I don't know how flexible it would be.

In any case, it would be nice to allow different resources to specify their own formats. Just supply a built-in parser for YAML/JSON/XML but let resources decide how to handle things themselves. It would be interesting, I think, to have even built-in resources use the same system (if they don't already).

[Josh @ Dreamland]

I feel the same way about XML, and I felt the same way about INI before I saw it was basically GNOME's .desktop format.

YAML is actually pretty easy to parse, but parsers do backflips and are relatively gross in concept. @Rusky I didn't consider offering multiple formats; I think we should have the specification call for it, but I personally don't want to include multiple parsers. Are you suggesting that the file should give a doctype, or that (1) the reader will be expecting the same format the writer uses and (2) an intervening human would be able to tell which format it was? I'm thinking the latter; doctypes are gross.

As for JSON, I recall that it was very similar to YAML. In fact, looking at sample JSON, it seems that YAML is just an ugly mix between Python and JSON (it uses Python's indentation grouping instead of {} like most of everything else).

So basically,
XML is out of the question unless the specification is to allow flexible formats (in which case ENIGMA will =NOT= be using XML anyway),
JSON is just better-known and more stable YAML and would be a practical consideration (I'll have to modify my YAML parser or strap on a standard one).
INI is an older format and would require compromise on both ends.
YAML makes Ism's skin crawl (the full spec makes my skin crawl) and would involve sizey compromise on either end to support fully, and
eYAML is a concise implementation of YAML that makes only Ism's skin crawl but would be too specific for a general format, thus requiring formats to be optional as per Rusky's proposal.

I am, in general, all for having the text format be up to the extension. However, it seems messy to specify that there is no specified format for these text files, and messier to constrain it to four or so well-known formats without having some doctype, which are nasty.
Should we settle on some means of allowing multiple formats, we still have to choose one for LGM extensions to use (I personally doubt we'll use more than one for just ENIGMA, though we should leave it open for future users to do so).

[Rusky]

I wasn't suggesting multiple built-in parsers; rather that you include a single parser for whichever format you like best and offer it to resource reader/writers, but allow them to choose whatever they like instead. Just keeping a consistent format between reading and writing should be enough to differentiate formats, and I don't really think it would be all that gross since all the built-in ones would be standard and external ones could be considered black boxes.

I'd go for JSON, with GML in its present state. However, a lot of projects that use scripting languages are configured through their language- if GML/EDL were to introduce something JSON-like (or even compatible) the best choice would be to simply reuse the parser- less code to maintain, less stuff to learn for new users/contributors, and potentially a more powerful configuration language (maybe write LGM extensions in EDL as well as game extensions?).

[IsmAvatar]

I like the ideas put forward here, and I can see we're very much on a similar page as far as formats go. I'd like to point out that we also don't have to restrict ourselves to the existing formats. Considering the flaws of the common formats, it might be a good idea to create a new format which would be easier to parse while still keeping the text fairly user-friendly for a text editor. Especially now that we have experience on what we *want* to parse (as opposed to what we *should* parse) so that we can design a parser that isn't just totally hacked together.

As for extensions in EDL, it's a noble idea, but it could be problematic when we consider that LGM itself doesn't recognize EDL - that's entirely ENIGMA-side. One of the handy things about LGM is that you can clip it from ENIGMA, put it by itself, and still run it, making it a compiler-less IDE (without Syntax Checking).

[Josh @ Dreamland]

I side with Ism on not using EDL, despite the unification it would offer. LGM is Java, and the point of ENIGMA is for game development. Put simply, we don't have the resources (read: necessary number of programmers) to allow LGM development in EDL.

If we were to design our own format and make it an official part of our spec, then I strongly believe we should cut up the YAML format and run with the bits we like. Define our own behavior for tab and space mixing (namely, treating tabs and spaces identically). I don't mind using {} for groups to eliminate all ambiguity, but I think if python could do without them, so can we. I do like JSON, except I don't think quotes around keys is necessary.

Give the format some thought. We can probably come to an agreement on the IRC.

[IsmAvatar]

I like it. I think we'll come to an agreement that will make us both happy.

Post made January 31, 2011, 03:36:29 am was deleted at the author's request.

[Rusky]

Python gets away without {} at the expense of being impossible to rescue when formatting gets messed up. Considering actual JavaScript object notation doesn't require quotes around keys, just dropping those from JSON might be okay. However, if everyone's okay with a custom format it might be good to drop the commas from JSON and use newlines instead (i.e. just add {} to YAML).

[freezway]

It must be written as a program (in SPL (yes, shakespeare) that compiles a SPL program that then takes an encrypted text file encased in a JPEG and creates a JSON file out of that...
well... still better than XML.

[RetroX]

Just butting in randomly, but I think that we should do something similar to what .DEB does and include a file named "enigma" in the ZIP that contains a certain sequence of bytes, which will always be the first file in the archive.  This will make it easy to identify.  I don't know what ZIP does to store things, but assuming that ZIP always compresses in the same way when the same sequence of files is given, it should work nicely.