ENIGMA Forums

Contributing to ENIGMA => Proposals => Topic started by: Josh @ Dreamland on January 28, 2011, 10:19:02 pm

Title: ENIGMA Game Format
Post by: Josh @ Dreamland on January 28, 2011, 10:19:02 pm
ENIGMA needs its own format for game storage. GM takes far too long to load and store some executables; I was thinking it would be best to use a format allowing for quick addition, subtraction, and modification of contained resources.

With that in mind, and with the interests of picking a format that works well with Java, I believe our best bet is to use ZIP.

Essentially, .ogf or .egm files would be a compressed directory containing a subdirectory for each resource, which would be in its own format. I was thinking that the format could be largely eYAML; for instance, a sprite would be given a file name reflecting its ID (sprite0000.ey), and would contain a name: and file: attribute indicating, of course, the actual name of the resource and the image file it contains. This way, the format would basically document itself.

Just as easily, however, individual resources could be kept in separate binary files with their own unique formats.

As an option, a manifest file could be kept, but I think it'd be nicer if users could open and manipulate the resources in any zip-capable file browser and text editor. ZIP also reduces the likelihood for total corruption, and there are many tools around which can repair damaged zip files. Plus, since ZIP is he medium of choice for Java, it is unlikely we will encounter such an issue if we are using Java's own ZIP functions for the same purpose.

Any other settings would be stored in the trunk of the Zip, along with the game information and other important metrics.

This concludes my proposal for the egm (ENIGMA) or ogf (Open Game-) format. If you believe you have a better idea, post it here, and I'll call you a trollwe'll have a reasonable debate.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on January 29, 2011, 12:31:57 am
I agree with the zip directory idea.
I also think textual resource information is a good idea especially earning it the "ogf" extension.

I'm concerned:
1) About using eYaml for the textual resource information. This is basically just a growing distaste for Yaml. I was hesitant about it at first, but went with it because it was basically the best format I could find for the job (which isn't saying much). Now that I've seen it get hacked to bits in the eYaml spec and the parsers that parse it in such a hacked up way that I wonder how it even sticks together... I can't help but look at eYaml with the same kind of feeling I get when I look at a clunker car that was assembled with odds and ends in my garage. Fortunately at this point in the project eYaml is a module that serves as a translation between the yaml file and the areas that use it. It could very easily be replaced and probably even interchanged with a XML module or any other module.

2) Binary support? Text is all well and grand, but some projects get massive, even after being zipped, and having everything in a text format... Just tell me what your plans are for binary support, and I'll be happy.
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on January 29, 2011, 05:46:13 am
Same as my plans for image formats. Each resource has a text file with the name and any other information that can be left text, and a reference to a binary file in the same directory. I don't have a problem with resources that would use only a binary file and a name simply having such info glued onto the beginning (we could standardize a format for that).

I won't defend YAML. Some of it is glued together. But it's much easier to parse than XML, and it's much easier to read and much harder to fuck up. Assuming INI is the format used in GNOME .desktop files, that is my secondary proposal.

Besides, I think that if the format is to earn the OGF extension, it should utilize a format that every language has a parser for, not just Python. So yeah, how's INI sound?
Title: Re: ENIGMA Game Format
Post by: luiscubal on January 29, 2011, 10:14:42 am
Possible formats:
1. XML
2. JSON
3. INI
4. YAML

I personally prefer JSON, even if Wikipedia says JSON is a subset of YAML.
I don't know about Java, but I've used JSON in C++ and it's pretty simple. Not to mention that being a subset of JavaScript means it's on its way to becoming a major language of the web.
It's also incredibly popular.

One other possibility is not mandating any format at all. Just demand a format for sprites, objects, rooms, etc. but not for custom resources. Then anything(plain text, XML, MP3) would be acceptable for custom resources.
Title: Re: ENIGMA Game Format
Post by: RetroX on January 29, 2011, 12:51:33 pm
I prefer YAML because it has the least amount of wasted space.

XML:
<category><subcategory><list><of><stuff><x>0</x><y>0</y></stuff></of></list></subcategory></category>

YAML:
x: 0
y: 0

Not really like that but you get the idea.
Title: Re: ENIGMA Game Format
Post by: Rusky on January 29, 2011, 02:16:30 pm
YAML looks like a nicer format, but I haven't had to deal with parsing it so it may not be the best. JSON is growing in popularity and might be a better choice than YAML, depending on what you want to store.

XML has much better parser support, but it is rather verbose and doesn't make much sense unless you're actually using it as an eXtended Markup Language rather than for configuration or whatever of the billions of other misuses there are.

INI seems like an old, tired format from the 90s. Not that that's a bad thing - it makes it relatively easy to parse - but I don't know how flexible it would be.

In any case, it would be nice to allow different resources to specify their own formats. Just supply a built-in parser for YAML/JSON/XML but let resources decide how to handle things themselves. It would be interesting, I think, to have even built-in resources use the same system (if they don't already).
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on January 29, 2011, 04:02:40 pm
I feel the same way about XML, and I felt the same way about INI before I saw it was basically GNOME's .desktop format.

YAML is actually pretty easy to parse, but parsers do backflips and are relatively gross in concept. @Rusky I didn't consider offering multiple formats; I think we should have the specification call for it, but I personally don't want to include multiple parsers. Are you suggesting that the file should give a doctype, or that (1) the reader will be expecting the same format the writer uses and (2) an intervening human would be able to tell which format it was? I'm thinking the latter; doctypes are gross.

As for JSON, I recall that it was very similar to YAML. In fact, looking at sample JSON, it seems that YAML is just an ugly mix between Python and JSON (it uses Python's indentation grouping instead of {} like most of everything else).

So basically,
XML is out of the question unless the specification is to allow flexible formats (in which case ENIGMA will =NOT= be using XML anyway),
JSON is just better-known and more stable YAML and would be a practical consideration (I'll have to modify my YAML parser or strap on a standard one).
INI is an older format and would require compromise on both ends.
YAML makes Ism's skin crawl (the full spec makes my skin crawl) and would involve sizey compromise on either end to support fully, and
eYAML is a concise implementation of YAML that makes only Ism's skin crawl but would be too specific for a general format, thus requiring formats to be optional as per Rusky's proposal.

I am, in general, all for having the text format be up to the extension. However, it seems messy to specify that there is no specified format for these text files, and messier to constrain it to four or so well-known formats without having some doctype, which are nasty.
Should we settle on some means of allowing multiple formats, we still have to choose one for LGM extensions to use (I personally doubt we'll use more than one for just ENIGMA, though we should leave it open for future users to do so).
Title: Re: ENIGMA Game Format
Post by: Rusky on January 29, 2011, 04:18:06 pm
I wasn't suggesting multiple built-in parsers; rather that you include a single parser for whichever format you like best and offer it to resource reader/writers, but allow them to choose whatever they like instead. Just keeping a consistent format between reading and writing should be enough to differentiate formats, and I don't really think it would be all that gross since all the built-in ones would be standard and external ones could be considered black boxes.

I'd go for JSON, with GML in its present state. However, a lot of projects that use scripting languages are configured through their language- if GML/EDL were to introduce something JSON-like (or even compatible) the best choice would be to simply reuse the parser- less code to maintain, less stuff to learn for new users/contributors, and potentially a more powerful configuration language (maybe write LGM extensions in EDL as well as game extensions?).
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on January 29, 2011, 09:24:39 pm
I like the ideas put forward here, and I can see we're very much on a similar page as far as formats go. I'd like to point out that we also don't have to restrict ourselves to the existing formats. Considering the flaws of the common formats, it might be a good idea to create a new format which would be easier to parse while still keeping the text fairly user-friendly for a text editor. Especially now that we have experience on what we *want* to parse (as opposed to what we *should* parse) so that we can design a parser that isn't just totally hacked together.

As for extensions in EDL, it's a noble idea, but it could be problematic when we consider that LGM itself doesn't recognize EDL - that's entirely ENIGMA-side. One of the handy things about LGM is that you can clip it from ENIGMA, put it by itself, and still run it, making it a compiler-less IDE (without Syntax Checking).
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on January 29, 2011, 11:02:10 pm
I side with Ism on not using EDL, despite the unification it would offer. LGM is Java, and the point of ENIGMA is for game development. Put simply, we don't have the resources (read: necessary number of programmers) to allow LGM development in EDL.

If we were to design our own format and make it an official part of our spec, then I strongly believe we should cut up the YAML format and run with the bits we like. Define our own behavior for tab and space mixing (namely, treating tabs and spaces identically). I don't mind using {} for groups to eliminate all ambiguity, but I think if python could do without them, so can we. I do like JSON, except I don't think quotes around keys is necessary.

Give the format some thought. We can probably come to an agreement on the IRC.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on January 30, 2011, 12:20:06 am
I like it. I think we'll come to an agreement that will make us both happy.
Title: Re: ENIGMA Game Format
Post by: Fede-lasse on January 31, 2011, 03:36:29 am
*signs peace treaty*
Title: Re: ENIGMA Game Format
Post by: Rusky on January 31, 2011, 11:58:20 am
Python gets away without {} at the expense of being impossible to rescue when formatting gets messed up. Considering actual JavaScript object notation doesn't require quotes around keys, just dropping those from JSON might be okay. However, if everyone's okay with a custom format it might be good to drop the commas from JSON and use newlines instead (i.e. just add {} to YAML).
Title: Re: ENIGMA Game Format
Post by: freezway on February 01, 2011, 01:24:55 am
It must be written as a program (in SPL (yes, shakespeare) that compiles a SPL program that then takes an encrypted text file encased in a JPEG and creates a JSON file out of that...
well... still better than XML.
Title: Re: ENIGMA Game Format
Post by: RetroX on February 02, 2011, 07:17:40 pm
Just butting in randomly, but I think that we should do something similar to what .DEB does and include a file named "enigma" in the ZIP that contains a certain sequence of bytes, which will always be the first file in the archive.  This will make it easy to identify.  I don't know what ZIP does to store things, but assuming that ZIP always compresses in the same way when the same sequence of files is given, it should work nicely.
Title: Re: ENIGMA Game Format
Post by: RetroX on April 26, 2011, 04:50:07 pm
Bump.

Format proposal:
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on April 26, 2011, 06:42:11 pm
Looks good and formal enough for the internal directory structure. I'd just need specifics for how to handle the leafs. I suppose we can deal with them one resource type at a time.

Figure out what you want to do with definitions, and how you plan on handling binary data in the game settings, such as the icon and the progress bar images. Once you do that, I'll get started on a prototype.
Title: Re: ENIGMA Game Format
Post by: RetroX on April 26, 2011, 06:50:57 pm
Well, I was assuming that whitespace definitions would only be temporary; in the end, it would end up being just another resource.  But that's not the case now.

I think that the best thing to do would be to create a config directory that has the icon/progress bar images as PNGs and a whitespace definitions.h.  In addition, I think that each resource should be modular; in other words, iteratively looped through.  I think that, eventually, it might be a good idea to make "Systems" modular as opposed to the ugly system that we have of the *_Systems folders.
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on April 26, 2011, 08:21:03 pm
Why do we need a data and settings directory? They should just be thrown into the resource's directory, with a manifest file.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on April 26, 2011, 08:51:38 pm
Don't worry about _Systems, because I keep pestering josh to make it modular because I can't keep dealing with it being hard-coded in each time.

As for definitions and family, it might be frugal to pretend like they are resources in anticipation of the future when that does happen. In the meantime, we can hardcode routines for dealing with them and loading them in as non-modular.
Title: Re: ENIGMA Game Format
Post by: RetroX on April 26, 2011, 08:52:27 pm
Why do we need a data and settings directory? They should just be thrown into the resource's directory, with a manifest file.
For ease of use and editing outside of LGM.

If someone wanted to look inside of the ZIP, it wouldn't be a pile of shit, but actually something comprehensible.
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on April 26, 2011, 11:21:57 pm
If they had the contents sorted by name, they would open the directory to find a list of pairs of text and binary files. sprite0.ey, sprite0.png, sprite1.ey, sprite1.png. It'd look fine, and it'd save moving back and forth between folders to make sure both are correct.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on April 27, 2011, 12:17:39 am
.png doesn't account for multiple subimages unless you're using a spritesheet.
Title: Re: ENIGMA Game Format
Post by: Josh @ Dreamland on April 27, 2011, 12:23:16 am
Or, say, a layered PNG? Most of the reasoning behind the PNG spec is how extensible it is. We could easily invest in a reader-friendly way to animate PNG.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on April 27, 2011, 01:31:50 pm
Fair enough.
Title: Re: ENIGMA Game Format
Post by: RetroX on April 27, 2011, 05:51:27 pm
There is an APNG format.  Just use that if you're really that inclined to use a single file.

It's better to just make a directory tree with multiple images.
Title: Re: ENIGMA Game Format
Post by: IsmAvatar on April 29, 2011, 08:13:24 pm
I'm not sure what kind of say the user has in the organization of a zip file - it may be a concern because it might make it impossible to manually create an egm file if they can't put the "enigma-package" file first.

Josh and I have discussed the Definitions et-al family idea, and came up with a radio button group along with each code that defines whether it's a Definitions, Initialization, GlobalLocals, or Cleanup code.