ENIGMA Forums

Outsourcing saves money => Issues Help Desk => Topic started by: edsquare on January 19, 2015, 06:59:51 pm

Title: Extension .egm does not match EGM?
Post by: edsquare on January 19, 2015, 06:59:51 pm
Taking advantage of Robert's offer:  ::)

As usual let us know if you have any more problems!

 :ohdear:

What's wrong with the EGM projects?

How can I save as .egm?
Title: Re: Extension .egm does not match EGM?
Post by: Goombert on January 19, 2015, 07:34:14 pm
Yeah this is a known bug, when adding multiple constants support I temporarily broke EGM format. Basically me and Josh really aren't sure what to do at this point.

Josh hates the EGM format, it saves rooms as binary blobs like the GMK format and it's barely extensible at all. What he would like is to basically rewrite the whole format with a proper spec and implement everything properly. We have a GitHub ticket filed on this:
https://github.com/enigma-dev/lgmplugin/issues/29

For now you can just use the old LGM if you want, though I don't think it will work with ENIGMA anymore. We really need to get this fixed ASAP but I specifically don't know what to do, I'll have to talk to Josh.
http://enigma-dev.org/docs/Wiki/LateralGM:_Revisions
Title: Re: Extension .egm does not match EGM?
Post by: edsquare on January 19, 2015, 07:43:51 pm
Yeah this is a known bug, when adding multiple constants support I temporarily broke EGM format. Basically me and Josh really aren't sure what to do at this point.

Josh hates the EGM format, it saves rooms as binary blobs like the GMK format and it's barely extensible at all. What he would like is to basically rewrite the whole format with a proper spec and implement everything properly. We have a GitHub ticket filed on this:
https://github.com/enigma-dev/lgmplugin/issues/29

For now you can just use the old LGM if you want, though I don't think it will work with ENIGMA anymore. We really need to get this fixed ASAP but I specifically don't know what to do, I'll have to talk to Josh.
http://enigma-dev.org/docs/Wiki/LateralGM:_Revisions

It will work with the enigma version it had, you see before reinstalling al my OS I made a backup of stuff, so I have a working LGM/ENIGMA 1.8.5

The gmx is the one that makes a folder and puts there your scripts, sprites, etc. Right?

Why not take that and improve it and then use that as the new egm?

Where do you set the extensions?

Maybe I'll have a look see
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on January 20, 2015, 04:34:58 am
Why can't we serialize the room data in an xml just like GM? How do you save GM files then? XML is extensible by design. Or you can do it in YAML or eYAML or whatever else we got there. I don't really see the problem. EGM itself is also actually quite extensible even now. Everything is in it's own folder and all we need are some XML files showing what is where. So if you add a new resource, you just add it to a xml config file and show it in LGM. If an XML entry is missing, then use a default value (one that would be used if you made a new project with File->New).
Title: Re: Extension .egm does not match EGM?
Post by: egofree on January 20, 2015, 04:53:41 am
Meanwhile, would not be possible to revert the egm file reader and writer to the state it was before you worked on constants ? Even if doesn't support constants.
Title: Re: Extension .egm does not match EGM?
Post by: Josh @ Dreamland on February 18, 2015, 10:59:07 pm
XML is the ugliest imaginable format. It's extensible by nature, yes, but it is extremely verbose. And while its hierarchy makes storing code obvious, the problem is simple enough to work around in YAML. As for XML's propensity for hierarchical data, well, rooms are pretty linear in the scheme of things. I posted a proposal somewhere (or maybe I just started coding it and didn't finish) about what I wanted rooms to look like. Consider something like this:

Code: (yaml) [Select]
name: room_0
background-color: 00C0FF
backgrounds:
- name: bkg_3
  depth: 1000
  h-repeat: yes
  v-repeat: no
- name: bkg_overlay
  depth: -1000
  h-repeat: no
  v-repeat: no
views:
- id: 0 # Optional for in-sequence
  visible: yes
  viewport:
    x: 0
    y: 0
    w: 640
    h: 480
  projection:
    x: 0
    y: 0
    w: 640
    h: 480
  follow:
    object: obj_player
    border: [32, 24] # Also allowed: separate lines for h/v
    speed: [4, 4]
object-format:
- id
- object-index
- x
- y
- rotation
object-names:
- obj_player
- obj_dirtwall
- obj_7
objects: [
  [1000001, 0, 64, 400, 0],
  [1000002, 1, 0, 432, 0],
  [1000003, 1, 32, 432, 0],
  [1000004, 1, 64, 432, 0],
  [1000006, 1, 96, 432, 0],
  [1000007, 1, 128, 432, 0],
  [1000008, 1, 160, 432, 0],
  [1000009, 1, 192 432, 0],
  [1000012, 2, 32, 432, 45] # obj_7 is still index 2 on our list, even if its id is really 7.
]
creation-codes:
- ind: 0
  code: |
    // Code begins here
    can_fly = false;
- ind: 8
  code: |
    // Code begins here
    text = "To jump, press [space].";

The "// Code begins here" bit is to prevent issues with code snippets wherein the first line is indented for whatever reason. This is important since YAML is indentation-based. XML deals with this by escaping < and >, which I also disapprove of. Comparatively speaking, the cost of YAML is cheap.

I'll also point out that the verbosity around viewport/projection can theoretically be replaced with [snip=yaml]viewport: {x: 0, y: 0, w: 640, h: 480}[/snip], although e-YAML doesn't presently support that.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on February 19, 2015, 09:47:13 am
You can use attributes in XML though, which greatly reduces verbosity.
Your example in xml:
Code: (xml) [Select]
<room name="room_0" background_color="00C0FF">
  <background name="bkg_3" depth="1000" h-repeat="yes" v-repeat="no"/>
  <background name="bkg_overlay" depth="-1000" h-repeat="no" v-repeat="no"/>
  <view id="0">
    <viewport x="0" y="0" w="640" h="480" />
    <projection x="0" y="0" w="640" h="480" />
    <follow object="obj_player" border_w="32" border_h="24" speed_w="4" speed_h="4" />
  </view>
<object_list>
  <obj name="obj_player" id="0" />
  <obj name="obj_dirtwall" id="1" />
  <obj name="obj_7" id="2" />
<object_list/>
<inst id="1000001" obj="0" x="64" y="400"/>
<inst id="1000002" obj="1" x="0" y="432"/>
<inst id="1000003" obj="1" x="32" y="432"/>
<inst id="1000004" obj="1" x="64" y="432"/>
<inst id="1000006" obj="1" x="128" y="432"/>
<inst id="1000007" obj="1" x="160" y="432"/>
<inst id="1000008" obj="1" x="192" y="432"/>
<inst id="1000009" obj="2" x="32" y="432"/>

<crc ind="1000001">
    can_fly = false;
</crc>

<crc ind="1000008">
    text = "To jump, press [space].";
</crc>
</room>
And it can be less verbose than that. I just like xml syntax more than YAML (I hate when tabs count as syntax), but I really don't like any of them.
Title: Re: Extension .egm does not match EGM?
Post by: edsquare on February 19, 2015, 05:41:55 pm
Where in the code are the gmk, gmx, etc formats declared, created and set?
Title: Re: Extension .egm does not match EGM?
Post by: Josh @ Dreamland on February 19, 2015, 09:36:04 pm
Even with that ugly indent scheme, the XML will be on the order of twice as large for a typical room. The only advantage of the XML is that named attributes allow you to elide attributes with default (probably zero or one) values. If this proves necessary, YAML also allows this using maps. So [snip=xml]<inst id="1000009" obj="2" x="32" y="432"/>[/snip] becomes [snip]{id: 1000009,obj: 2,x: 32,y: 432}[/snip], which is pretty meager savings, but that's still the worst-case, and large rooms will be mostly instances. The ease of parsing of (E)YAML vs XML is also a major feature. And if we're going to go toe-to-toe with full specs, XML will lose outright by YAML's superior support for primitives, node types, and even node references.

Not to mention that the best thing about YAML is that users' code appears verbatim in the format, with the exception that it is indented. I'll grant that it is unlikely for code to contain "</crc>", and unlikely for a given user to want to edit the file manually, and that together this means it's extremely unlikely that a user would want to paste in code with a closing tag, but correctly-formatted code uses HTML entities in place of reserved characters (and there are lots), so in general, a user can't open the room file and copy the code somewhere else, because [snip]if (x &lt; xprevious)[/snip] is not valid EDL.

The comment line we would place in the YAML would be optional; it prevents cases where valid code entered in an IDE would break the format. In the 99 percent case, pasting in a well-formatted piece of code and indenting it twice will suffice, because the first line usually has the least amount of indent. The 99 percent case of the 1 percent case is where someone decided to indent their opening doc comment. They will have to either not do that, or place the opening YAML comment, which we might consider simplifying to, eg, eight or more hyphens.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on February 20, 2015, 10:56:35 am
Actually XML parsing with things like RapidXML takes almost the same amount of cycles as strlen. I believe current YAML implementations take more.
But my beef isn't about the speed or the features we will never use, it's about the whole thing being ugly to use, just like python is ugly to use because of the indent bullshit. But I guess all of that is subjective, so save it as you want. Just make sure every parameters is optional, so EGM never breaks again.
Title: Re: Extension .egm does not match EGM?
Post by: DaSpirit on February 20, 2015, 08:19:02 pm
Actually XML parsing with things like RapidXML takes almost the same amount of cycles as strlen. I believe current YAML implementations take more.
I read that too. Parsing XML is easy. Meanwhile, YAML has so many rules that need to be checked. I like TOML best, but it's more for config files than project files.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on November 10, 2015, 07:14:47 am
This is still an open question. EGM rooms are still binary and we need to fix that. I still vote for XML on how fast it is to parse and how "compact" it can actually be. Also, it is quite easy to implement and use. Josh's arguments didn't convince me and I think I gave good examples. But I don't have anything against e-YAML though. We already use it to save object data and we could use it in rooms as well. If this wasn't LGM side I would gladly do it, as I made an XML format for saving and loading in one of my projects and it was trivially easy.
I also added a function to e-YAML parser to return if a value exists, so we could do default values as well. Sadly LGM and the parser/compiler doesn't actually do this checking, so even by missing an optional value we get segfaults. I can fix the parser/compiler part, but not the LGM part.
Title: Re: Extension .egm does not match EGM?
Post by: Josh @ Dreamland on November 15, 2015, 06:41:28 pm
If you're interested in XML, you may as well use it for everything. And for that price, you may as well use GMX. My understanding is that the two aren't very disparate.
Of course, LateralGM still (and presumably always) sucks at working with both of those formats, so it's not as though binary room data is our only fish to fry.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on November 16, 2015, 07:16:40 am
I cannot expand GMX without breaking it (which some will try to load in GM:S and fail). I actually don't have anything against it and it is very similar to what I think EGM should be - a zip file packing XML resource descriptors with custom (even binary) data, like images and sound. That is what DOCX is compared to DOC. And a lot of other formats are like that too. And EGM is basically the same thing, but uses a non-standard markup language (e-YAML and some other which I don't even remember) and binary room data. I want everything that can change between versions to be non-binary. I just don't see why we need to create two different markup languages (or in YAML case, an extension) just to save textual data? Historically it has always been good to not reinvent the wheel. Especially if nobody in this project finishes the things that they start (including me in many cases). So better is to use something existing so all of this could be fixed in 3 days, instead of making a new language for 3 years and never finish.
Title: Re: Extension .egm does not match EGM?
Post by: Goombert on November 16, 2015, 09:17:13 am
Harri, the reason for the extension was because it is a subset of YAML, so it's easier to implement. At least that is what Josh told me.

I'm still in favor of XML for GMX and YAML for EGM, otherwise there will be practically no difference between the formats and no advantage of one over the other. Also you should be able to add additional properties to GMX without Studio complaining about the file. Studio will only complain if you remove properties or you change the structure of existing ones. But you can and should be able to add a whole new tag, LGM will be doing this as of the next release to keep the id's in there from a GMK file for backwards compatibility. I've tested this and it seems to load fine in Studio, but if you save the file back, Studio will not save the id tags and will remove them.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on November 17, 2015, 06:38:51 am
There are a lot of other things wrong with GMX as well though, like the fact the tags and attributes are not consistent. You already pointed that out in the past and you have documented it here: http://enigma-dev.org/docs/Wiki/GMX_Format
So what I want is a clean, consistent, backwards compatible, future compatible and extendable format. Making EGM with YAML just because GMX uses XML is stupid at best. We already want to support new things like custom resources, includes in EGM, version control with an extracted format (EGX? And I know GM has GMZ here), reordering the resources with several different types in one folder and so on. I cannot do any of that without breaking GMX, which I don't think we need to use at all.
But YAML ain't the only thing - we need EEF as well, because e-YAML cannot save code. But I guess it wouldn't be that pretty in XML either.
Long story short - I still stand by XML because it's more standard then coming up with two of our own formats, even if one of them is a subset, which doesn't mean anything if we are the only ones who implement it.
But in the end I don't really care what you choose (like I mentioned previously), but at least remake the f-ing thing e-YAML+EEF so it at least doesn't break.

We also need a proper specification for EGM, because now it is very ad-hoc. We don't have version information in EGM, so if we do change something we cannot even differentiate. So it is very easy to break it. We also need a lot more delta checks there, so things that are default are not saved. Like every EGM saves an empty .rtf file (still 163 bytes) which in 99.9% cases are not used, because who the hell uses that information thing anyway. It was used during the age of examples back in 2005, but not today.
Title: Re: Extension .egm does not match EGM?
Post by: FroggestSpirit on December 18, 2015, 03:17:11 pm
Is this still being worked on? I might be interested in taking a look at it. I think YAML looks a little cleaner than XML, but it seems that they both do the job. I'm honestly tired of working with old GM formats, so if I could do anything to help speed this up, I'd be glad to.
Title: Re: Extension .egm does not match EGM?
Post by: TheExDeus on February 11, 2016, 07:28:45 am
It is always an open issue. I use EGM exclusively for about a year now as it is the only format in LGM which saves ENIGMA properties (chosen systems, enabled extensions etc.) and so I don't need to re-enable stuff all the time. The rooms broke at one point because of the room data change, but it didn't really influence me as I usually only have one instance per room.
What I currently need more than non-binary room data is extracted format so I can properly version control it in GIT. Basically EGX format. I don't think it would be that hard to do in LGM as it is the same format as EGM but without zip compression. I really badly need this.