Extension .egm does not match EGM?

[edsquare]

Taking advantage of Robert's offer: 

As usual let us know if you have any more problems!

 

What's wrong with the EGM projects?

How can I save as .egm?

[Goombert]

Yeah this is a known bug, when adding multiple constants support I temporarily broke EGM format. Basically me and Josh really aren't sure what to do at this point.

Josh hates the EGM format, it saves rooms as binary blobs like the GMK format and it's barely extensible at all. What he would like is to basically rewrite the whole format with a proper spec and implement everything properly. We have a GitHub ticket filed on this:
https://github.com/enigma-dev/lgmplugin/issues/29

For now you can just use the old LGM if you want, though I don't think it will work with ENIGMA anymore. We really need to get this fixed ASAP but I specifically don't know what to do, I'll have to talk to Josh.
http://enigma-dev.org/docs/Wiki/LateralGM:_Revisions

[edsquare]

Yeah this is a known bug, when adding multiple constants support I temporarily broke EGM format. Basically me and Josh really aren't sure what to do at this point.

Josh hates the EGM format, it saves rooms as binary blobs like the GMK format and it's barely extensible at all. What he would like is to basically rewrite the whole format with a proper spec and implement everything properly. We have a GitHub ticket filed on this:
https://github.com/enigma-dev/lgmplugin/issues/29

For now you can just use the old LGM if you want, though I don't think it will work with ENIGMA anymore. We really need to get this fixed ASAP but I specifically don't know what to do, I'll have to talk to Josh.
http://enigma-dev.org/docs/Wiki/LateralGM:_Revisions

It will work with the enigma version it had, you see before reinstalling al my OS I made a backup of stuff, so I have a working LGM/ENIGMA 1.8.5

The gmx is the one that makes a folder and puts there your scripts, sprites, etc. Right?

Why not take that and improve it and then use that as the new egm?

Where do you set the extensions?

Maybe I'll have a look see

[TheExDeus]

Why can't we serialize the room data in an xml just like GM? How do you save GM files then? XML is extensible by design. Or you can do it in YAML or eYAML or whatever else we got there. I don't really see the problem. EGM itself is also actually quite extensible even now. Everything is in it's own folder and all we need are some XML files showing what is where. So if you add a new resource, you just add it to a xml config file and show it in LGM. If an XML entry is missing, then use a default value (one that would be used if you made a new project with File->New).

[egofree]

Meanwhile, would not be possible to revert the egm file reader and writer to the state it was before you worked on constants ? Even if doesn't support constants.

[Josh @ Dreamland]

XML is the ugliest imaginable format. It's extensible by nature, yes, but it is extremely verbose. And while its hierarchy makes storing code obvious, the problem is simple enough to work around in YAML. As for XML's propensity for hierarchical data, well, rooms are pretty linear in the scheme of things. I posted a proposal somewhere (or maybe I just started coding it and didn't finish) about what I wanted rooms to look like. Consider something like this:

Code: (yaml) [Select]

name: room_0
background-color: 00C0FF
backgrounds:
- name: bkg_3
  depth: 1000
  h-repeat: yes
  v-repeat: no
- name: bkg_overlay
  depth: -1000
  h-repeat: no
  v-repeat: no
views:
- id: 0 # Optional for in-sequence
  visible: yes
  viewport:
    x: 0
    y: 0
    w: 640
    h: 480
  projection:
    x: 0
    y: 0
    w: 640
    h: 480
  follow:
    object: obj_player
    border: [32, 24] # Also allowed: separate lines for h/v
    speed: [4, 4]
object-format:
- id
- object-index
- x
- y
- rotation
object-names:
- obj_player
- obj_dirtwall
- obj_7
objects: [
  [1000001, 0, 64, 400, 0],
  [1000002, 1, 0, 432, 0],
  [1000003, 1, 32, 432, 0],
  [1000004, 1, 64, 432, 0],
  [1000006, 1, 96, 432, 0],
  [1000007, 1, 128, 432, 0],
  [1000008, 1, 160, 432, 0],
  [1000009, 1, 192 432, 0],
  [1000012, 2, 32, 432, 45] # obj_7 is still index 2 on our list, even if its id is really 7.
]
creation-codes:
- ind: 0
  code: |
    // Code begins here
    can_fly = false;
- ind: 8
  code: |
    // Code begins here
    text = "To jump, press [space].";

The "// Code begins here" bit is to prevent issues with code snippets wherein the first line is indented for whatever reason. This is important since YAML is indentation-based. XML deals with this by escaping < and >, which I also disapprove of. Comparatively speaking, the cost of YAML is cheap.

I'll also point out that the verbosity around viewport/projection can theoretically be replaced with [snip=yaml]viewport: {x: 0, y: 0, w: 640, h: 480}[/snip], although e-YAML doesn't presently support that.

[TheExDeus]

You can use attributes in XML though, which greatly reduces verbosity.
Your example in xml:

Code: (xml) [Select]

<room name="room_0" background_color="00C0FF">
  <background name="bkg_3" depth="1000" h-repeat="yes" v-repeat="no"/>
  <background name="bkg_overlay" depth="-1000" h-repeat="no" v-repeat="no"/>
  <view id="0">
    <viewport x="0" y="0" w="640" h="480" />
    <projection x="0" y="0" w="640" h="480" />
    <follow object="obj_player" border_w="32" border_h="24" speed_w="4" speed_h="4" />
  </view>
<object_list>
  <obj name="obj_player" id="0" />
  <obj name="obj_dirtwall" id="1" />
  <obj name="obj_7" id="2" />
<object_list/>
<inst id="1000001" obj="0" x="64" y="400"/>
<inst id="1000002" obj="1" x="0" y="432"/>
<inst id="1000003" obj="1" x="32" y="432"/>
<inst id="1000004" obj="1" x="64" y="432"/>
<inst id="1000006" obj="1" x="128" y="432"/>
<inst id="1000007" obj="1" x="160" y="432"/>
<inst id="1000008" obj="1" x="192" y="432"/>
<inst id="1000009" obj="2" x="32" y="432"/>

<crc ind="1000001">
    can_fly = false;
</crc>

<crc ind="1000008">
    text = "To jump, press [space].";
</crc>
</room>And it can be less verbose than that. I just like xml syntax more than YAML (I hate when tabs count as syntax), but I really don't like any of them.

[edsquare]

Where in the code are the gmk, gmx, etc formats declared, created and set?

[Josh @ Dreamland]

Even with that ugly indent scheme, the XML will be on the order of twice as large for a typical room. The only advantage of the XML is that named attributes allow you to elide attributes with default (probably zero or one) values. If this proves necessary, YAML also allows this using maps. So [snip=xml]<inst id="1000009" obj="2" x="32" y="432"/>[/snip] becomes [snip]{id: 1000009,obj: 2,x: 32,y: 432}[/snip], which is pretty meager savings, but that's still the worst-case, and large rooms will be mostly instances. The ease of parsing of (E)YAML vs XML is also a major feature. And if we're going to go toe-to-toe with full specs, XML will lose outright by YAML's superior support for primitives, node types, and even node references.

Not to mention that the best thing about YAML is that users' code appears verbatim in the format, with the exception that it is indented. I'll grant that it is unlikely for code to contain "</crc>", and unlikely for a given user to want to edit the file manually, and that together this means it's extremely unlikely that a user would want to paste in code with a closing tag, but correctly-formatted code uses HTML entities in place of reserved characters (and there are lots), so in general, a user can't open the room file and copy the code somewhere else, because [snip]if (x &lt; xprevious)[/snip] is not valid EDL.

The comment line we would place in the YAML would be optional; it prevents cases where valid code entered in an IDE would break the format. In the 99 percent case, pasting in a well-formatted piece of code and indenting it twice will suffice, because the first line usually has the least amount of indent. The 99 percent case of the 1 percent case is where someone decided to indent their opening doc comment. They will have to either not do that, or place the opening YAML comment, which we might consider simplifying to, eg, eight or more hyphens.

[TheExDeus]

Actually XML parsing with things like RapidXML takes almost the same amount of cycles as strlen. I believe current YAML implementations take more.
But my beef isn't about the speed or the features we will never use, it's about the whole thing being ugly to use, just like python is ugly to use because of the indent bullshit. But I guess all of that is subjective, so save it as you want. Just make sure every parameters is optional, so EGM never breaks again.

[DaSpirit]

Actually XML parsing with things like RapidXML takes almost the same amount of cycles as strlen. I believe current YAML implementations take more.

I read that too. Parsing XML is easy. Meanwhile, YAML has so many rules that need to be checked. I like TOML best, but it's more for config files than project files.

[TheExDeus]

This is still an open question. EGM rooms are still binary and we need to fix that. I still vote for XML on how fast it is to parse and how "compact" it can actually be. Also, it is quite easy to implement and use. Josh's arguments didn't convince me and I think I gave good examples. But I don't have anything against e-YAML though. We already use it to save object data and we could use it in rooms as well. If this wasn't LGM side I would gladly do it, as I made an XML format for saving and loading in one of my projects and it was trivially easy.
I also added a function to e-YAML parser to return if a value exists, so we could do default values as well. Sadly LGM and the parser/compiler doesn't actually do this checking, so even by missing an optional value we get segfaults. I can fix the parser/compiler part, but not the LGM part.

[Josh @ Dreamland]

If you're interested in XML, you may as well use it for everything. And for that price, you may as well use GMX. My understanding is that the two aren't very disparate.
Of course, LateralGM still (and presumably always) sucks at working with both of those formats, so it's not as though binary room data is our only fish to fry.

[TheExDeus]

I cannot expand GMX without breaking it (which some will try to load in GM:S and fail). I actually don't have anything against it and it is very similar to what I think EGM should be - a zip file packing XML resource descriptors with custom (even binary) data, like images and sound. That is what DOCX is compared to DOC. And a lot of other formats are like that too. And EGM is basically the same thing, but uses a non-standard markup language (e-YAML and some other which I don't even remember) and binary room data. I want everything that can change between versions to be non-binary. I just don't see why we need to create two different markup languages (or in YAML case, an extension) just to save textual data? Historically it has always been good to not reinvent the wheel. Especially if nobody in this project finishes the things that they start (including me in many cases). So better is to use something existing so all of this could be fixed in 3 days, instead of making a new language for 3 years and never finish.

[Goombert]

Harri, the reason for the extension was because it is a subset of YAML, so it's easier to implement. At least that is what Josh told me.

I'm still in favor of XML for GMX and YAML for EGM, otherwise there will be practically no difference between the formats and no advantage of one over the other. Also you should be able to add additional properties to GMX without Studio complaining about the file. Studio will only complain if you remove properties or you change the structure of existing ones. But you can and should be able to add a whole new tag, LGM will be doing this as of the next release to keep the id's in there from a GMK file for backwards compatibility. I've tested this and it seems to load fine in Studio, but if you save the file back, Studio will not save the id tags and will remove them.