Pages: 1 2 »
  Print  
Author Topic: Lateral GM question  (Read 6175 times)
Offline (Unknown gender) lonewolff
Posted on: October 01, 2014, 09:36:00 PM
"Guest"


Email
Hi Guys,

How heavily (if at all) is ENIGMA tied in to LGM?

Is it possible to compile anything in ENIGMA straight from the command line in MinGW for example?
Logged
Offline (Male) Rusky
Reply #1 Posted on: October 01, 2014, 10:15:36 PM

Resident Troll
Joined: Feb 2008
Posts: 955
MSN Messenger - rpjohnst@gmail.com
View Profile WWW Email
ENIGMA is a plugin to LateralGM. LGM loads the game file and passes that information on to ENIGMA, which then does its own thing.

Robert did some work on getting it to compile from the command line; I don't recall how much of LGM he used or if it still works: http://enigma-dev.org/forums/index.php?topic=2047
Logged
Offline (Unknown gender) Darkstar2
Reply #2 Posted on: October 01, 2014, 10:17:45 PM
Member
Joined: Jan 2014
Posts: 1244

View Profile Email
The IDE interfaces using the PLUGIN enigma.jar.  As I explained to you in PM, the one flaw of LGM is that it saves the project into one file.....that is the file that resides in memory and from memory gets processed by the plugin..........LOL - the proper way to do it would be to save your project like you would in GMS, using folders, and having the IDE compile reading from disk......it would be far more efficient, smaller memory footprint, and more stable........a fully working CLI would also open the doors for easy IDE building.  You CAN use LGM without the ENIGMA plugin, for editing your project, but to compile and build you need enigma.jar which is a plugin to LGM.
Logged
Offline (Male) Goombert
Reply #3 Posted on: October 01, 2014, 10:18:41 PM

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3110

View Profile
LGM is entirely its own project, many people who don't chose to use ENIGMA still use LGM to restore corrupted files from GM8. The command line is still working but it may be broke atm from the adding preCreationCode because I don't think they took the time to add it to my CLI, but I am planning to go fix that sometime and it is able to build a number of GMX projects without LateralGM


Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) egofree
Reply #4 Posted on: October 02, 2014, 02:22:32 AM
Contributor
Joined: Jun 2013
Posts: 603

View Profile Email
As I explained to you in PM, the one flaw of LGM is that it saves the project into one file

You are talking about the egm format, but you can save also the project as a gmx project.
Logged
Offline (Unknown gender) TheExDeus
Reply #5 Posted on: October 02, 2014, 03:39:53 AM

Developer
Joined: Apr 2008
Posts: 1872

View Profile
The only thing LGM does is give information to ENIGMA's parser (the .dll) which then generates the code which is then compiled. This is required because ENIGMA itself cannot open project files (gmk, gmx, egm, etc.) and interpret the information in them. Robert made a CLI that loads gmx (and maybe egm), and talks to the .dll directly. This means LGM is not actually required for ENIGMA to work and ENIGMA isn't required for LGM to work.
Logged
Offline (Female) IsmAvatar
Reply #6 Posted on: October 02, 2014, 09:27:42 AM

LateralGM Developer
LGM Developer
Location: Pennsylvania/USA
Joined: Apr 2008
Posts: 886

View Profile Email
I'd just like to share a bit of history here. (Author's note. After writing this, I realized that a lot of it is already on the wiki page for CLI)
When Josh and I teamed up, we investigated a few ways to open communication between the two. Initially, I had a really hard time trying to figure out how to set up ENIGMA as a dll and actually send information to it because Java's linking support (JNI) was woefully inadequate (and I doubt it's improved) without the use of an additional library like JNA.
So we decided to make ENIGMA an exe, save the game as a file in our own (temporarily) proprietary format, then launch ENIGMA with that filename. The question then becomes how does LateralGM make sense of what ENIGMA wants to tell the user so that LateralGM can present it in a convenient format (such as highlighting problem lines, or pulling out errors from warnings). One possibility was to use files, which I think we tried briefly, but it was too slow and hard to manage, and it was difficult for ENIGMA to communicate when it was done writing to the file so LGM knew it was allowed to read it, so we gave that idea up and had LateralGM read the stdout and stderr from ENIGMA and parse it.
That was about as much fun as trying to read a message written in blood.

Then one day JNA got its act together and started hitting the gym. LGM was able to use it because it wasn't as fat as it used to be, and could be made even skinnier by me performing a little liposuction on it. This gave me an opportunity to experiment with talking with ENIGMA as a dll rather than an exe.

The functions that evolved from this are documented on the wiki and a specification was written up for it. Any IDE can use these to interface with ENIGMA. You could use these functions to create your own CLI for ENIGMA.

In fact, shortly after that, I wrote a ENIGMA CLI (Command Line Interface) and built it into The Plugin so that you could actually run the plugin without LateralGM, as a CLI, and simply pass it a file (which was one of the driving forces and use cases behind the EGM format) and some command line flags/arguments and have it compiled. I continued to periodically maintain the CLI throughout my active participation in the project. I can't speak for how it has fared since I left, but I would be willing to bet that the code is still in there and it can't be too hard to fix it up and get it working again if it broke.


Quote from: Darkstar2
the proper way to do it would be to save your project like you would in GMS, using folders, and having the IDE compile reading from disk......it would be far more efficient, smaller memory footprint, and more stable
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
« Last Edit: October 02, 2014, 09:32:42 AM by IsmAvatar » Logged
Offline (Male) Goombert
Reply #7 Posted on: October 02, 2014, 05:00:11 PM

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3110

View Profile
Quote from: IsmAvatar
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
Some of the confusion was promulgated by me IsmAvatar. The only performance gains that it would offer would be because the CLI itself would be native C++, but I do disagree with you on the front that having them always on the disk will improve performance because LGM only has to write the resources that have changed. If LGM itself was native C++, having an intermediate CLI in C++ would not improve performance, at all, period. So yes I want to thank you for pointing out some misconceptions there.

Also, she is right about Build Mode, but I still believe it could be accomplished with the use of a CLI and the combination of a library, which is what me and Josh would like to attempt. Also for all intents and purposes I do not like calling libraries dll's or shared objects, dll is not a platform agnostic term.
« Last Edit: October 02, 2014, 05:03:30 PM by Robert B Colton » Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) Darkstar2
Reply #8 Posted on: October 03, 2014, 12:38:19 AM
Member
Joined: Jan 2014
Posts: 1244

View Profile Email
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game

I disagree here, unless you still have an old PATA drive working at 20-30MB/s :D now people use SATA2/3 with blazzing speeds, even SSD.  The advantage of disk to disk
compiling is efficiency - for smaller games, the tradeoff is minimal, but for massive/larger games, the tradeoff is worth it.  For developers wanting to make massive games, would be more efficient that way.
In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time :D

I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros.

Quote
in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.

I'm sorry but LGM is very annoying to work with and highly unstable.  I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal.
Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this. :D

My solution to this soon will be my written from scratch C++ dynamic asset handling engine + some changes.......but barely have time now !

[/quote]
Logged
Offline (Male) Goombert
Reply #9 Posted on: October 03, 2014, 03:10:19 AM

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3110

View Profile
Quote from: Darkstar2
In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time
Their improvements are limited to about the same as Java's, they use Delphi.NET, while being a less popular language is likely to not be as powerful as Java which is more regularly maintained for multiple platforms. Any performance gain they make with their IDE written in a managed language will not be much greater than any gain LGM will have by doing the same. Also they do not do exactly as I proposed, they also copy the project and use a disk asset cache. For passing small files LGM may actually be faster going through memory with JNA than Studio in that particular case, evidence for this is that mainly large binary images and what not are what truly slows LGM down. Additionally we don't have a half-assed programming language, projects with ENIMGA have to be parsed by all of JDI and GCC, which albeit is most likely lighter than Clang/LLVM

At any rate you keep missing the point that having a purely native IDE itself, passing through memory is still quicker, the disk is only useful for IDE's written in managed languages.

Quote from: Darkstar2
I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros.
Examine the following screenshot, latest Studio (I literally just installed updates), open a blank project, open LGM with Project Chaos, and LGM seems to be only a few MB's ahead in memory usage (the Project Chaos file is 12.5MB's; mainly the result of OGG audio files)


Quote from: Darkstar2
I'm sorry but LGM is very annoying to work with and highly unstable.  I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal.
Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this.
You should start reporting these crashes more frequently so that we can determine the cause, I've only heard you mention of them but I've not once received an error report with a log and stack trace even though one is provided in a dialog to you for every exception.
« Last Edit: October 03, 2014, 03:21:10 AM by Robert B Colton » Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Unknown gender) TheExDeus
Reply #10 Posted on: October 03, 2014, 06:09:19 AM

Developer
Joined: Apr 2008
Posts: 1872

View Profile
I also think saving to disk and then using CLI could be faster, because writing to disk really isn't a slow operation. Even a larger project in LGM saves to disk in less than 2-3 seconds. And even then I'm sure it's no the speed of the drive, but the Java's code which is doing conversion. Imagine if LGM only saved files that have changed. Then it wouldn't matter at all. My idea on how this would work is this for EGM:
You open .egm project. LGM loads it in ram.
You run it. LGM makes a copy of the project into a temp directory (a copy operation should be really fast). Then it runs CLI on it.
You make changes to an object and sprite, but not save the project.
Run it again. LGM modifies the appropriate files in .egm which is in the temp directory. This could only be a change of few kb. Then it runs CLI on it again.

Now if you loaded non-egm format. Then it would require a conversion. Like so:
You open .gmx project. LGM loads it in ram.
You run it. LGM converts it to .egm and saves in temp folder. This will be slower than a copy, but not very much. Then it run CLI on it.
You make changes to an object and sprite, but not save the project.
Run it again. LGM modifies the appropriate files in .egm which is in the temp directory. This could only be a change of few kb. Then it runs CLI on it again.
You then save it. LGM saves to the .gmx like it does it now. gmx should allow saving only changes as well, while gmk and other older formats would require writing the whole file. Isn't much of a problem though.
Logged
Offline (Male) Goombert
Reply #11 Posted on: October 03, 2014, 06:22:10 AM

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3110

View Profile
Yes I agree as well Harri, but it wouldn't offer much over just a native IDE, but of course you should only load files as they are opened in editors instead of in bulk, like every other IDE. I was also planning on just making you have to save before running the game to get changes, because of course you could still revert the frames if you like since this behavior has been fixed in the new LGM release, this also is like many other IDE's like event Studio which makes you save before changes are reflected
Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Offline (Female) IsmAvatar
Reply #12 Posted on: October 03, 2014, 09:30:57 AM

LateralGM Developer
LGM Developer
Location: Pennsylvania/USA
Joined: Apr 2008
Posts: 886

View Profile Email
I'm seeing a bit of disagreement over what I said. I'd like to clarify.
I'm certainly not arguing against a CLI. As I said, not only did I write a CLI myself, but I did pretty much all of the work on making the API that opened the doors for a CLI in the first place.
My point was that it makes more sense for LateralGM to use ENIGMA as a library rather than an executable, and I outlined several reasons. The main reason is the interactive feedback that ENIGMA can give to LateralGM. That stuff *could* be done on a command line, sure, but we already tried that and discovered it was a really really bad idea - and the whole deal with files vs memory was really quite minor in that discovery. Library calls are just all around cleaner. You don't need a parser, and you don't need to keep updating that parser with more complexity whenever you change something. It also affords us the ability to pass callbacks to each other, meaning that ENIGMA not only interacts with LateralGM, but it opens up certain LGM abilities to ENIGMA (such as opening, populating, and closing informational windows like console logs, errors, and progress dialogs). Again, if you try to do that with files, you need some sort of clever dirty way of detecting when the file is done.

As far as GMS and EGM goes, I'm glad that file operations are getting fast enough, and I'm glad that these formats give us individual resource saving/loading. This certainly does open the door for a lot of new things with CLI and passing files, which I believe we can make even better use of through library calls.


@TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things:
* In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory.
* While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory.
* The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files.
Logged
Offline (Unknown gender) TheExDeus
Reply #13 Posted on: October 03, 2014, 10:57:43 AM

Developer
Joined: Apr 2008
Posts: 1872

View Profile
I guess the only concern for me is to make 64bit working with the dll. Because some people (like sorlok) cannot compile large projects on windows, because he runs out of memory and the whole thing crashes. Thought maybe CLI would help in that, because it wouldn't share memory with LGM. On the other hand of course it's better if it did share memory, but as you pointed out, LGM makes a copy for the dll anyway. So I guess we could optimize on that front.
Logged
Offline (Male) Goombert
Reply #14 Posted on: October 03, 2014, 10:51:13 PM

Developer
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3110

View Profile
IsmAvatar, the first whole part of your post I think we understood just fine, nobody is advocating for getting rid of the callbacks, no way at all.

Quote from: ismAvatar
@TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things:
* In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory.
* While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory.
* The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files.
Here's where you seem to be kind of confused about our proposal though, that even Josh wants. If the project exists on disk, you don't have to load anything, the user clicks a resource and it is loaded and then they can change it in the editor, when you hit the save button all changed resources are committed to disk, and we would require you to save before changes would be reflected at runtime. This is like many other IDE's, but it would make LGM much better at working with large project files, even a native IDE should not have resources loaded that it is not concurrently working with or the user has any interest in. Does that make sense? But these changes would mean abandoning the GMK format because it is not easily expandable for this functionality, but EGM being ZIP's and GMX's being zip friendly those formats can do it.

But basically the reason we haven't done this yet is because I've wanted to make LGM very very stable with some awesome features, such as the search tree, before any GMK support is dropped. LGM in its current state is basically becoming better than GM8.1 or GM ever was as an IDE pretty much.
Logged
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.

Pages: 1 2 »
  Print