ENIGMA Forums

General fluff => General ENIGMA => Topic started by: lonewolff on October 01, 2014, 09:36:00 PM

Title: Lateral GM question
Post by: lonewolff on October 01, 2014, 09:36:00 PM
Hi Guys,

How heavily (if at all) is ENIGMA tied in to LGM?

Is it possible to compile anything in ENIGMA straight from the command line in MinGW for example?
Title: Re: Lateral GM question
Post by: Rusky on October 01, 2014, 10:15:36 PM
ENIGMA is a plugin to LateralGM. LGM loads the game file and passes that information on to ENIGMA, which then does its own thing.

Robert did some work on getting it to compile from the command line; I don't recall how much of LGM he used or if it still works: http://enigma-dev.org/forums/index.php?topic=2047
Title: Re: Lateral GM question
Post by: Darkstar2 on October 01, 2014, 10:17:45 PM
The IDE interfaces using the PLUGIN enigma.jar.  As I explained to you in PM, the one flaw of LGM is that it saves the project into one file.....that is the file that resides in memory and from memory gets processed by the plugin..........LOL - the proper way to do it would be to save your project like you would in GMS, using folders, and having the IDE compile reading from disk......it would be far more efficient, smaller memory footprint, and more stable........a fully working CLI would also open the doors for easy IDE building.  You CAN use LGM without the ENIGMA plugin, for editing your project, but to compile and build you need enigma.jar which is a plugin to LGM.
Title: Re: Lateral GM question
Post by: Goombert on October 01, 2014, 10:18:41 PM
LGM is entirely its own project, many people who don't chose to use ENIGMA still use LGM to restore corrupted files from GM8. The command line is still working but it may be broke atm from the adding preCreationCode because I don't think they took the time to add it to my CLI, but I am planning to go fix that sometime and it is able to build a number of GMX projects without LateralGM

(http://i.imgur.com/CMrXnmN.jpg)(http://i.imgur.com/F8Y4njD.png)
(http://i.imgur.com/CEaXL7x.jpg)
Title: Re: Lateral GM question
Post by: egofree on October 02, 2014, 02:22:32 AM
As I explained to you in PM, the one flaw of LGM is that it saves the project into one file

You are talking about the egm format, but you can save also the project as a gmx project.
Title: Re: Lateral GM question
Post by: TheExDeus on October 02, 2014, 03:39:53 AM
The only thing LGM does is give information to ENIGMA's parser (the .dll) which then generates the code which is then compiled. This is required because ENIGMA itself cannot open project files (gmk, gmx, egm, etc.) and interpret the information in them. Robert made a CLI that loads gmx (and maybe egm), and talks to the .dll directly. This means LGM is not actually required for ENIGMA to work and ENIGMA isn't required for LGM to work.
Title: Re: Lateral GM question
Post by: IsmAvatar on October 02, 2014, 09:27:42 AM
I'd just like to share a bit of history here. (Author's note. After writing this, I realized that a lot of it is already on the wiki page for CLI (http://enigma-dev.org/docs/Wiki/CLI))
When Josh and I teamed up, we investigated a few ways to open communication between the two. Initially, I had a really hard time trying to figure out how to set up ENIGMA as a dll and actually send information to it because Java's linking support (JNI) was woefully inadequate (and I doubt it's improved) without the use of an additional library like JNA.
So we decided to make ENIGMA an exe, save the game as a file in our own (temporarily) proprietary format, then launch ENIGMA with that filename. The question then becomes how does LateralGM make sense of what ENIGMA wants to tell the user so that LateralGM can present it in a convenient format (such as highlighting problem lines, or pulling out errors from warnings). One possibility was to use files, which I think we tried briefly, but it was too slow and hard to manage, and it was difficult for ENIGMA to communicate when it was done writing to the file so LGM knew it was allowed to read it, so we gave that idea up and had LateralGM read the stdout and stderr from ENIGMA and parse it.
That was about as much fun as trying to read a message written in blood.

Then one day JNA got its act together and started hitting the gym. LGM was able to use it because it wasn't as fat as it used to be, and could be made even skinnier by me performing a little liposuction on it. This gave me an opportunity to experiment with talking with ENIGMA as a dll rather than an exe.

The functions that evolved from this are documented on the wiki (http://enigma-dev.org/docs/Wiki/EnigmaDriver) and a specification (http://enigma-dev.org/docs/Wiki/ENIGMA-IDE_interface) was written up for it. Any IDE can use these to interface with ENIGMA. You could use these functions to create your own CLI for ENIGMA.

In fact, shortly after that, I wrote a ENIGMA CLI (Command Line Interface) and built it into The Plugin (http://enigma-dev.org/docs/Wiki/Plugin) so that you could actually run the plugin without LateralGM, as a CLI, and simply pass it a file (which was one of the driving forces and use cases behind the EGM format) and some command line flags/arguments and have it compiled. I continued to periodically maintain the CLI throughout my active participation in the project. I can't speak for how it has fared since I left, but I would be willing to bet that the code is still in there and it can't be too hard to fix it up and get it working again if it broke.


Quote from: Darkstar2
the proper way to do it would be to save your project like you would in GMS, using folders, and having the IDE compile reading from disk......it would be far more efficient, smaller memory footprint, and more stable
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
Title: Re: Lateral GM question
Post by: Goombert on October 02, 2014, 05:00:11 PM
Quote from: IsmAvatar
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
Some of the confusion was promulgated by me IsmAvatar. The only performance gains that it would offer would be because the CLI itself would be native C++, but I do disagree with you on the front that having them always on the disk will improve performance because LGM only has to write the resources that have changed. If LGM itself was native C++, having an intermediate CLI in C++ would not improve performance, at all, period. So yes I want to thank you for pointing out some misconceptions there.

Also, she is right about Build Mode, but I still believe it could be accomplished with the use of a CLI and the combination of a library, which is what me and Josh would like to attempt. Also for all intents and purposes I do not like calling libraries dll's or shared objects, dll is not a platform agnostic term.
Title: Re: Lateral GM question
Post by: Darkstar2 on October 03, 2014, 12:38:19 AM
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game

I disagree here, unless you still have an old PATA drive working at 20-30MB/s :D now people use SATA2/3 with blazzing speeds, even SSD.  The advantage of disk to disk
compiling is efficiency - for smaller games, the tradeoff is minimal, but for massive/larger games, the tradeoff is worth it.  For developers wanting to make massive games, would be more efficient that way.
In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time :D

I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros.

Quote
in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.

I'm sorry but LGM is very annoying to work with and highly unstable.  I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal.
Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this. :D

My solution to this soon will be my written from scratch C++ dynamic asset handling engine + some changes.......but barely have time now !

[/quote]
Title: Re: Lateral GM question
Post by: Goombert on October 03, 2014, 03:10:19 AM
Quote from: Darkstar2
In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time
Their improvements are limited to about the same as Java's, they use Delphi.NET, while being a less popular language is likely to not be as powerful as Java which is more regularly maintained for multiple platforms. Any performance gain they make with their IDE written in a managed language will not be much greater than any gain LGM will have by doing the same. Also they do not do exactly as I proposed, they also copy the project and use a disk asset cache. For passing small files LGM may actually be faster going through memory with JNA than Studio in that particular case, evidence for this is that mainly large binary images and what not are what truly slows LGM down. Additionally we don't have a half-assed programming language, projects with ENIMGA have to be parsed by all of JDI and GCC, which albeit is most likely lighter than Clang/LLVM

At any rate you keep missing the point that having a purely native IDE itself, passing through memory is still quicker, the disk is only useful for IDE's written in managed languages.

Quote from: Darkstar2
I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros.
Examine the following screenshot, latest Studio (I literally just installed updates), open a blank project, open LGM with Project Chaos, and LGM seems to be only a few MB's ahead in memory usage (the Project Chaos file is 12.5MB's; mainly the result of OGG audio files)
(http://i.imgur.com/bUO0Z2O.png)

Quote from: Darkstar2
I'm sorry but LGM is very annoying to work with and highly unstable.  I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal.
Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this.
You should start reporting these crashes more frequently so that we can determine the cause, I've only heard you mention of them but I've not once received an error report with a log and stack trace even though one is provided in a dialog to you for every exception.
Title: Re: Lateral GM question
Post by: TheExDeus on October 03, 2014, 06:09:19 AM
I also think saving to disk and then using CLI could be faster, because writing to disk really isn't a slow operation. Even a larger project in LGM saves to disk in less than 2-3 seconds. And even then I'm sure it's no the speed of the drive, but the Java's code which is doing conversion. Imagine if LGM only saved files that have changed. Then it wouldn't matter at all. My idea on how this would work is this for EGM:
You open .egm project. LGM loads it in ram.
You run it. LGM makes a copy of the project into a temp directory (a copy operation should be really fast). Then it runs CLI on it.
You make changes to an object and sprite, but not save the project.
Run it again. LGM modifies the appropriate files in .egm which is in the temp directory. This could only be a change of few kb. Then it runs CLI on it again.

Now if you loaded non-egm format. Then it would require a conversion. Like so:
You open .gmx project. LGM loads it in ram.
You run it. LGM converts it to .egm and saves in temp folder. This will be slower than a copy, but not very much. Then it run CLI on it.
You make changes to an object and sprite, but not save the project.
Run it again. LGM modifies the appropriate files in .egm which is in the temp directory. This could only be a change of few kb. Then it runs CLI on it again.
You then save it. LGM saves to the .gmx like it does it now. gmx should allow saving only changes as well, while gmk and other older formats would require writing the whole file. Isn't much of a problem though.
Title: Re: Lateral GM question
Post by: Goombert on October 03, 2014, 06:22:10 AM
Yes I agree as well Harri, but it wouldn't offer much over just a native IDE, but of course you should only load files as they are opened in editors instead of in bulk, like every other IDE. I was also planning on just making you have to save before running the game to get changes, because of course you could still revert the frames if you like since this behavior has been fixed in the new LGM release, this also is like many other IDE's like event Studio which makes you save before changes are reflected
Title: Re: Lateral GM question
Post by: IsmAvatar on October 03, 2014, 09:30:57 AM
I'm seeing a bit of disagreement over what I said. I'd like to clarify.
I'm certainly not arguing against a CLI. As I said, not only did I write a CLI myself, but I did pretty much all of the work on making the API that opened the doors for a CLI in the first place.
My point was that it makes more sense for LateralGM to use ENIGMA as a library rather than an executable, and I outlined several reasons. The main reason is the interactive feedback that ENIGMA can give to LateralGM. That stuff *could* be done on a command line, sure, but we already tried that and discovered it was a really really bad idea - and the whole deal with files vs memory was really quite minor in that discovery. Library calls are just all around cleaner. You don't need a parser, and you don't need to keep updating that parser with more complexity whenever you change something. It also affords us the ability to pass callbacks to each other, meaning that ENIGMA not only interacts with LateralGM, but it opens up certain LGM abilities to ENIGMA (such as opening, populating, and closing informational windows like console logs, errors, and progress dialogs). Again, if you try to do that with files, you need some sort of clever dirty way of detecting when the file is done.

As far as GMS and EGM goes, I'm glad that file operations are getting fast enough, and I'm glad that these formats give us individual resource saving/loading. This certainly does open the door for a lot of new things with CLI and passing files, which I believe we can make even better use of through library calls.


@TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things:
* In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory.
* While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory.
* The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files.
Title: Re: Lateral GM question
Post by: TheExDeus on October 03, 2014, 10:57:43 AM
I guess the only concern for me is to make 64bit working with the dll. Because some people (like sorlok) cannot compile large projects on windows, because he runs out of memory and the whole thing crashes. Thought maybe CLI would help in that, because it wouldn't share memory with LGM. On the other hand of course it's better if it did share memory, but as you pointed out, LGM makes a copy for the dll anyway. So I guess we could optimize on that front.
Title: Re: Lateral GM question
Post by: Goombert on October 03, 2014, 10:51:13 PM
IsmAvatar, the first whole part of your post I think we understood just fine, nobody is advocating for getting rid of the callbacks, no way at all.

Quote from: ismAvatar
@TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things:
* In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory.
* While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory.
* The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files.
Here's where you seem to be kind of confused about our proposal though, that even Josh wants. If the project exists on disk, you don't have to load anything, the user clicks a resource and it is loaded and then they can change it in the editor, when you hit the save button all changed resources are committed to disk, and we would require you to save before changes would be reflected at runtime. This is like many other IDE's, but it would make LGM much better at working with large project files, even a native IDE should not have resources loaded that it is not concurrently working with or the user has any interest in. Does that make sense? But these changes would mean abandoning the GMK format because it is not easily expandable for this functionality, but EGM being ZIP's and GMX's being zip friendly those formats can do it.

But basically the reason we haven't done this yet is because I've wanted to make LGM very very stable with some awesome features, such as the search tree, before any GMK support is dropped. LGM in its current state is basically becoming better than GM8.1 or GM ever was as an IDE pretty much.
Title: Re: Lateral GM question
Post by: Darkstar2 on October 03, 2014, 11:31:39 PM
Robert, what are you talking about ? I have posted many times logs since I joined here, many times we discussed the stability issues of LGM and showed you reports, and you said the problem is with enigma.jar (the plugin) and not the IDE itself, these problems start to happen eventually - the longer I work in LGM, eventually pressing anything and an error pops up relating to Java, VM, memory, etc.
And yes even with the settings file tweak setting memory to max.

Some are so easy to reproduce, just frequently save your project, eventually it will error out and the entire MFing project gets corrupted., including backups.

I'd be glad to post those logs, but that won't change anything :D

Title: Re: Lateral GM question
Post by: Goombert on October 03, 2014, 11:43:27 PM
Quote from: Darkstar2
and you said the problem is with enigma.jar (the plugin) and not the IDE itself, these problems start to happen eventually - the longer I work in LGM,
Right exactly, that's the plugin, ENIGMA, not LateralGM. I only notice it when editing ENIGMA source engine files though, not just having LGM open or compiling multiple times, if I don't change source files I have no issues. Also I only seem to have a crash when mainly working with templates or STL because of how big the error logs are and it gets worse over time before crashing. That's an issue I'd like to work out.

Quote from: Darkstar2
Some are so easy to reproduce, just frequently save your project, eventually it will error out and the entire MFing project gets corrupted., including backups.
I literally saved Project Chaos over and over and over and over about a hundred times, and the JVM never tops 305mb RAM usage, just stays between 299 and 305.
(http://i.imgur.com/e2tF3Bq.png)
Title: Re: Lateral GM question
Post by: Darkstar2 on October 03, 2014, 11:53:06 PM
I know that Robert, actually memory usage is not an issue but the error shows VM errors, out of memory, not found, etc.
I will post more examples when I get them.
I have not worked in ENIGMA for a while as I was busy lately.
Title: Re: Lateral GM question
Post by: Goombert on October 04, 2014, 12:04:54 AM
Alrighty, but yeah lots of bug fixes have been made, and I can't fix things without knowing about them... so how else will they ever get fixed? etc etc
Title: Re: Lateral GM question
Post by: IsmAvatar on October 06, 2014, 01:26:29 PM
Is dropping LGM's GMK support even necessary? Couldn't we just drop ENIGMA's GMK support (which is pretty much dropped at this point anyways) and just have the plugin convert your game from GMK to EGM?
Title: Re: Lateral GM question
Post by: Goombert on October 06, 2014, 01:56:22 PM
Heh well, we actually aren't all that incompatible with GMK's, a major portion of games compatible are in GMK format, like Project Mario, and Project Chaos, Minecraft Example, Mark Overmars FPS, Overmars Game of Life. Some games also use properties that GMX does not store, such as removing the transparency color of sprites. Converting these games to another format would break some things here and there. We also just happen to be a lot more compatible with GMK games because they get rid of all the bad coding techniques we've yet to implement, but also add a ton of new ones, see ds accessors.
Title: Re: Lateral GM question
Post by: Josh @ Dreamland on October 08, 2014, 06:54:38 PM
That's the plan, Ism. But make no mistake—ENIGMA is best to remain a DLL.

ENIGMA being compilable as a DLL is not a reason to not have a CLI. In general, command-line interfaces are not nearly as responsive, and we're dealing with software in which this latency counts. Originally, you pressed a syntax check button when you wanted to know if your GML was valid. Would you want a disk I/O over an entire directory while we read in script and function names, just so we can report to you that this one code you've changed two lines of is valid? Maybe you do, I don't know. But do you want us implementing continual syntax checking that way? You'd have to be crazy. Even if we cached all the engine data to a file, you wouldn't want us reading that behemoth every time you stopped typing for a second.

Moreover, what if LateralGM wants to invoke the compiler for more accurate code completion? For formatting? For automatic indentation? If ENIGMA were confined to a separate executable, even with pipes, I/O overhead would be insufferable.

Anyway. That said, yes; EGM and GMX eliminate the need to ever have all resource data loaded in memory. This means that the reasons for having LGM pass ENIGMA resource data are dying or dead. THAT SAID, the ability for the engine to work out of EGM files needs to come first. I will not myself implement nor encourage any other individual to implement logic for reading an EGM and appending it to an executable at this phase. My recommendation is for the ENIGMA plugin to keep the game saved in /tmp/lateralgm/<datetime>/game.egm/* as an EGM directory (presently not supported for some reason?) and for the game to load resources out of that directly. Writing resources to the executable is an archaic practice that should only be offered as a final redistributable build option.