lonewolff
|
|
Posted on: October 01, 2014, 09:36:00 pm |
|
|
"Guest"
|
Hi Guys,
How heavily (if at all) is ENIGMA tied in to LGM?
Is it possible to compile anything in ENIGMA straight from the command line in MinGW for example?
|
|
|
Logged
|
|
|
|
|
|
|
|
|
IsmAvatar
|
|
Reply #6 Posted on: October 02, 2014, 09:27:42 am |
|
|
LateralGM Developer
Location: Pennsylvania/USA Joined: Apr 2008
Posts: 877
|
I'd just like to share a bit of history here. (Author's note. After writing this, I realized that a lot of it is already on the wiki page for CLI) When Josh and I teamed up, we investigated a few ways to open communication between the two. Initially, I had a really hard time trying to figure out how to set up ENIGMA as a dll and actually send information to it because Java's linking support (JNI) was woefully inadequate (and I doubt it's improved) without the use of an additional library like JNA. So we decided to make ENIGMA an exe, save the game as a file in our own (temporarily) proprietary format, then launch ENIGMA with that filename. The question then becomes how does LateralGM make sense of what ENIGMA wants to tell the user so that LateralGM can present it in a convenient format (such as highlighting problem lines, or pulling out errors from warnings). One possibility was to use files, which I think we tried briefly, but it was too slow and hard to manage, and it was difficult for ENIGMA to communicate when it was done writing to the file so LGM knew it was allowed to read it, so we gave that idea up and had LateralGM read the stdout and stderr from ENIGMA and parse it. That was about as much fun as trying to read a message written in blood. Then one day JNA got its act together and started hitting the gym. LGM was able to use it because it wasn't as fat as it used to be, and could be made even skinnier by me performing a little liposuction on it. This gave me an opportunity to experiment with talking with ENIGMA as a dll rather than an exe. The functions that evolved from this are documented on the wiki and a specification was written up for it. Any IDE can use these to interface with ENIGMA. You could use these functions to create your own CLI for ENIGMA. In fact, shortly after that, I wrote a ENIGMA CLI (Command Line Interface) and built it into The Plugin so that you could actually run the plugin without LateralGM, as a CLI, and simply pass it a file (which was one of the driving forces and use cases behind the EGM format) and some command line flags/arguments and have it compiled. I continued to periodically maintain the CLI throughout my active participation in the project. I can't speak for how it has fared since I left, but I would be willing to bet that the code is still in there and it can't be too hard to fix it up and get it working again if it broke. the proper way to do it would be to save your project like you would in GMS, using folders, and having the IDE compile reading from disk......it would be far more efficient, smaller memory footprint, and more stable Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
|
|
« Last Edit: October 02, 2014, 09:32:42 am by IsmAvatar »
|
Logged
|
|
|
|
Goombert
|
|
Reply #7 Posted on: October 02, 2014, 05:00:11 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM. Some of the confusion was promulgated by me IsmAvatar. The only performance gains that it would offer would be because the CLI itself would be native C++, but I do disagree with you on the front that having them always on the disk will improve performance because LGM only has to write the resources that have changed. If LGM itself was native C++, having an intermediate CLI in C++ would not improve performance, at all, period. So yes I want to thank you for pointing out some misconceptions there. Also, she is right about Build Mode, but I still believe it could be accomplished with the use of a CLI and the combination of a library, which is what me and Josh would like to attempt. Also for all intents and purposes I do not like calling libraries dll's or shared objects, dll is not a platform agnostic term.
|
|
« Last Edit: October 02, 2014, 05:03:30 pm by Robert B Colton »
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
Darkstar2
|
|
Reply #8 Posted on: October 03, 2014, 12:38:19 am |
|
|
Joined: Jan 2014
Posts: 1238
|
Not quite. Disk operations are extremely expensive - they are slow, so there goes your "efficiency" argument. For older GM games (pre-GMX), games *had* to be loaded entirely into memory. Having ENIGMA as a DLL allowed us to share that memory -- compared to a file where both ENIGMA and LGM would keep separate copies of the game
I disagree here, unless you still have an old PATA drive working at 20-30MB/s now people use SATA2/3 with blazzing speeds, even SSD. The advantage of disk to disk compiling is efficiency - for smaller games, the tradeoff is minimal, but for massive/larger games, the tradeoff is worth it. For developers wanting to make massive games, would be more efficient that way. In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros. in memory. As for stability, read my history above. In addition, streams are buffered, and if any buffer gets too large before LGM got a chance to read it, the entire JVM would lock up and LGM would freeze. Additionally, parsing text from ENIGMA is hardly stable and rarely yields anything useful. Having specific memory structures that LGM and ENIGMA pass back and forth allows ENIGMA to not only give LGM more useful information (see above), but also affords us some really neat features, like Build Mode, where the compiler can actually edit the game (or allow the user to edit the game) and send that information back to LGM.
I'm sorry but LGM is very annoying to work with and highly unstable. I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal. Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this. My solution to this soon will be my written from scratch C++ dynamic asset handling engine + some changes.......but barely have time now ! [/quote]
|
|
|
Logged
|
|
|
|
Goombert
|
|
Reply #9 Posted on: October 03, 2014, 03:10:19 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
In any case GMS now does it that way, leaves minimal MEMORY footprint, keeps your IDE fast and more stable........Try to work with massive projects and see how slow the IDE gets, this goes the same for GMS......though they made some very good improvements there in terms of dynamic asset managements.....was about damn time Their improvements are limited to about the same as Java's, they use Delphi.NET, while being a less popular language is likely to not be as powerful as Java which is more regularly maintained for multiple platforms. Any performance gain they make with their IDE written in a managed language will not be much greater than any gain LGM will have by doing the same. Also they do not do exactly as I proposed, they also copy the project and use a disk asset cache. For passing small files LGM may actually be faster going through memory with JNA than Studio in that particular case, evidence for this is that mainly large binary images and what not are what truly slows LGM down. Additionally we don't have a half-assed programming language, projects with ENIMGA have to be parsed by all of JDI and GCC, which albeit is most likely lighter than Clang/LLVM At any rate you keep missing the point that having a purely native IDE itself, passing through memory is still quicker, the disk is only useful for IDE's written in managed languages. I am aware that RAM speed exceeds that of disk by far but in this case the tradeoff has its pros. Examine the following screenshot, latest Studio (I literally just installed updates), open a blank project, open LGM with Project Chaos, and LGM seems to be only a few MB's ahead in memory usage (the Project Chaos file is 12.5MB's; mainly the result of OGG audio files) I'm sorry but LGM is very annoying to work with and highly unstable. I often have to close and re-open due to segfaults, VM memory errors, etc......My system is powerful, up to date, heaps of RAM, and yes I tweaked the settings file to the maximum xmx allowed for java.......same deal. Working with massive projects with LGM is a turn off, sadly........GMS's IDE improvements set it miles apart - never thought I'd say this. You should start reporting these crashes more frequently so that we can determine the cause, I've only heard you mention of them but I've not once received an error report with a log and stack trace even though one is provided in a dialog to you for every exception.
|
|
« Last Edit: October 03, 2014, 03:21:10 am by Robert B Colton »
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
|
Goombert
|
|
Reply #11 Posted on: October 03, 2014, 06:22:10 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
Yes I agree as well Harri, but it wouldn't offer much over just a native IDE, but of course you should only load files as they are opened in editors instead of in bulk, like every other IDE. I was also planning on just making you have to save before running the game to get changes, because of course you could still revert the frames if you like since this behavior has been fixed in the new LGM release, this also is like many other IDE's like event Studio which makes you save before changes are reflected
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
IsmAvatar
|
|
Reply #12 Posted on: October 03, 2014, 09:30:57 am |
|
|
LateralGM Developer
Location: Pennsylvania/USA Joined: Apr 2008
Posts: 877
|
I'm seeing a bit of disagreement over what I said. I'd like to clarify. I'm certainly not arguing against a CLI. As I said, not only did I write a CLI myself, but I did pretty much all of the work on making the API that opened the doors for a CLI in the first place. My point was that it makes more sense for LateralGM to use ENIGMA as a library rather than an executable, and I outlined several reasons. The main reason is the interactive feedback that ENIGMA can give to LateralGM. That stuff *could* be done on a command line, sure, but we already tried that and discovered it was a really really bad idea - and the whole deal with files vs memory was really quite minor in that discovery. Library calls are just all around cleaner. You don't need a parser, and you don't need to keep updating that parser with more complexity whenever you change something. It also affords us the ability to pass callbacks to each other, meaning that ENIGMA not only interacts with LateralGM, but it opens up certain LGM abilities to ENIGMA (such as opening, populating, and closing informational windows like console logs, errors, and progress dialogs). Again, if you try to do that with files, you need some sort of clever dirty way of detecting when the file is done.
As far as GMS and EGM goes, I'm glad that file operations are getting fast enough, and I'm glad that these formats give us individual resource saving/loading. This certainly does open the door for a lot of new things with CLI and passing files, which I believe we can make even better use of through library calls.
@TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things: * In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory. * While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory. * The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files.
|
|
|
Logged
|
|
|
|
|
Goombert
|
|
Reply #14 Posted on: October 03, 2014, 10:51:13 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
IsmAvatar, the first whole part of your post I think we understood just fine, nobody is advocating for getting rid of the callbacks, no way at all. @TheExDeus: While it's true that writing only the changes to the file could be faster than sending the whole file in memory, we need to remember a few things: * In LGM, the file is already in memory. Sending that memory to ENIGMA does not require any effort, because it's shared memory. It's just a library function call to tell ENIGMA to actually read that memory. * While having LGM save the file would be faster that way, you completely forgot that ENIGMA (cli) would now need to load the whole file. With library calls, the whole file is already in memory. * The main reason for the slowdown in communicating the file to ENIGMA is because of a few limitations we had to deal with at the time which we continue to streamline. Mainly, we decided to rewrite the entire file into new slots of memory for passing to ENIGMA. That's completely unnecessary. If we just wrote the changes to memory and marked those as changes (so ENIGMA knew that those are the only parts that changed), that would be a better comparison to the individual resources written to file - and it would necessarily be faster because memory is always faster than files. Here's where you seem to be kind of confused about our proposal though, that even Josh wants. If the project exists on disk, you don't have to load anything, the user clicks a resource and it is loaded and then they can change it in the editor, when you hit the save button all changed resources are committed to disk, and we would require you to save before changes would be reflected at runtime. This is like many other IDE's, but it would make LGM much better at working with large project files, even a native IDE should not have resources loaded that it is not concurrently working with or the user has any interest in. Does that make sense? But these changes would mean abandoning the GMK format because it is not easily expandable for this functionality, but EGM being ZIP's and GMX's being zip friendly those formats can do it. But basically the reason we haven't done this yet is because I've wanted to make LGM very very stable with some awesome features, such as the search tree, before any GMK support is dropped. LGM in its current state is basically becoming better than GM8.1 or GM ever was as an IDE pretty much.
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
|