Roadmap?

[Dragonite]

I was wondering if we have a roadmap or any future milestone for ENIGMA.
Someone once said the future of the project was enigma+emake+RadialGM. How is each of those projects going? At what state do we want them to be in 6 months for now?
Thanks in advance.

[daz]

Can't really speak to the others, but Poly and I have taken an interest in RGM and plan to adopt a more agile workflow around it.

Right now, RGM does practically nothing. It can sort of load projects, but cannot run them. And if that's an RGM or emake bug I haven't figured out yet.

Most of the editors in RGM are very partially implemented. It's not possible to make a game in RGM in its current state.

That said, I think roadmaps are pretty hard. Pretty much everyone here's got a full time job. And we don't know who will contribute for a week and or a single commit before leaving. I've done it myself, many times.

But if it looks like we're making some solid progress I might start doing some RGM progress report posts or binary builds for people to test. On the whole I'm very interested in investing in RGM to finally get rid of Java, and to be overall more efficient. There's nothing quite like trying to open a script in LGM and it taking more than a second to load on an overclocked multicore CPU...


I think the whole point behind emake was to make it easier to separate out the bits surrounding the actual build process for games. I think it's probably been a bit of wasted effort, given that nobody else aside from the core IDE developers on the ENIGMA project would bother with it.

The "core" of ENIGMA (the runner) is getting pretty regular updates. There's a lot of backlog there, and it seems like it requires a lot more experience to jump in there. But I've had some inconsistencies across graphics cards (that I personally don't own), and things like Mac OS have been in a broken state for a bit that others are looking into. But it works pretty well as it is, I think.

[Darkstar2]

I was wondering if we have a roadmap or any future milestone for ENIGMA.
Someone once said the future of the project was enigma+emake+RadialGM. How is each of those projects going? At what state do we want them to be in 6 months for now?
Thanks in advance.

It would be nice indeed to have a road map - but it's understandable that there aren't any - Unlike a closed project which has a regular and paid staff, this project relies on the free will of contributors - so due to the unpredictable aspect it's probably hard to have road maps that can be close to accurate.

For YYG it took them a long time to cough up one massively horrible IDE that nobody bloody asked for, so it all comes down to talent as well, not only numbers, though talent + numbers do help.

I've made personal predictions that RGM would likely take years to a final working product, though perhaps core ENIGMA updates and performance enhancements are happening continually which is great IMO, and despite what people say about Java and LGM, it is still a workable product, it just take some work around and tweaking to get desired results from projects.........  I think one thing I would like to see resolved in ENIGMA is that nasty backspace bug that occurs for unknown reason even on blank projects with nothing, otherwise it is my impression that LGM has gotten far more stable, I dunno if it is the fact it is run under Win10, new JAVA version, etc, but I'm able to work on longer projects for longer periods before it crashes and I have to restart LGM.

So for RGM I'm not "expecting" its release anytime soon as I know these things take time and without  a team it is reasonable to think it could very well take a couple of years - I mean it would take 1-2 years on full time development, so yeah who knows.

Whatever it takes - I've lost all hope for GMS anyway as it is heading in the wrong direction, updates are getting more scarce and farther apart as time goes by, so ENIGMA can still shine even with its faults.

[Goombert]

Daz is pretty accurate about his analysis. Everybody wants a roadmap, so here, I'll lay it out for the next 1-2 years.

* Grow the CI coverage by another 25% over the next year. This includes completion of binary buffers and testing of them. Coverage of objects, inheritance, and collisions.
* Continue improving and maintaining the ENIGMA engine and LateralGM. This includes new features and smaller fixes as users request them.
* Finish the EGM writer thus completing the EGM library. Test EGM project serialization to keep it working.
* Finish refining the asset compilation stages so that emake becomes a functioning remote asset compiler. This includes finalizing ideas of how RadialGM and the like should attach assets to the exe locally or remotely.
* Continue developing RadialGM into a stable and usable IDE. This includes completion of additional editors, refinement, and integration of the above completed APIs.

I would like to see resolved in ENIGMA is that nasty backspace bug

I gave backspace some testing today, couldn't see anything different from the regular input. Still need to be able to reproduce it before it can be fixed. I assume the reason you are having trouble with this at all was when we introduced SDL since the input logic was generalized. It's very easy to make mistakes when refactoring existing code when that code is not tested. Though, testing input is a little trickier, also why GUIs are tricky to test.

[Darkstar2]

see anything different from the regular input. Still need to be able to reproduce it before it can be fixed. I assume the reason you are having trouble with this at all was when we introduced SDL since the input logic was generalized. It's very easy to make mistakes when refactoring existing code when that code is not tested. Though, testing input is a little trickier, also why GUIs are tricky to test.

That's the problem, it is an edge case scenario, and the source is unknown, and what's even trickier is that in the past some projects did not exhibit it, but after doing a lot of process of elimination could not find the issue as it is likely not caused by one or even a few items.   Not for some reason even compiling a blank project causes it, no matter what settings are used, I have tried every possible combination and scenarios.  Also this issue was experienced by others who confirmed it.  So it is impossible to come up with a reproduction test case.  Just compile any blank project in windows and COMPILE IT, then execute it and when the window opens, hit the back space key on the keyboard - The process will freeze (the mouse icon will turn into an hour glass) and eventually the process will crash with an error.

It is also impossible to DEBUG, because this issue does NOT exist when RUNNING the project from LGM.  It only occurs in COMPILE mode.  I've also spent quite a number of days / weeks making changes at compiler level, and same, so it is not compiler / optimization / settings related.

So it's one of the worst bugs ENIGMA has had IMO, and we both thought the D3D reset bug was a hard one, until we both worked on it back in the years only for you to discover it was a 1 line quick fix after the clues I provided    But this backspace thing is a pain in the arse!  But why the bloody backspace key, this is quite intriguing to be honest.

[Goombert]

Alright, a couple of things, because I just reproduced it. First, I tried building an empty game and running it but that doesn't seem to work for me. The exe just crashes when I try to double click it. So I tried building the 3D Cubes Demo in compile mode and that worked. I press backspace and bam, instant crash. We'll have to figure it out.
https://github.com/enigma-dev/enigma-dev/issues/1743

Edit: Ok, I tracked it down to the addition of backspace behavior for [snip]keyboard_string[/snip] so it should be an easy fix.

[Darkstar2]

Alright, a couple of things, because I just reproduced it. First, I tried building an empty game and running it but that doesn't seem to work for me. The exe just crashes when I try to double click it. So I tried building the 3D Cubes Demo in compile mode and that worked. I press backspace and bam, instant crash. We'll have to figure it out.
https://github.com/enigma-dev/enigma-dev/issues/1743

Edit: Ok, I tracked it down to the addition of backspace behavior for [snip]keyboard_string[/snip] so it should be an easy fix.

Yeah I sent in the past some reproductions and examples, but since results vary a lot there is not ONE specific thing that reproduces it.  I've had this issue happen even on projects WITHOUT the use of any keyboard activity, keyboard string, etc, YET in the past I have been able to compile stuff WITHOUT the backspace bug, and then later discovered that declaring too many variables caused the backspace, but then with further tests I found it was not conclusive, since it appears this can be reproduced many ways and unknown.  Also it does not occur in RUN mode, but COMPILE mode only, so why are there instances where it happens and instances where it does not.  And upon further pulling of updates the problem got worse - In the past compiling a blank project did not cause the back space issue, now it occurs on blank projects.  So according to you, is the keyboard_string the main reason for ALL cases or are there possibly other causes ? And how to explain why it happened in some scenarios and not others, is there interference between keyboard_string and something else ?

[Dragonite]

Thanks for everyone's answers.
Goombert, are you welcoming pull requests to put documentation on each of those projects' repo? I'd like to contribute that way, if possible. Thanks for posting an updated roadmap. I think it makes it easier for everyone else to work towards those goals.
I do agree with daz. The runner works pretty well as it is (at least in my experience). 99% of the issues I had were caused by the IDE or by the lack of language features.

[Goombert]

So according to you, is the keyboard_string the main reason for ALL cases or are there possibly other causes ?

Yeah, probably. Read the issue on GitHub, if you look at the stack produced when I hit backspace you'll see it's calling into some Enclave Win32 functions. An Enclave is a way of isolating code within the same executable. When [snip]keyboard_string[/snip] is empty, and length-1 is passed to substr, there's integer underflow which causes it to be INT_MAX. So what happens is it tries to copy too many characters and overflows the stack (which GCC probably confined to an enclave). It only appears in some situations because of link time optimization. Again, behavior here is undefined, which just causes it to look random.

Anyway, it's a very simple fix and I'll send it momentarily. We just need to check if the string is empty before erasing a character.

Goombert, are you welcoming pull requests to put documentation on each of those projects' repo? I'd like to contribute that way, if possible.

It depends on exactly what you mean. We've actually stuck to documenting things here on our Wiki, which is more powerful, rather than doing so on the actual projects. We tend to only document exceptional things in the actual repos. Though, JDI is completely doxy commented. The Wiki is basically our version of the GM manual, but can also go into advanced topics that the typical user never needs to read. This line is drawn between "Getting Started"/"Documentation" and "Development Contribution".
https://enigma-dev.org/docs/Wiki/Main_Page

The runner works pretty well as it is (at least in my experience).

Well, hold your horses there buckaroo, we don't actually have an interpreter. There are actually other projects attempting to build an open source GM runner, but the ENIGMA project is closer to what YoYoCompiler does (and was also invented earlier; ENIGMA compiler was started in 2007). We take your GML/EDL and convert it to C++, then compile it and link it to our engine also written in C++. There are actually use cases for a runner, even if you compile, that's why YoYo kept their runner even after creating the YYC. A runner is better for... well.... running the game quickly. If you are developing in a hurry, you don't care so much what the runtime performance is, you just want to launch the game quickly to see if your code changes worked. A runner launches quicker because it doesn't go through compiling the whole game and optimizing it. Compiling your game is better for releases because it will optimize the game and make it run faster, which makes something like ENIGMA or YYC better for releasing games.

Thanks for posting an updated roadmap.

You're absolutely welcome! I hope it gives some insight on what ENIGMA needs to prioritize at this point basically.

[Dragonite]

It depends on exactly what you mean. We've actually stuck to documenting things here on our Wiki, which is more powerful, rather than doing so on the actual projects. We tend to only document exceptional things in the actual repos. Though, JDI is completely doxy commented. The Wiki is basically our version of the GM manual, but can also go into advanced topics that the typical user never needs to read. This line is drawn between "Getting Started"/"Documentation" and "Development Contribution".
https://enigma-dev.org/docs/Wiki/Main_Page

I meant things like having that roadmap put somewhere like a ROADMAP.md inside the repository or something like that.
My issue with the wiki is that information there seems to me outdated. The ENIGMA:ToDo page was last edited in 2013.

I feel the repo could contain the updated roadmap and a short/mid-term To Do (for things we would attempt to get done sooner). Some issues could be labeled as good "first issues" for those who want to contribute. Perhaps a CONTRIBUTING.md file saying how and where people could help.
The RadialGM README could make the alpha state of it more clear to anyone passing by.
Mainly these kind of things.

Well, hold your horses there buckaroo, we don't actually have an interpreter. There are actually other projects attempting to build an open source GM runner, but the ENIGMA project is closer to what YoYoCompiler does (and was also invented earlier; ENIGMA compiler was started in 2007). We take your GML/EDL and convert it to C++, then compile it and link it to our engine also written in C++. There are actually use cases for a runner, even if you compile, that's why YoYo kept their runner even after creating the YYC. A runner is better for... well.... running the game quickly. If you are developing in a hurry, you don't care so much what the runtime performance is, you just want to launch the game quickly to see if your code changes worked. A runner launches quicker because it doesn't go through compiling the whole game and optimizing it. Compiling your game is better for releases because it will optimize the game and make it run faster, which makes something like ENIGMA or YYC better for releasing games.

Sorry for not wording myself correctly. I meant the engine itself. I believe YYG uses the term "runner" for the actual engine nowadays, and calls the interpreted mode "VM".

[Goombert]

And how to explain why it happened in some scenarios and not others, is there interference between keyboard_string and something else ?

The fix for the backspace issue is now merged. And yes, any bug that overwrites the stack can muck up the whole program, that's why it's also the nature of buffer overflows and security vulnerabilities.
https://github.com/enigma-dev/enigma-dev/commit/1ab91aefefe8fd7019ffbab876448b7bf65463b0

Let me slightly correct myself, some of the problems you may have had, perhaps building an empty game, may have been unrelated. I am still investigating issues I am having with empty games.
https://github.com/enigma-dev/enigma-dev/issues/1745

The ENIGMA:ToDo page was last edited in 2013.

Ok, I've put the roadmap on that page, that's a good idea. Let me mention something though, and this is not as bad as it sounds. Many of the tasks on that page actually still need done. One of the reasons it's not so bad they haven't been done is because a few of them don't really matter to the users. For example, moving resources from arrays and vectors to our own templated container. That's not something the end user will really notice even in performance if it's done correctly. It will only improve our code maintainability.

Some issues could be labeled as good "first issues" for those who want to contribute.

Josh started doing this actually. However, it's kind of not working because I gobbled up all of the good first issues since they were so easy and I want ENIGMA to work well. I can't help it so much I tend to be a bit of a busy body, but I will continue trying to identify good first issues and labeling them.

I believe YYG uses the term "runner" for the actual engine nowadays, and calls the interpreted mode "VM".

I'm not so sure, I'm just going by the origin of the term. The origin was an executable, called the runner, which contained the interpreter/VM, which would be simply file copied and have your games assets appended to it in the data section. I suppose I should refer to it as VM now, though its behavior in modern GMS is not really that different from that in the classic GM.

[Dragonite]

Ok, I've put the roadmap on that page, that's a good idea. Let me mention something though, and this is not as bad as it sounds. Many of the tasks on that page actually still need done. One of the reasons it's not so bad they haven't been done is because a few of them don't really matter to the users. For example, moving resources from arrays and vectors to our own templated container. That's not something the end user will really notice even in performance if it's done correctly. It will only improve our code maintainability.

Okay, cool. Is it ok to report here if I find more pages with outdated information?

Josh started doing this actually. However, it's kind of not working because I gobbled up all of the good first issues since they were so easy and I want ENIGMA to work well. I can't help it so much I tend to be a bit of a busy body, but I will continue trying to identify good first issues and labeling them.

Didn't know of that, awesome.

I'm not so sure, I'm just going by the origin of the term. The origin was an executable, called the runner, which contained the interpreter/VM, which would be simply file copied and have your games assets appended to it in the data section. I suppose I should refer to it as VM now, though its behavior in modern GMS is not really that different from that in the classic GM.

Yeah, these days the normal build mode compiles your project into bytecode and it runs it inside a VM. It's much faster than GM8.1, but much slower than YYC.

[Goombert]

Okay, cool. Is it ok to report here if I find more pages with outdated information?

Sure, you can say so here, or Discord, or IRC, or wherever you prefer. I can also say that since you already have a forum account, you already have a Wiki account, so you are actually free to update content as you see fit. I just have to mention here that I must ask that you please consider copyright before uploading content to our servers. I've really only had one problem with somebody uploading copyright stuff in the past like 6 years~ I've been here, so it's not a big issue. Just make sure you either have the rights or the permission of the author to use any content you try to upload. Use your own original work rather than using somebody else's in general.

Yeah, these days the normal build mode compiles your project into bytecode and it runs it inside a VM. It's much faster than GM8.1, but much slower than YYC.

Right, I haven't dived too much into this part of the engineering problem here. I mean, ENIGMA and GM are both software to solve an engineering problem, RAD game development. You may be interested in speaking sometime to Josh or Rusky or other individuals who are more knowledgeable in this domain. There's actually something funny if I recall about YoYo's use of LLVM, in other words, it's not using LLVM to the fullest. For example, it still uses Clang to link or something like that. Actually, I'll share this post with one of these people and maybe they'll explain it better than me.

For example, I mean the lack of execute_string and other dynamic programming language features.