Pages: [1]
  Print  
Author Topic: Pure ENIGMA  (Read 867 times)
Offline (Unknown gender) TheExDeus
Posted on: March 27, 2016, 07:44:01 AM

Developer
Joined: Apr 2008
Posts: 1914

View Profile
Pure ENIGMA
This is a project I am slowly working. It is a version of ENIGMA engine that will be useful without LGM or the parser. Basically making ENIGMA as pure C++11 engine. But it will still use instances and events, which will be added trough code. At the beginning I will try using LGM as well, but I will either start creating my own IDE in ENIGMA, or later ditch it and just use a C++ code editor.
My reasons for this change:
  • LGM is not really maintained, and while Robert is back to fix some bugs, I don't know if there are any plans for the future.
  • ENIGMA parser is not maintained and haven't been fixed or changed in years. Josh had a new one coming, but the idea died. EDL is as powerful as the parser, and sadly the parser is not powerful enough.
  • ENIGMA engine itself is quite powerful and easy to use, but right now it is only usable trough LGM and the parser - both problems which I already described.

In the Pure ENIGMA branch the parser will only be used in a limited way. The new EDL will also be a stricter as a result (so the parser must do a lot less). The new language will be valid C++11 in almost every respect, so mandatory semicolons, all variables will have to be defined and so on. It will also support classes and structs, which are extremely important to make the existing code faster and easier to read. For example, ds_ functions are very verbose and pain in the ass to use and in most cases could instead be replaced by std::vector if desired.
From parser side only these things are currently planned to be used:
  • Variables won't be local to instance by default (like in GM) and instead will always require "local type v;" to be defined locally. This will not allow the use of uninitialized variables, which is the current bane of ENIGMA and it can be hardly debugged. So if a variable is prefixed with "local" it will be added to the object class.
  • Variable types will be mandatory. By default ENIGMA uses type "var" or "variant" whenever a variable is defined without a type, but here it will have to be specified. So "local var a = 0;" is the same as "a = 0;" right now and "var a = 0;" is the same as "var a = 0;" right now. This is required for better error detection as well as to allow greater amount of types, like classes. You will also be able to use C++ "auto".
  • The ID's of instances (and any other types like sprites or backgrounds) will be separate classes. So (983019709).x won't work, but "auto inst = instance_create(10,10,obj_bullet); inst.x = 10;" will. They will still be part of a larger global structure, so it would be possible to iterate them or access them trough ID (like instance_get_by_id() or something similar). This is again required for error detection. Currently every function takes an integer ID, so you cannot at compile-time show an error if a user uses draw_sprite to draw a background resource. It is often the case that the error will not even appear at run-time. This greatly slows down development and allows large amount of errors.

The whole ENIGMA engine will also be rewritten (iteratively) to use C++11 whenever possible. Right now it is used in some places (new functions), but not in previously written systems or the parser. So we have large amounts of code that does something C++11 already does by itself (to_string functions). This should also make the code faster. For example we could make the whole math library constexpr. What it means is that the compiler will execute these functions and replace the code by result whenever the argument for these functions is known at compile time. Like "cos(0)" in step event will currently be recalculated all time, while with constexpr it will be replaced with (1) instead.
Another example would be in the parser. C++ doesn't allow switch(x) statements to have anything other than an integer as x. EDL allows strings as well. It allows that by rewriting that switch() in the parser. It hashes the argument, compares with hashes, the compares the string itself and them jumps. So this is EDL:
Code: (EDL) [Select]
var x = "Hello";
switch (x){
        case "Hell": b = 0; break;
        case "Again": b = 1; break;
        case "Hello": b = 2; break;
}
And this is generated C++:
Code: (C) [Select]
var x = "Hello";
      const variant $s0value =((x));
      switch(enigma::switch_hash($s0value))
      {
        case 2245469:
        if($s0value =="Hell")goto $s0c0;
        goto $s0nohash;
        case 63193920:
        if($s0value =="Again")goto $s0c1;
        goto $s0nohash;
        case 69609650:
        if($s0value =="Hello")goto $s0c2;
        goto $s0nohash;
        default:
        $s0nohash:
        break;
        $s0c0:
        b = 0;
        break;
        $s0c1:
        b = 1;
        break;
        $s0c2:
        b = 2;
        break;
       
      }
     
    }
   
  }
  ;
I guess it does the second if() check because the hashes could have a collision. In C++11 this can be like so (if enigma::switch_hash was rewritten to be constexpr):
Code: (EDL) [Select]
var x = "Hello";
switch (enigma::switch_hash(x)){
        case enigma::switch_hash("Hell"): b = 0; break;
        case enigma::switch_hash("Again"): b = 1; break;
        case enigma::switch_hash("Hello"): b = 2; break;
}
For collision checking additional if checks could be added. We could also use 64bit int's for hashes. The result will be just as fast code, but greatly reduced parser complexity (even though the switch() code is only about 200 lines).

The cleaned up ENIGMA engine will have sized types everywhere. This is actually quite essential for a cross-platform game engine, but we slacked at this. We can use float and double as they are standard sizes, but we shouldn't use "int, unsigned int, short, long" and so on. And we use them a lot. Instead we should use "int32_t" or "uint32_t".
The ENIGMA engine also consists of A LOT of unmaintained systems. For Pure ENIGMA I will focus my efforts only on a limited number of systems which I can maintain and check. If others want to maintain something more, they are welcome. So in Pure ENIGMA there will be only GL3.3 graphics system (no D3D or GL1). D3D is only useful if we want to port ENIGMA to XBOX (which nobody plans to do) and GL1 is not useful for anything anymore - it is only supported by more than 15 years old PC's. Anything newer than that can run GL3 and any new smaller device (phones, RPi, Nvidia Jetson etc.) supports GLES, which is a lot closer to GL3. GLES is also something I plan to support or gladly allow someone else to make.

As you can see this idea is in no way compatible with GM. ENIGMA hasn't really been ideologically compatible in years and in reality it never was. So I this is meant to make ENIGMA a good game engine in its own right. I use ENIGMA in my everyday life and even commercial projects have been made in it. And trough these I have seen the potential of ENIGMA. It is the easiest, while being the fastest and smallest ENGINE out there. Recently I made a quite complex model editor for 3D printing in ENIGMA. The executable, including images and even a scripting engine, was only 3.8mb, while also running >1000 FPS on my home PC (up to 2.5k FPS). And the GUI is only about 6k-7k LOC. No programmers I have worked with has ever seen anything like that. I think we have struck raw diamond with ENIGMA, but we sadly nobody here wants to polish it to get a precious gem.

I will add new toughs to this topic as I move along. Discussion and ideas are welcome.
« Last Edit: March 27, 2016, 07:46:18 AM by TheExDeus » Logged
Offline (Male) Josh @ Dreamland
Reply #1 Posted on: March 27, 2016, 02:00:56 PM

Prince of all Goldfish
Developer
Location: Ohio, United States
Joined: Feb 2008
Posts: 2946

View Profile Email
I like the idea, and encourage you to go for it. Macros in C++ can do a lot of what the parser does, if you keep your code syntactically correct. It's a bit ugly to do that, but that's what it takes, sometimes.

Please don't get me wrong—I also see ENIGMA as a diamond in the rough. But I believe a huge contributing factor to that was the IDE. However, even implementing a GM-like engine for single applications in other languages led to simpler, more beautiful code for me, in the past, so there is probably something to the idea of trying for a pure-code implementation.

I'm sorry if I give the impression of not caring about the project. While you have been looking at the engine and seeing the potential for what it could be without Game Maker holding it back, I have been looking at the frontend and imagining what it could be. I think you'll find that in both cases, offering about 85% feature parity is trivial, and as you hinted, that's all ENIGMA has really ever done.

So, by all means, keep us posted. Know that on the IRC, ideas are constantly being kicked around for what an IDE should look like. Robert's off to college, and other maintainers are getting on with their lives. It might surprise you to learn that Quadduc is actually hanging around the IRC, though he doesn't say much. Sad thing is, I think others (particularly Robert) are waiting for me to make a move, and I'm waiting for confidence that I'm not going to blow it. As my life at work continues to solidify into something that I don't have the power, and therefore the need, to influence, I am beginning to invest more and more time into thinking this through.

I'm going to go ahead on a bit of a rant, because I find it therapeutic. You can probably stop reading here.





I recognize that I am making a classic mistake of overthinking actions before taking them. As a kid, I didn't have this problem. Not to take credit for the countless hours of hard work you've put into this project in the last eight years, but my childlike belief that just slapping code down would make everything work is a huge  part of why this project exists right now. And as I work close to other engineers who don't commit anything without PRDs and design docs and tests and all this other administrative minutia that I'd have gawked at as a child, and as Rusky occasionally bombards newcomers with articles from others like us who have discovered how easy it is to point out design practices that are not correct, I continue to think about the ones that are. I guess at this stage in my life, ENIGMA as it stands today is the answer to "What would happen if I just cowboy-coded everything?" ...and I remain unconvinced that it can ever be the best answer. Or even close to it. I'm not looking for mathematical perfection; I'm looking to remove all these little gotchas and bugs.

When I was younger, I always felt like this was all fine, because the things that suck can just be recoded. But I recoded that damn compiler, what, four times since I was 15? Each time I came into it knowing five times as much as I did the last time, easily. Hell, the first time I wrote a GML→C++ parser, it was in GML. I knew hardly any C++ at the time; I was fourteen. Talk about bootstrapping.

But now that I'm a full-time employee, working for a company that maintains projects dozens of times larger than ENIGMA, and am surrounded by people who all share the same fears that started forming in my mind when I was 20, I'm seeing the costs that came about every time I recoded the parser. Each new version did a dozen things better, and exhibited a half a dozen regressions. Did you know that the second time I recoded it, I forgot to add comments until I tested it out on a full-blown game?

You might be thinking, why not just recode pieces of it? In general, that's a better idea, but all of these pieces have a fundamental dependency in them that is not easily upheld: JDI. We'll get back to that in a bit.

After all, I'm smarter now; I feel as if I'm a hundred times better at coding than I ever was when I was working on ENIGMA. As a consequence, the compiler seems daunting to me—not because it's so much code, or a hard problem, but because it does so many weird, little things to get everything right, and then it still doesn't get everything right. It never has; it's always been some weird maximization problem that I never fully established had a solution. Because as a kid, I didn't plan the whole project out; I just built the pieces of it, and they indeed fit together into something that looked like what I set out to build. It wasn't just "build a compiled Game Maker." It never was. That's an easy project; it'd take me a few weeks to do that. It was "build something that looks like Game Maker, but compiles to C++ and is really fast." I could even do that much now, with a few executive decisions that would just be made. But then, look at "really fast." What does that even mean?

I know you were able to get your tool to run really fast in ENIGMA, but ENIGMA is a game engine. Part of making ENIGMA really fast is handling things like sprite batching (which Robert implemented something like) and spacial partitioning for collisions (an idea that as a kid, I could never work out). Even now, I don't have a great answer for how to do it seamlessly—without the user having to manage it—in a way that's actually efficient. Does GM even do that? For fuck's sake, Dr. Overmars was teaching a class on collision handling. That was his thing! Yet he designed GM to use an x variable and a y variable. Now I have to update the quad tree twice every time you move the object yourself! It's that kind of little shit that bugs me about this whole operation. How do I design around that? Do I choose to just not care? Kid me wouldn't have thought about it for more than ten minutes, I suspect, as much as kid me loved efficiency. But then, kid me wasn't so great at that, anyway.

I made a lot of weird decisions as a kid. Did you know the first local variable access code literally copied all the variables out into temporary storage so they could be accessed generically? The entire instance was copied somewhere every time you needed to access some_obj.x. This only existed for a few weeks, before I coded the current implementation, but even now, I'm not sure that the current implementation is the best way. Look how fast some of these new scripts are, using lookup tables for their variables. That would have made things easier.

Not to mention the decisions I made involved maintaining a C++ parser for ENIGMA's purposes. If anything is going to show you how dangerous an idea that is, it should be C++11, which JDI still doesn't support. As much as I think LibTooling is a fucking cow, it would have been worth using just to narrow the scope of the project. These words are bitter and they hurt me as I type them; it's such a vapid, meaningless statement, until you realize just how much work actually goes into maintaining all these little-shit dependencies. You just have to accept you can't have a non-bloaty ENIGMA compiler while still supporting the ever-changing C++ specification in the background.

It's 2016, now, and LibTooling still isn't where I needed it to be as a kid. My argument for keeping my C++ compiler (JDI) around was that it not only granted us the definitions we needed from the engine, but a lot of the logic needed in parsing C++ was also intrinsic to EDL. Consider overload resolution: ENIGMA has really poor overload handling, and for a while, I expected JDI to be the cure for that. It was unfortunate coincidence that I finished JDI just as C++11 was coming out; if I had known all the new features of C++11, and baked them into JDI, we might be having a very different conversation right now. But then, if I'd known those features, maybe that'd leave me where I am today, and I wouldn't have written JDI at all, for an understanding that maintaining two compilers just isn't a good idea—even if they're compiling virtually the same language.

Perhaps instead, I'd have modified LibClang to add support for with(). I'm not sure; I was an industrious kid. It still wouldn't have bought me out of the quad tree problem, but it would have at least spared you the problems you've been having.

The one philosophy I had as a child that I will always keep with me is that when you design something, you need to take a step back, stop thinking in terms of the tools you have, and start thinking about the best possible way to say something. Thinking outside the box, as it were. I remember designing ENIGMA's show_menu functions in GTK. I'm still a little proud of them, even though they're a little backward. It doesn't meet every use case, but it meets a few of them superbly. And even now, at this huge company I work for, that's exactly what the engineers around me seem to be trying to realize.

So I'm looking at the big picture—bigger than I ever looked as a child—to find what I really believe is the best way, end to end, to make a damn game. And if I find it, you can bet your ass that I'll be back.





That's enough rant. The point is, I don't have all the answers, anymore. In truth, I never did; I was just happy to keep working without a plan, and that's changed for various reasons, but possibly not right reasons. Some soul searching is still in order, I suspect. I can't make you any commitments, other than that I'll be around.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Unknown gender) TheExDeus
Reply #2 Posted on: March 28, 2016, 08:25:22 AM

Developer
Joined: Apr 2008
Posts: 1914

View Profile
Quote
I have been looking at the frontend and imagining what it could be.
I do get that. I loved the way GM handled things and it is how I started programming in the first place. I have said it here before - I like that GM can literally do anything and in fact with little code. You can make a 2D tower defense game in 2k lines, just like you can make a drawing program in 2k lines, ray tracer in 2k lines, file explorer or a 3D FPS. The way GM is designed allowed this unbelievable amount of flexibility with very amount of programming effort. I haven't seen anything like this in my years of trying out game engines. Using Unity or Unreal4 would sound absurd if you wanted to make a file browser that ordered icons by hue, but GM (and in turn ENIGMA) can do it in few hundred lines. At the same time if you wanted you could create a 3D RTS game, the most basic of which would also take a few hundred lines. And I think one of the reasons for it is because we lack this higher-level logic that goes in the back of all those other engines. They do occlusion culling, scene management, physics, AI and much more inside the engine at the back - even if you don't need it. And this higher-level integration allows rapid 3D game creation because of the pipeline it has. GM and ENIGMA never had any real game creation pipeline. There is no GUI/HUD editor, 3D room editor, changing something at runtime (there was the Build mode in ENIGMA) and so on. I believe we could have it. And trough extensions we could create everything else necessary, like the scene manager.

I did mention that I wanted the Pure ENIGMA to use only C++ in code editor and ditch LGM if possible. But that is because of the limitations of LGM, not the IDE idea itself. I like the object system, event system and all the other great things Overmars came up with, but I just want to code in C++, not EDL, so I could efficiently use the features C++ provides. My idea is to create several tools, which initially will be separate, like HUD editor. Then later these tools will come together in an IDE, not too dissimilar to LGM.

I think we need an IDE using ENIGMA internally, otherwise we won't ever have a WYSIWYG editor. But the fact that both GM and ENIGMA is so powerful and easy without it just shows how genius is the basic idea behind it.

Quote
But now that I'm a full-time employee, working for a company that maintains projects dozens of times larger than ENIGMA, and am surrounded by people who all share the same fears that started forming in my mind when I was 20, I'm seeing the costs that came about every time I recoded the parser. Each new version did a dozen things better, and exhibited a half a dozen regressions. Did you know that the second time I recoded it, I forgot to add comments until I tested it out on a full-blown game?
I also work with these kinds of design docs and usage diagrams but I often see how they do actually fail. You cannot plan ENIGMA from start to finish on a piece of paper and then start coding. Requirements will change, problems will come up which you would have never though of and sometimes even better idea will come to mind. Basic idea needs to be known and there are a lot of software practices we really need, like integration testing, version releases and support branches. We haven't released a new version of ENIGMA in over a year because I haven't personally had the time or strength to do it. And of course totally recoding something is also not often possible, which is why it must happen in parts. ENIGMA is quite modular, even if not perfectly so, and we can recode and cleanup large parts of it without doing it 100% from scratch.

Quote
I know you were able to get your tool to run really fast in ENIGMA, but ENIGMA is a game engine. Part of making ENIGMA really fast is handling things like sprite batching (which Robert implemented something like) and spacial partitioning for collisions (an idea that as a kid, I could never work out).
ENIGMA is more than a game engine. It can be an engine for anything, which is the same thing I came to conclude about GM. And we of course shouldn't forget about game development as well, but most of these things overlap. In my tools I use the GUI system I created which renders using the graphics system, so I actually use sprite batching (and now texture atlas's) a lot. The whole graphical interface renders in 1 draw call because of it. All these things which we need for games are usable elsewhere, but we need the possibility to disable it all. Larger game engines have these things very connected inside it, so it is not easy to replace one collision system with another for example. If want to be very specific to games, then we probably need a scene manager that controls drawing, occlusion, batching and so on. I think most of these things could be made modular. But I guess the biggest problem here is that ENIGMA never had them. Saying that I use ENIGMA wrong because it is a game engine doesn't really apply if ENIGMA isn't really a game engine. It's more like a framework, a la SDL.

So how about using C++11 for most the things parser is doing? I think we can support with(){} (which is a macro right now) among other things just fine. The only issue is dot access. obj_controller. must access the first existing instance of obj_controller. in the room. This requires iteration, unless we keep a list (map maybe) of object->instance relations which are updated on every instance_destroy(). My idea is to make it less compatible with GM not because I want to, but I would need to. If there is a way to make it compatible with GM without making it ugly, I would do it.
Logged
Offline (Male) Josh @ Dreamland
Reply #3 Posted on: March 28, 2016, 09:57:14 PM

Prince of all Goldfish
Developer
Location: Ohio, United States
Joined: Feb 2008
Posts: 2946

View Profile Email
You'll always want the parser to handle locals; it's just too much boilerplate, otherwise. You'll find there's a tradeoff between how pretty the code looks, and how pretty the object declarations look. I'm afraid you can't have both.

I'd recommend creating a sort of vtable class containing references to locals for each object.

In my plans for the newer ENIGMA, I described using instance_t to refer to instances, instead of int. This would allow you to swap out the typedef to int for something better—or possibly just a wrapper class around it.

Code: (C++) [Select]
struct instance_t {
  object_locals *ref;
  enigma_vtable *vtable;
 
  instance_t(int compatibility_id): instance_t(
      (object_locals*) fetch_instance_by_int(compatibility_id)) {}
 
  instance_t(object_locals *direct_ref):
      ref(direct_ref), vtable(ref->get_vtable()) {}
 
  enigma_vtable* operator->() { return vtable; }
};

If you set your vtable up correctly, access would then look something like inst.ref->*inst->x.

I've pasted a quick proof of concept.

You can squeeze a little more pretty out, if you really need, by creating convenience functions in instance_t or some derivative thereof, of the calibur int &one() { return ref->*vtable->one; }. Makes life just a little more complicated, as this means you can't declare instance_t using partial types (currently, only the implementation of the constructor actually depends on anything concrete, and that dependency is easily moved and/or eliminated).

Unfortunately, that's genuinely the best I can give you. You still have to parse the code for local variable declarations, or else write a whole lot of boilerplate. Or, I guess, just a little boilerplate, so you have a separate list of locals. Still, ew.
Logged
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Male) Goombert
Reply #4 Posted on: April 02, 2016, 10:23:47 PM

Contributor
Location: Cappuccino, CA
Joined: Jan 2013
Posts: 3060

View Profile
Yes, I am trying to roll out a new version of LGM that fixes a lot of the issues people have been having. I also created a stable branch that continues on from 16b4 and only includes the most important patches from master. I support what you are doing, I always wanted to see an SDK like version of ENIGMA that could be bound to any programming language.
Logged
Pages: [1]
  Print