ENIGMA Development Environment

Greetings, all; I have a few small pieces of news to report.

First and foremost, recent parse progress is here. Lines 2-110 were read in by the program as input; lines 114 and forward are complete program output. It should be a completely capable C parser by the end of tomorrow. Note: I said "C" parser, not "C++" parser. The difference being massive.

Second off, Gary is working on our mailserver issues. New members can't register at the moment because the server is incapable of sending mail. For those of you who have attempted to register, please hold tight, we will resend your emails once the mailserver is back up.

Lastly, I have added two BBCode tags.
The first is called "bubble." [bubble]This is the default bubble[/bubble]. You can also specify your own color [bubble=red]for maximum overkill[/bubble].
The second is called "snip." I introduced this tag in the past in a custom BBParser, but now I added it to SMF. The snip tag allows you to put snippets of code inline with the rest of your sentence. To demonstrate, I will give usage.

The code [snip][bubble]Hello, world![/bubble][/snip] gives this: [bubble]Hello, world![/bubble]

The code [snip][bubble=green]Hello, world![/bubble][/snip] gives this: [bubble=green]Hello, world![/bubble]

The code [snip][snip]Hello, world![/snip ][/snip] gives this: [snip]Hello, world![/snip]

And now you know everything.

Perhaps later I will add syntax highlighting again and we can all live happily ever after. Perhaps.

Yes, we're having some "Git problems." Ideka just suggested we switch to Mercurial, in fact, but it'd be pretty pathetic if some trouble with our choice of version control software was causing us all this hassle. In reality, there are a number of nasty things afoot.

Our only issue with git's capabilities is its inability to store binary files, something SVN never showed its issues with. The rest of the issues all had to happen eventually, it's just that by switching to git now, we've busted the grand problem piñata.

The other issues are all related to deployment. What's worse is that a new STDC just hit the mainstream repositories on Linux, and ENIGMA's current parser doesn't get along with it. That timing is in itself bad, because I am just now recoding that parser anyway to be able to share code with ENIGMA.

Now, the rotten, dried up cherry on top of this shit sundae is that the forum's having email trouble, so new registrants can't get their confirmation email. I can't say whether this is related to the influx of new users whom we are only managing to confuse to tears with the endless--to borrow HaRRi's word--maze that is the installation process on ANY platform now. But there's that, too.

So, everything is tangled, and along with general bugfixes, IsmAvatar and I are working hard to untangle it. In addition to that, you probably can't see the progress from the forum because the project has been split into separate pieces now. The part of the parser I am working on is a separate project in itself now. I hope that this will improve its chances of being adopted by, for example, IDEs that need definitions for code completion. This is a large-ish aspiration due to the diversity of code with which the parser needs to be able to interface, but never say never.

Proceed with Comments/Concerns/Complaints/General Hatemail.  Except cheeseboy.

I was working on this new parser yesterday, and I thought it'd be kind of neat to make my AST generator export SVGs. So I did.

So, given this expression: 2 << 1 + 6 / 2 - 2 / 2 << 8 % 5 + 5 / 2

The AST looks like this, step by step:

ast << create_token_dec_literal("2",1);


ast << create_token_operator("<<",2);


ast << create_token_dec_literal("1",1);


ast << create_token_operator("+",1);


ast << create_token_dec_literal("6",1);


ast << create_token_operator("/",1);


ast << create_token_dec_literal("2",1);


ast << create_token_operator("-",1);


ast << create_token_dec_literal("2",1);


ast << create_token_operator("/",1);


ast << create_token_dec_literal("2",1);


ast << create_token_operator("<<",2);


ast << create_token_dec_literal("8",1);


ast << create_token_operator("%",1);


ast << create_token_dec_literal("5",1);


ast << create_token_operator("+",1);


ast << create_token_dec_literal("5",1);


ast << create_token_operator("/",1);


ast << create_token_dec_literal("2",1);



neathuh

Just posting to wish everyone a merry Christmas and whatnot.

I've not seen Ism in a few days, but polygone and HaRRi are working on this and that, and I've committed the first revision of my new C parser. Forgoing future holiday distractions, I should be done with it in the very near future; possibly before this year is over.

Not sure about you people, but I have things to celebrate and family to visit.

Cheers, of the Christmas variety.

We've had an influx of new members, lately, and while I can't be certain that more than a few of them are humans rather than AI-complete spam bots, much less attest to how they found us, I do know that ENIGMA's wiki is among the first (often the first) Google result for many GM functions. Issue is, our documentation on them was laid out by a bot, and thus is incomplete.

What I'm getting at is that the Wiki is likely how people have been or will be stumbling upon our site, and it doesn't look very good if we've got a thousand undocumented functions people are looking for information on. My first order of business upon finding an undocumented function in a Wiki is to curse and leave, not investigate the project.

That said, I'd like to divert some focus to it, and I'd certainly not mind help. If you would like to contribute to the project in some way, helping document the function on the Wiki would be a good one. Please refer to: http://enigma-dev.org/docs/Wiki/Function:Documentation Feel free to read that if you're in the mood for a novel, or just head to http://enigma-dev.org/docs/Wiki/Category:Function:Needs_documenting and start filling shit out.

Anyway, as for the state of the rest of the project:

• Ism and I have been working on integrating the new text editor, of which we have been releasing test versions in the IRC. We won't be committing anything until we're sure nothing is getting trampled by it.
• Polygone has been working on the 3D functions, as you can probably tell by the 3D games showing up on the EDC.
• I've been working on the new C parser, which I will be keeping as a separate project on which ENIGMA will "depend" (we'll be keeping a copy of it in our repo, same as always, so don't worry).

So basically, the only source of commits lately has been Polygone working on 3D, since the rest of us are working on chunks of software with which to replace existing chunks.

Cheers; go work on the Wiki.

The ENIGMA project has hit a snag recently. Basically, our collective ideologies have become tangled.
It looks something like this:

.

Ism and I are both sorting issues with our projects. Ism is dealing with Java's ill-equipped generics, trying to structure LGM to be more extensible for future releases. As far as you need to be concerned, this is so we can add our Definitions and Overworld resources, as well as other potential resources down the road.

As for myself, right now I am dealing with a small issue regarding a dynamic type that implements implicit accessors and setters for generic data, which has presented a number of "how should I"s, but in the big picture, my concern is for the progression of ENIGMA's parser.

One point is certain: I need to rewrite or do serious work on the C parser. ENIGMA is presently ill-equipped to make distinctions between the following sample lines:

(id)
(int)
(show_message)
(100001)

This has led to issues in interpreting some GM6 code unless extraneous parentheses are added, which is unacceptable to newcomers, as well as to issues implementing some of C++'s more desirable features, such as complicated ternary expressions. This is because ENIGMA needs information from C.

The purpose of the C++ parser—be it the current one, a rewrite, or Clang—is to provide ENIGMA's EDL parser with information about available types, functions, and other constructs. For example, ENIGMA would not know what var a; meant if the C++ parser couldn't understand var's header, and it would not understand what show_message was if it could not read function definitions. The parser also needs to be able to resolve complicated types for further error checking. Hence, we need some mechanism of parsing C++ sources accurately enough for ENIGMA to produce correct C++ code.

I am faced with two options which I have filtered out as the best.

• Rewrite the C++ parser to support the new ISO and to better support the old.
  
  This will mean partitioning ENIGMA's compiler into two projects—one to parse EDL, and one to work with C++—and maintaining them both. I would choose to split the two so that any other party interested in parsing C++ could potentially join in, though it has proven in the past to be unlikely for this to happen.
  
  The benefits of doing this would include minimalistic size and features tailored specifically to ENIGMA and similar projects, meaning—if I code it right—it could potentially work faster than Clang due to a lack of need to lex and check code inside functions. The resulting project would also be less than a megabyte in size (ENIGMA's current hand-rolled C++ parser is 300 KB of source).
  
  
• Drop the existing C parser altogether and outsource to Clang.
  
  Clang is an LLVM frontend. Certain members of this forum, I'm sure you're aware, blow lots of smoke about LLVM, but in general we would want to avoid it because it will run users somewhere between half a gigabyte and a complete gigabyte of disk space, and MinGW LD/MSys would still be necessary (which is presently the largest component of ENIGMA).
  
  Clang, and its support for LLVM, would bring unprecedented benefit to the project, but not unwarranted benefit given its size. If ENIGMA was compatible with LLVM, by nature of the huge amount of support for the LLVM project, ENIGMA would be able to use, export to, and interact with a half a dozen other languages, notably JavaScript, Lua, Python, and an interpreter for C++ which, depending on its speed, could mean a native method of doing execute_string(), or could just be something crappy to avoid (we have only just learned of its existence, and I have little faith in the ability of anything to both parse and interpret C++ in a reasonable timeframe). In short, we would be getting not only C++ functions, but Lua and JavaScript functions as well, if desired.
  
  The issue is not only the huge size. Ostensibly, I could invest a few hours each update into isolating the segments of Clang required to simply parse C++ and give me info about it. Therein lies the issue; I am well aware that at this point, Clang sounds like the clear choice, but whether or not I choose Clang, I am left with something to maintain. Getting just libclang, its Clang dependencies, and the necessary headers from the LLVM svn to compile alongside ENIGMA will take work, research, trial, and error, and will cost about 50MB in the SVN. Then any time Clang updates something, I have to try updating my copy without stomping all over the modifications. I can't measure at the moment how messy it will be; only that the process is not streamlined.
  

That said, I am torn between the two options. I need people to say, "50 MB and a potential shitton problems that aren't yours, in exchange for four languages? That sounds worth it to me!" Or to say, "for that price, just write your own."




TL;DR version:

(1) Custom	(2) Clang
Tiny (Less than 1MB); fast, pointed runtime	50 MB; Parses EVERYTHING, though quickly
Gives precisely the needed information, no more, no less	Gives general information that can likely be used to meet all of ENIGMA's purposes.
	Supports interfacing with other languages (Lua, Python, JavaScript) at the cost of hundreds of megabytes on top of Clang
Likely to be sole maintainer, responsible for all aspects including any potential errors. This would be no different from now. At worst, it could mean a second recode in the future, but ideally I would make the code sufficiently extensible to prevent that this time.	Maintenance involves separating Clang from LLVM as cleanly as possible every time an update is made; any parse errors are not the responsibility of the ENIGMA team, and may or may not be dealt with in a timely manner. Potentially, we'd be facing another MinGW fiasco. (See #13297)

Additional Q/A:
dazappa: Clang is a "frontend to LLVM." So you would use Clang but not LLVM? And what's the final size decision, 1gb or 50mb?
Josh@Dreamland: Well, clang has LLVM dependencies, so I would be cutting LLVM into little pieces and throwing away the ones I don't need. 50MB is the projected size after I throw away the little pieces.

dazappa: Would you rather maintain 50mb of shit that you don't know, or 1mb of shit you do?
Josh@Dreamland: Good question. Ideally, for Clang, the maintenance would just be updating Clang headers, adding any more pieces of LLVM that become necessary, and making sure it compiles as though the configure script had run. Maintaining a megabyte of C++ parser can be just as difficult, if not more so as my responsibility extends beyond making sure it simply compiles.

dazappa: Well, you failed to write the C++ parser happily the first time, so you think you'd be able to do it better the second time? Using clang might save you time if you can get it setup and be able to easily update the headers like you think.
Josh@Dreamland: I do think I'd be able to do better the second time. As you can see, the current version succeeds for the most part, with one warning it throws three times. If I code the second version knowing everything I do from the first, and with all that shit in mind, I should be able to get it to play nice.

dazappa: ...Until you realize you have to rewrite it for a 3rd time.
Josh@Dreamland: I'm going to use a system very similar to the recursive descent scheme Rusky talks about. Basically, it would use the body of the current C parser, which invokes a function to handle the token in the context of each statement. Instead of calling this big mash-up switch statement that makes a hundred if checks, the parser would call one function based on the current context and pass it the token, and that function would deal with it appropriately. This means adding a new type of statement to the list would be pretty easy.

Go to town.

So today concluded the 21st Ludum Dare competition. I didn't make this public due to the fact that it was a last-minute decision made twelve hours into the 48 hour contest, but I decided to enter. Dazappa did as well.

Here are our games:
Dazappa's Crystal Conduit
My own Cave Escape

Okay, so my title's not that creative. Nor is my game itself, really. Point is, the two of us entered this competition.

For those who are unfamiliar, the Ludum Dare is a 48 hour game making competition. Competitors are given precisely that long to develop and "perfect" their games. Hence, my engine has serious physics and collision issues, and I imagine dazappa would have preferred more time, too. This year, with Notch from Minecraft participating, the server was so overloaded with interested peoples that they had to extend the deadline to give people a chance to wait out the server issues.

While, given the competition (especially this time around), it is astronomically unlikely for either dazappa or me to win, that was not the reason I entered (though I'd certainly not mind if one of us did win). I can't speak for dazappa, but I entered for three reasons,

1) To put ENIGMA to the test and show what it can do
2) To register a blip on the engines radar in the name of ENIGMA
3) As a learning experience for myself

At this stage, you'd think I wouldn't have that much left to learn about game design, right? Well, you're wrong, but that's beside the point. I wanted to learn not about game design itself, but game design in ENIGMA. Using only the features of ENIGMA a typical user would utilize, I discovered quickly a small number of bugs, quirks, and features that should be added. While most of the features I thought would come in handy were already planned, I did manage to come up with a few extras. Here is my list of discoveries made, bugs to fix, and features to add:

• Any sort of iterator tampering in draw will kill you. Not just depth change; room change as well.
• Initializers are ignored while parsing with() statements. For example, with (object) { int a = x; } will use x from the calling scope.
• Script scoping is incorrect in with(). This->script() is used when ::script() should be instead (::script is a global copy that sets scope manually via with()).
• LateralGM's highlighter and code editor in general suck. We should probably finish the one I mostly finished.
• Multiple inheritance is a must for complicated games. Also, LGM has an inheritance loop checking issue in which the check seems to return parent->parent != null.
• switch() {} doesn't coerce enums; this is an issue because switching sprite_index will be an O(N) operation unless you hard-code the IDs.
• The instance creation codes in rooms should be bumped up in priority, and it should be easier to enumerate and manage those codes.
• It's high time for ENIGMA array literals. And, well, a new type resolver so it can tell what's an array.
• The room editor needs a way to drag select and move efficiently.
• Design mode would really expedite platform gaming development, and a slow-motion feature should be added to it (something to quickly set room_speed.
• Since the collision system depends on the positioning tier, which contains hspeed and vspeed, an option should be allowed to let the user control the two.
• I found myself really wanting the planned "Overworld"/"Map" resource.
• I've still not fixed local const.

Other issues: Does instance_count grow with each create event at room start in GM as it does in ENIGMA?

Other than that, I was mostly in a pissy mood at IsmAvatar, because I encountered about 60 problems that EGM would fix. No ++, no alpha channels in the default format (GM6) (This cost me another our of work), no way of declaring C-level globals.

Also, polygone's framerate fix fucked me a bit, because it only fixes it on his computer, and packing AL in with the engine made filesizes huge. Something needs done about those two, too.

At that, I'm tired. Good night.
If anyone else entered the Ludum Dare, do let me know.

All right, with nearly a hundred revisions since our last newspost (which was quite small), Polygone recommended that I make a new one. So here it is.

What's been done?
I implemented some of GM's syntax quirks. Assignment = to comparison ==, mostly, along with about 10 bug fixes for bugs discovered in TGMG's mass compilation sprees.

As part of this, I gutted the syntax checker, which was written before we had genuine intention of supporting C++ macros. Yeah, pretty ancient. The new one is in its early childhood (meaning half of it is written).

switch() has been implemented at the repeated request of kkg; at the moment, you can switch() anything that can be stored in "variant". Down the road, you'll be able to switch anything for which there is a switch_hash() overload. As you may have gathered from that previous statement, ENIGMA's switch() is hashed, unlike Game Maker's. This introduces an incompatibility: Putting the default: label first doesn't fuck everything over. <_< Other than that, you'll find that the only incompatibility is the massive increase in performance.

Polygone and HaRRiKiRi have been hard at work implementing functions and DND actions. Polygone, as practice while learning C++, has supplied an implementation of all data structure functions (ds_*), except ds_*_read/write() and ds_set_precision, "whatever the fuck that shit is." He also went on a DND-implementing spree; more trivial, but probably more likely to be noticed (it's surprising how many people fuss about really simple DND tiles missing, just because they threw them in random places).
As for HaRRi, he would like to just point out that C++ sucks, but that he has nonetheless (In addition, of course, to the huge number of font functions he has implemented) been working on an implementation of paths. For which, apparently, IsmAvatar has coded a writer to the EXE. You can expect to be seeing those sometime "in the next 42 years," even if I have to move them somewhere afterward.

Specifically, I would move them into the extensions directory. I spent a little bit establishing a modular system for incorporating underused variables and functions which will allow users to choose which ones they need, disabling the rest which would otherwise just be wasting memory and filesize. I have already moved alarm functions there (including action_set_alarm). With a little luck, Ism will hook that system up formally before I die of old age.

TGMG has been hard at work doing minor tweaks and function implementation to up compatibility further as well. He even wrote a quick, crude implementation of tiles (Don't worry; they're probably still faster than Game Maker's). I'll be replacing his present tile storage and drawing methods with more efficient ones down the road. Specifically, precompiled ones--there's no getting faster, assuming the ratio of fill to vertices is good (which it should be). We collaborated for a while to get some more GM quirks working, and actually made really good progress.

What does this mean for the state of the project?

Presently, our GM compatibility standings are as follows: 81 percent of GM examples harvested from 64Digits compile in ENIGMA when given fillers for missing functions. That means that the system is sound enough to manage all the dynamics of GML provided only that a few more functions are implemented in the backend. Why do nearly 20% fail? "Unforseen circumstances." Namely, they fail where GM's own syntax checker failed.

For example, take these codes:
for (;;;;;i = 0;;;; i < 10;; ;; ; ;i += 1;;; ) {}
// Game Maker doesn't care about semicolons in for(;;), so people got in the habit of doing this:
for (i = 0; i < 10; i += 1;) {}
// It's still wrong, but GM supports it.

if not_a_function
  (100001).variable = true; // GM was okay with this, because instance_nearest(x,y,object0).x would cause a runtime error.

That's right,  instance_nearest(x,y,object0).x would cause a runtime error. Not a syntax error. GM's parser was so poor, it didn't know what to make of that code either. So it did it wrong, and people took advantage of that. But now ENIGMA's left with around 200 games that do that sort of shit. Not necessarily just those two; those two are just common ones that stuck with me. I have a whole laundry list of stupid shit like that to deal with.

As for the project's embrace for C++, with the syntax checker finally supporting macros, it's looking healthier than ever. I may soon add support for full-blown preprocessors.

Anyway, wish us luck. And enjoy the changes.