General fluff => Announcements => Topic started by: Josh @ Dreamland on December 24, 2012, 08:27:54 am

Title: Christmas Plans
Post by: Josh @ Dreamland on December 24, 2012, 08:27:54 am
First off, Merry Christmas to all of you. Hope you are enjoying some time off for the holidays.

The parser is behaving to expectation, which is great, considering I told you all that it would do everything short of walk on water. However, there are still two problems I see in ENIGMA that I fear will go uncorrected until they bite someone in the ass; I am going to address both of them and one personal problem here.

If you are a developer, try to pay a little attention. At least to the first two.

Problem 1: The extension system is subtly broken.
I don't know if anyone noticed this (I think HaRRi has stumbled upon it?), but the extension system is not as modular as it was designed to be due to issues with uninformed sideways casting. The compiler handles all the casting; the linker is not involved. Thus, extensions think they are the only class that enigma::object_locals inherits virtually. Issue is, they are not. Thus, since alarms are first alphabetically, they're the only extension which will work.

Contrary to popular belief, extensions were designed to facilitate adding "heavy" functions as opposed to "groups of two or more" functions. By "heavy," I mean functions that have weighty dependency that might drive someone who is interested in disk and memory efficiency to drop us a complaint about it. Maybe someone doesn't need a 16 integer array in each object (talking of alarms). Maybe it really bothers someone that their objects all need a path_index variable, as well as position and time variables for their path. The point is, extensions make it so they can remove the entire system from their game, including the local variables that weigh down their objects.

To clarify, the issue is simply that when they need alarms and paths, both extensions assume that their bytes are the first in the structure, and one of them must logically be wrong. The fix is simply to do the casting from within the engine file (which is informed as to what each extension looks like and which extensions are being included). The detriment? We have data misread and life is hard.

Presently, you access the current instance (ENIGMA's equivalent of this) using a global. There are actually a few problems with doing this:
By the last one, I mean loops such as with() need to be extremely cautious about the length of time for which they change the current instance pointer. So far, this has only really entailed being careful to set it back at the end of the loop. It also, again, destroys any hope for threading anything to do with instances.

The solution: I am turning the instance addressing system on its ear.
The solution is actually pretty simple. We dump the globals that represent the current instances/iterators (namely, [snip]ENIGMA_global_instance[/snip] and [snip]instance_event_iterator[/snip]) and we replace them with a parameter given to ENIGMA functions which modify the current instance.

So basically, instead of this function:
Code: (cpp) [Select]
void motion_set(int dir, double newspeed)
    enigma::object_graphics* const inst = ((enigma::object_graphics*)enigma::instance_event_iterator->inst);

We have this function:
Code: (cpp) [Select]
void motion_set(enigma::object_graphics* enigma_this, int dir, double newspeed)

So, not a huge difference, but enough that it will mean some chaos.

If you are worried about the parameter, don't be. That's where the new parser comes in. I have not added this yet, as the pretty printer is not written. I want everyone on board with this idea before I ship it.

This idea has been up on the proposals board for some time; we're just finally to a point where I think I'll have the free time to deal with it. I will put everything up in the ENIGMA-JDI branch before I begin the migration. I'll make a new newspost to let you know the time has come to tackle it and do regression testing, when I'm ready for the change myself.

Again, the benefits are a fixed extension system, the option of threading, and a more stable instance addressing system.

As for the central iterator list (the list of active iterators to be conscious of when deleting shit), I am still happy with it. Though I may refactor it slightly to maintain links in the list correctly instead of moving deleted iterators back (in case an iterator is reused which is expecting one type of object and instead finds another type by mistake). For now, it should be fine.

Problem 2: The Platform-Graphics bridge is ill-conceived and horrible.
The engine directory structure gives the impression that you can use OpenGL or DirectX as your library on Windows. Presently, this isn't the case, as both systems require the window to be initialized in a special way. You may be familiar with WGL and GLX; they are, respectively, the Windows GL interface and the GL-X11 interface. Code for these monsters is found right in the Platforms folder—So far we've avoided possible issues using preprocessors, but these hurt compile time and are in general unattractive.

It's been a clusterfuck so far, because we're dealing with three entities in the Platforms folder instead of one:

It's messy to separate those three items, which is why we have issues.

Solution: Create a new folder of bridge systems.
I don't care where in SHELL this folder is created, but it needs to contain folders such as Win32-OpenGL, Win32-DirectX, X11-OpenGL, etc, for each valid pairing of Window System - Graphics System.

We may also want to separate out the window system code and put it with the widget system code. This is leading to minor technicalities on Linux: The window is governed by raw X11, while the widgets are governed by GTK+. It hasn't led to any problems, but it has the potential to do so—especially on pairings in the future (eg, if some poor bastard tries to write QT widgets).

Since much of the code in the Win32/ folder was written by me when I was 16, it may be a good idea to comb over it again, anyway. If this can serve as an excuse to do so, go ahead and let it.

Personal Problem: I return to school after two weeks.
I have 14 days of freedom before I return to school, and I already need to start making preparations for that. Depending on how I tackle this semester, it may be even more work than the last. The good news? Two things: (1) the course which will generate the most work? It's on game design. That's right, the thing we've all been doing since we were 12. (2) I am taking five courses instead of six, and one of them is philosophy. For those of you who have not taken a college philosophy course, they are an easy, effortless, and even fun A if you have an open mind and don't mind some discussion.

The bad news: The game I design has to use Ogre (or Unity or XNA—Not even considering the latter, not wanting the former since I use Linux), which I have never used before.

The other good news: I intend to deal with this gracefully by writing an Ogre extension for ENIGMA. If I have to deal with Ogre, this project may as well benefit from it.

Now you are all up to speed. In summary, brace for impact.
Title: Re: My Plans
Post by: Josh @ Dreamland on December 24, 2012, 08:52:28 am
Also, IsmAvatar, if you're feeling left out, I have plans for you, too. I was thinking that I could forward you a frontend to the EDL tokenizer so you could get a linear lex of the code for use in the editor. Combined with the parser, it would let you format the code manually, and could also be used for syntax highlighting if you wanted dead (preprocessed-out) code to remain unhighlighted and brace/whatever matching to work accordingly. It would also let you identify which loop break and continue statements belong to, like Eclipse does. It would also let you discriminate between shit used as identifiers, shit used as declarators, and shit used as functions.

Really up to you, though; I'm not gonna put forward the effort if you have no interest in the idea.

Tell me especially if you are interested in it for bracket/parenthesis matching so I can add that in now.
Title: Re: Christmas Plans
Post by: TheExDeus on December 24, 2012, 12:45:47 pm
Sounds like a plan. I have a few problems with ENIGMA which should be fixed (and with the parser might as well will be):
1) The problems with "with(){}" in scripts. As well as global variables and such in scripts.
2) LGM's code editor auto-complete fubar'd. Doesn't scroll with the code and actually should be possible to be modified by design (like putting it at the bottom permanently).

I haven't actually checked the newest version, so maybe all this is fixed. I have quite a few via-ass made extensions for ENIGMA. Maybe there should be a separate section to upload them? Also, documentation has really stalled, but using Doxygen style to comment code really should speed that up.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 24, 2012, 01:31:27 pm
The system described above will fix (1). Or at least it should; I'm not sure what problem you're talking about.
IsmAvatar will have to be persuaded into doing (2), unless I finally tackle making LGM compile. Not signing up for that.

I have been using Doxygen for the new parser/systems as the plan became solid enough to permit. If you want to start documenting things with Doxygen, go ahead. There's not much benefit to doing so, though, when the systems that most need documented will need large, independent Doxygen comments anyway.
Title: Re: Christmas Plans
Post by: TheExDeus on December 24, 2012, 07:26:01 pm
I though using Doxygen with just adding comments before functions like:
Code: [Select]
 * @brief This function takes the most simplest derivative of data_in (which can be a vector, deque, queue etc.) and outputs it in data_out.
 * After the function data_out.size() will equal data_in.size() and the return value will be the maximum of data_out.
 * @param data_in Input data container. Can be any STL container that supports [], size() and resize(). So vectors, queue, deque etc.
 * @param data_out Output data container. Can be the same types like input can, but they don't have to match.
 * @param step Optional step size for the derivative. To get the most precise result it should stay 1.
 * @return Maximum value of data_out container
This is for another project where I use Doxygen, so this is not connected with ENIGMA.

There should be a template or an outline on how to do this. There are like 10 ways to comment things with doxygen and this is just one of them.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 25, 2012, 01:06:11 am
Yes, you'll find JDI is riddled with those comments. They won't help you understand how a system works as a whole, only how individual functions work. If you want to document the system, you have to write up a huge, separate comment using the @page directive, then link to it from a main page (or leave it up to the user to find it in the ToC). Kind of annoying, and not something that is likely to happen considering it's just as easy to create pages on the wiki for that purpose.

I use the same style for my comments, though I don't normally put the * on every line, and I don't always explicitly say @brief.
Title: Re: Christmas Plans
Post by: TheExDeus on December 25, 2012, 06:26:50 am
Well brief is needed for the "brief" description. If you don't do that, then in function list there will be no description and you will have to click on it to jump to the large description. Brief would be extra useful here, as it would be the line that is shown in auto-complete box.

Though trough writing this I realized that you probably meant, that you have the "first line is brief" thing enabled in Doxygen, not that you don't use brief at all.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 25, 2012, 12:27:20 pm
For future reference, by default, the first sentence is used as the brief.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 27, 2012, 12:04:32 pm
Also, who among you is interested in the idea of encapsulating EDL types and functions in their own namespace? It would make it easier to deal with noise from library implementations, but it would require putting lots of shit into that namespace and using lots of shit from STDC.

In other words, we'd no longer have problems with time or list being illegal variable names unless the user specifically said, [snip=EDL]using std::list;[/snip] or [snip=EDL]using namespace std;[/snip], which I intend to unify between all viable backends (meaning, in JavaScript as well).

The new parser is much, much more adept at making these distinctions, so I can probably get it to allow using types and shit as variable names (especially in compatibility mode), but this would be a fix-all.
Title: Re: Christmas Plans
Post by: forthevin on December 28, 2012, 09:18:04 pm
Encapsulation of EDL types and functions would be nice, especially if it solves the problems with "time" and "list" being used as variable names. How easy would it be to set up the system such that EDL types and functions can reside in both the global namespace and the EDL namespace? If it is easy, I would like the system to be set up that way, since that should enable easy step-wise migration.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 28, 2012, 09:52:06 pm
Simple as placing "using namespace <name here>;" in the global scope. Just like you would any other namespace. I'll set ENIGMA up to read from :: or ::edlwhatever based on a macro. After everything's migrated, or we think everything's migrated, you'll just flip that flag and try compiling.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 29, 2012, 06:06:12 pm
Threw you together an avatar, HaRRi. It's all generic and shit.
(https://dl.dropbox.com/u/1052740/img/harriavatar96.png) (https://dl.dropbox.com/u/1052740/img/harriavatar96_um.png) (https://dl.dropbox.com/u/1052740/img/harriavatar96_um2.png) (https://dl.dropbox.com/u/1052740/img/harriavatar96_um3.png) (https://dl.dropbox.com/u/1052740/img/harriavatar2.png)

Take your pick.

Forthevin gets one next. If you have any requests, state them; otherwise your avatar, too, will be made using the first InkScape filters I see.
Title: Re: Christmas Plans
Post by: kkg on December 29, 2012, 09:45:07 pm
they all look terrible
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 29, 2012, 10:45:51 pm
*shrug* I didn't spend very much time on them.
Filters->ABC->Noise Transparency
Place gears, add simple gradients, subtract circles from gears. Done.
Title: Re: Christmas Plans
Post by: TheExDeus on December 29, 2012, 10:58:54 pm
I like the current one just fine. I can make my own if I so desire.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 30, 2012, 01:21:56 am
Very well. At least it's somewhat original. Just likely to get recycled a lot.

In other news, I went ahead and pushed my current working copy of the parser. A lot of integration needs done, but the parser is basically complete. I spent the last day or so shelling out all the old compiler warnings and doing some major reorganization.

This may be done by tomorrow night.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 31, 2012, 02:21:57 am
Didn't happen. Ended up cleaning up some mess in JDI instead, correcting some bugs I was unaware of. Also did some much-needed testing; any test codes welcome so I have something to check before we're ready to unleash this on the masses. Hopefully TGMG will be around to automate that for us again.

In the meantime, the parser reads and gives a dump of test_gml.h in the JDI branch if you build the CBP in either of the standalone modes. It should build without warnings, except in the placeholder file that declares empty versions of the old parse functions to prevent link errors, because the new parser is still not integrated.

In fact, I'm still working out how many passes I should handle up front and how many I should leave to the language plugin. I want to minimize both numbers, but especially the latter, so as to facilitate easy extension of the compiler.
Title: Re: Christmas Plans
Post by: polygone on December 31, 2012, 07:07:39 am
Josh run through the examples posted on the EDC.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on December 31, 2012, 09:44:41 am
I will when it is building games again. For now, all it can do is single code pieces.

Today will be spent moving code around and writing a pretty printer. If all goes well, I'll finish both things and be done. Otherwise... More delay.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 02, 2013, 02:40:59 pm
Just so everyone knows, I am adding something to ENIGMA's specification after the parser is plugged in:
Code: (EDL) [Select]
object0.variable = 10;
int a = object0.id;
a.variable = 20;
object0[a].variable = 30;

Humans can tell what that's doing, but a parser cannot necessarily make that call. Maybe if I spent a great lot of time on it, it could, but in general you should expect this output from the C++ printer:

Code: (CPP) [Select]
((ENIGMA_OBJ_object0*)fetch_instance_by_objectid(object0))->variable = 10;
int a = enigma::fetch_instance_by_objectid(object0)->id;
enigma::varaccess_variable(a) = 20;
((ENIGMA_OBJ_object0*)fetch_instance_by_id(a))->variable = 30;

The idea is that the switch statement used in varaccess_variable is removed, and does not even need to be generated if `variable' is never accessed in that way. It'll still be generated as needed for compatibility, of course.

Note as well that I may optimize it further and generate an inline function for each object as needed, or just access them inline (more likely the former for reasons of error checking and general prettiness). Still, you get the idea.
Title: Re: Christmas Plans
Post by: TheExDeus on January 02, 2013, 04:07:51 pm
Wait, is object0 an object or an instance? If its an object and object0.id returns the first instance's ID, then how does object0[a].variable = 30; work? Like why in the parsed code "a" is not used anywhere?
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 02, 2013, 04:13:26 pm
Because copypasta fail. Thanks. Fix'd.

The variable object0 is an object name, so it's an object_index.
Title: Re: Christmas Plans
Post by: polygone on January 02, 2013, 07:41:30 pm
I don't get the point.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 02, 2013, 09:10:20 pm
You wouldn't.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 03, 2013, 04:47:15 pm
Quick update:

Sorry that this is taking so long. While the parser is basically plugged in, I did not even consider how much work is involved in the other major operation I am taking care of: abstracting the language being compiled to.

The compiler now knows literally nothing about the host language; I have to move tons of ancient code from the global scope to various subclasses to facilitate having multiple implementations for multiple languages. It is genuinely a pain.

I have no accurate ETA, but I'm going to be spending 90% of my waking time on it. It must be done before Sunday night.
Title: Re: Christmas Plans
Post by: forthevin on January 05, 2013, 04:02:10 pm
A couple of thoughts and comments:

First off, spending 90% of one's waking time on something sounds less than healthy. Wouldn't it make more sense to postpone the new parser and compiler if it takes unexpectedly more time? ENIGMA seems to work pretty well at the moment, so I have trouble seeing the hurry.

Second off, I am happy to announce that the particle systems system is coming along well. Several examples work now, including the fireworks example from the manual (http://gamemaker.info/en/manual/412_08_example), as well as a nice fire effect example I found (http://blog.martincrownover.com/gamemaker-examples-tutorials/particle-example-fire/). There is still a lot missing, including effects, changers, destroyers, deflectors and attractors, but the system is quite usable at the moment.

Third off, ENIGMA has been very functional for quite a while, and I would therefore like to propose that a forum is created, named Works in Progress, where games in development can be shown and talked about.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 05, 2013, 06:01:09 pm
The new parser's is important to me because it means we can finally have extensions and additional host languages (eg, JavaScript). I've made the compiler so modular locally that TGMG should have *zero* difficulty finally making the Android part work out of the box.

Moreover, the compiler was in need of some redesign—a lot of it was thrown together hastily, and in a scatterbrained manner. I can't expect other people to maintain it if I can't even cleanly describe its pieces. After this, it should be much easier to tell what is going on in the compiler code.
Title: Re: Christmas Plans
Post by: TheExDeus on January 06, 2013, 08:31:25 am
Also, I do have many problems with the current parser. Some are bugs, others are just a few suggestions that would be very great improvement over GM.
Title: Re: Christmas Plans
Post by: The 11th plague of Egypt on January 08, 2013, 07:31:29 am
I can't expect other people to maintain it if I can't even cleanly describe its pieces. After this, it should be much easier to tell what is going on in the compiler code.
Well, good luck then. I'm getting curious about how this compiler works, I'd like to see some documentation.
Title: Re: Christmas Plans
Post by: Josh @ Dreamland on January 09, 2013, 12:49:33 am
See the commit message from my last commit to get an idea of where I am.

Restructured and reorganized fuck-everything. Changed around fuck-everything.
My brain hurts from thinking about this shit for so long, and some of it still needs modification.
I have encapsulated most of everything correctly.

Remaining considerations:
- Events.res locals are still loaded by lang_CPP; they should be loaded universally.
- Where are events even parsed? I totally lost track. Being honest.
- We need a system for denoting where something was declared, for error reporting purposes.
- JDI handles some of its AST features funny
  - It checks the left-hand of a function call and bails if it's not a function or cast
  - It crams all the literal types into one AST node type
  - It doesn't handle error reporting very well, and, to top it all off,
  - It isn't fucking modular or extensible at all; nothing is abstract or virtual
- I have no idea if a second pass before pretty-print is actually required
- None of this outside the basic parser has been tested, and it's not blowing me away, either
- No amount of therapy will make this commit okay

I have done some more thinking, and determined that if we want to properly support overloading EDL functions regardless of host language (C and JavaScript being prime examples of languages which do not support overloading), we need a second pass after I'm aware of the types of everything in order to pick the correct overload.

Still haven't decided what to do with generics, for that matter, but that's another case.