ENIGMA Forums

General fluff => Announcements => Topic started by: Josh @ Dreamland on February 18, 2010, 11:33:14 PM

Title: Rejoice
Post by: Josh @ Dreamland on February 18, 2010, 11:33:14 PM
Everything I wanted to work at this juncture works now. It's almost bittersweet; the teeth-grating hard part is seemingly over.
There is a problem, though. Typical student-made C lexers take upwards of minutes to parse the STL headers. I figured since I didn't need much of the data a compiler needs, I could do a lexless run-over to gather desired information, thus saving oodles of time. Well, it worked, but not well enough. On average processor load on my pretty average computer, parsing all the STL headers takes between one and two seconds. The latest result being 1.53662. Honestly, it's nothing to cry about, but this means I shouldn't just parse this stuff each compile, meaning Ism and I have to work something out to have LGM load ENIGMA as a DLL. This way, it will be able to retain such information.

I had typed up a report of what comes next on the school computers, but those have lately been failing to post here. I guess I'll summarize:

- As soon as I have access to a Windows box, I will reparse the STL headers on it, and then test windows.h to make sure it works (I have no doubt that it will). I also need to test the GL headers.
- Development will need to be carried out on said Windows device for a few cycles. This is because I assume said device (the one I have picked out) is 64 bit. Serprex had some problems with 64 bit machines in the past; I need to be sure they do not persist.
- Speaking of serp, I need his version of the system. He has been optimizing it for a while now. This will probably all be done inside tomorrow.
- Since the development of ENIGMA's parser, a new GM format has been released, along with a new EGM File format. ENIGMA needs to be versed in this new format. The parser also needs updated to use the CFile parser's list instead of the small list of built-ins I used to keep. This is a large step on the road to full extensibility.

And speaking of extensibility, with it come implications. Logically, I can't optimize for run speed, compile time, and output size all at once. So, here's what seems to be destined:
Other optimizations become difficult to describe and list due to number and insignificance (they only add up in the long run). Not to mention I'm really tired.

Before I go though, I should elaborate on the implications of having removable systems.
Each of ENIGMA's systems is designed now to be optional. Since R3, when compile time was too high, these groups of related functions have been divided into individual source files. The problem with doing so is that many files in ENIGMA are highly interdependent. Many sources need a few members from instances, most often id, sprite_index, and mask_index. To give access to these variables without requiring all variables, R4 will require a super-parent, if you will, that contains only the most basic of locals. Other locals, like speed and direction, will be stored in a derived parent from which all other objects will derive. This will allow for all non-essential locals to be removed without major recompile.

*yawns* I need sleep today.
Title: Re: Rejoice
Post by: Rusky on February 19, 2010, 09:44:37 AM
How fast is "quite fast?" Also, how much can you separate the actual generated c++? One object per file? That would speed up compilation time quite a bit.

Also, your parsing the STL headers is rather redundant. You parse them to give EDL access, and then GCC parses them to compile the C++. You can't get the GCC-parsed version, but Clang has a much better architecture than GCC and when its C++ support is a little more mature you could probably get it to parse the headers for you and share the AST. Clang is also way faster and smaller than GCC while supporting the same things.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 19, 2010, 11:07:54 AM
Anymore i think you're sharing these things just to have something to share. "When its C++ support is a little more mature"? When the hell is that? Should I just wait for them to figure out what they're doing? Mine took a while, but it's done now (or as done as it needs to be). Let me know when Clang is more... mature.

As for "quite fast," it was used in lieu of a number. It meant "much faster than R3 as far as can presently be discerned." Though one object in each file would be nice (and doable) if LGM kept track of which objects were modified since last compile, I don't believe it does, so there isn't much point to putting them in separate files at the moment (I'll ask Ism about it next I see her, actually).
Title: Re: Rejoice
Post by: luiscubal on February 19, 2010, 12:37:08 PM
So you mean you currently merge the whole game into a single file. If so, then I don't see how can 1-2 seconds per compilation be that much.

Also, since so much of the compilation time is spent on the header parsing, rather than linking, I don't see how DLLs can help. You still need the STL/ENIGMA headers.
Title: Re: Rejoice
Post by: The 11th plague of Egypt on February 19, 2010, 04:43:56 PM
Wonderful news. This seems like a great piece of software engineering. Hell, I'm coding in C++ right now, but I can't quite understand much of your previous posts. :o

However, getting rid of all those unused functions and variables will cut down on the file size, which is something that got asked for a number of times.
Title: Re: Rejoice
Post by: Rusky on February 19, 2010, 06:10:17 PM
Clang has a production quality C compiler. It is written in C++ and is self-hosting. LLVM has a better optimizer than GCC. I say it's not ready with C++ yet because that's what they say. They're actively working on getting it to compile major C++ projects like Boost. I bring up Clang because you're having so much trouble parsing C++ yourself, and it has a C++ parser separated from its code generation, and good C++ support isn't far off.
Title: Re: Rejoice
Post by: RetroX on February 19, 2010, 06:52:41 PM
As far as I know, removing unused functions, etc. has a G++ flag, someplace.  Either way, it's great that the parser does it.

And for the record, I have 64-bit Arch Linux installed, in the case that you need a test for a 64-bit Linux machine.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 19, 2010, 09:20:49 PM
luis--
All the object data is in one file, but built-in functions are in many, many different files.

plague--
I'm hoping so.

Rusky--
It would be a great option if the team guaranteed its readiness. Until then, I'll stick with what I've worked on.

Retro--
I've not found such a flag, but I'd think there is one.
I will definitely need such a test done. The time will come for that.
Title: Re: Rejoice
Post by: antidote on February 19, 2010, 10:23:26 PM
http://old.nabble.com/how-to-remove-unused-code--td15548009.html
Title: Re: Rejoice
Post by: Rusky on February 19, 2010, 10:26:29 PM
Of course, I wasn't suggesting moving to Clang right now.
Title: Re: Rejoice
Post by: freezway on February 19, 2010, 11:38:42 PM
wooohoooo cant wait for R4! i am really in need of game maker on linux!
Title: Re: Rejoice
Post by: Game_boy on February 20, 2010, 07:44:11 AM
Quote from: Fede-lasse
Tremendous size? I thought ENIGMA was supposed to make them small, and, in particular, fast, and not fast by GM standards. 0_o

I think he means in testing mode, not when you compile it for release. Actually I think this way is a very good idea, since in GM you could test you game, make a change on one line and test it again in under a minute (whereas that's never been possible in C/C++ for large projects).

So when you need compile speed to be minimal (during testing), it compiles really fast and you don't need to care about size or performance since you're only testing a tiny fraction of gameplay.

And when it compiles for real you get almost the speed of actual C/C++, with a long compile time (but that only needs to be done once) and small executable size.

Is that what you meant Josh?

Title: Re: Rejoice
Post by: Josh @ Dreamland on February 20, 2010, 10:15:42 AM
Game_Boy--
Precisely.

Everyone else--
Listen to Game_Boy.

@"Perhaps you could implement a function set ... to make his/her application look out for errors"
The point of removing error checking is to save on multiple ifs each call. That's part of the reason GM is slow; the runner can never be recompiled to your needs. I would be no more happy to find that a user of my finished game had it throw an error on them (especially one that could be ignored) than to find that it had just acted funny or died (in the case that ignoring the error was inevitable).

@"you/the team has to work more on the error messages"
That's been a goal since the beginning.
Title: Re: Rejoice
Post by: Rusky on February 20, 2010, 12:16:48 PM
In GM you can tell it to log errors but not display them. That way when the game acts funny you still get the errors. I think that's a far better way of doing things than just ignoring errors- using compilation means a lot more errors get caught at compile time and the ones that are really runtime  don't cause enough slowdown to justify removing error checking.
Title: Re: Rejoice
Post by: RetroX on February 20, 2010, 12:21:59 PM
http://old.nabble.com/how-to-remove-unused-code--td15548009.html
Yep, that's exactly what I was looking for.  This is what I usually use on compile:
g++ file.cpp -o file -Os -s
and now when I use this:
g++ file.cpp -o file -Os -s -ffunction-sections -fdata-sections -Wl,-gc-sections

It is slightly or significantly smaller depending on how many unused functions/variables/types/whatever that there are.
Title: Re: Rejoice
Post by: score_under on February 20, 2010, 06:13:26 PM
types
These are used at compile time only.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 20, 2010, 11:00:32 PM
A bounds check every time a sprite is drawn? That's just one of them that really sticks in my mind. I might prevent the untimely death of the game by modding the index by the number of sprites, but aside from that... Hell, I wouldn't even just mod it. I'd waste another ((1 << ceil(log2(sprite_count))) - sprite_count) * sizeof(sprite_struct*) bytes (as in, wrap the sprites to a greater power of two) and use an & instead.

It's the little, "oh, this won't really hurt anything" ideas that end up in projects behaving like GM. "Oh, all those local variables are negligible." "Oh, using a map to store array elements won't slow anything." Whatever I can save, I will, or it'll all build up into slow gunk.

Also, score_, I'm not sure if all structs are compile-only in the sense that they will not affect size if unused. I'm thinking vtables might be stored regardless (Not a C thing).
Title: Re: Rejoice
Post by: Micah on February 21, 2010, 12:05:54 PM
Premature optimization is the root of all evil.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 21, 2010, 12:38:37 PM
Quote
Well, I meant more like being able to check functions only in certain moments where the user would want it using the functions, hopefully without too major speed loss. E.g. you could call "error_start()" to start being error checking session and "error_stop()" to stop it, or perhaps "error_object(obj, true)" to check for errors that occur within a certain object and "error_object(obj, false)". Just suggesting...

The former (error_start) leaves two options.
1) Have an if() to check if it's enabled, which wastes the same a mount of time as just doing the damn error checking
2) Use function pointers that can be swapped on error_start(), which removes any chance of inlining.

The latter (error_object) is a terrible idea by any standard. Now instead of one if(), we're looking up whether we are supposed to be generating errors for that object in a... map? Array? Either of them are awful in one way or another (map's lookup time, array's allocation time).

I'd be better off just leaving the errors than to add functions for them, but I'd rather not have them when they aren't needed.
Title: Re: Rejoice
Post by: IsmAvatar on February 21, 2010, 03:18:12 PM
Quote
Premature optimization is the root of all evil.
I believe this is where Money comes from, too - premature optimization.
Title: Re: Rejoice
Post by: Micah on February 21, 2010, 08:00:43 PM
Money is incredibly convenient and great. The love of money is the root of all evil. So all you get from premature optimization is greed, not actual money.
Title: Re: Rejoice
Post by: RetroX on February 21, 2010, 09:16:58 PM
types
These are used at compile time only.
I wouldn't know, but thanks for letting me know.
Title: Re: Rejoice
Post by: luiscubal on February 22, 2010, 11:48:34 AM
All that matters for the compiled version is not the type, but rather the *size* of the type.
So, if "weirdstruct" and "long long" happen to have the same size, the compiled version does not distinguish both.
An important exception to this rule is double vs int, which happen to have different assembly instructions(opcodes) and registers on some platforms, so in this case compiled code makes a distinction. Even then, only opcodes and registers differ. It can be really hard to figure out the original type of something. typedefs in particular are pretty much like #define.
Title: Re: Rejoice
Post by: serprex on February 22, 2010, 10:46:12 PM
Quote
(03:18:56) Nark Pvermars: draw_mandlebrot
(03:18:57) Nark Pvermars: do it
(03:18:58) Nark Pvermars: fgt
Enigma now supports more drawing functions than GM. At least 2dwise, for now. But don't worry, we'll have d3d_draw_mandlebrot to beat YYG there too when the time comes

And d3d_draw_mandlebrot will have an UNMANLY label too
Title: Re: Rejoice
Post by: score_under on February 23, 2010, 02:15:56 PM
It's funny because neither of you can spell mandelbrot.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 23, 2010, 05:17:23 PM
Oh yes, that's funny. I'm dyslexic, asshole. ...haha.
Fine, maybe I just can't spell old dead people.
Title: Re: Rejoice
Post by: IsmAvatar on February 23, 2010, 05:19:30 PM
It can't be worse than GM, where one of the constants was cr_arrrow.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 23, 2010, 05:21:34 PM
BAHAHAHAHAHAHAHAHAHAHHAHAHAHA
How'd the documentation for that look?
Title: Re: Rejoice
Post by: RetroX on February 23, 2010, 08:54:20 PM
It can't be worse than GM, where one of the constants was cr_arrrow.
...are you serious?
Title: Re: Rejoice
Post by: freezway on February 23, 2010, 10:18:35 PM
just checked it... not only is there cr_arrrow there is ALSO  the correct spelling... cr_arrow.
Title: Re: Rejoice
Post by: Micah on February 23, 2010, 11:50:11 PM
The correct spelling was added in some version, but the misspelled version was kept for backwards compatibility.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 24, 2010, 06:58:36 AM
Boy, I can't wait to add that.
Title: Re: Rejoice
Post by: Rusky on February 26, 2010, 11:14:41 AM
Umm. How do you plan to turn error checking on or off at runtime without an if? You could use indirection and have two version of every function, I guess. O_o
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 26, 2010, 01:12:24 PM
...Nothing error related functions can do to stop or resume errors can ever make the code more efficient. Easy way is to use if's, copying code logically doubles size and makes inlining impossible.
Title: Re: Rejoice
Post by: luiscubal on February 26, 2010, 04:52:23 PM
You could use an #if instead of regular ifs. I'm pretty sure that doesn't affect inlining.

Quote
the misspelled version was kept for backwards compatibility.
Unfortunately, it's a sad reality software development must face. These bugs are so trivial to find and fix, yet their consequences are so harsh that we must think twice before doing so and, in some cases, they are actually kept.
Title: Re: Rejoice
Post by: Josh @ Dreamland on February 26, 2010, 07:16:50 PM
#if is how I was going to do it. But that's compile time, so it means we can't have such a function anyway.
Title: Re: Rejoice
Post by: serprex on March 02, 2010, 05:37:28 AM
Blame Josh and the tendency to spell in the manner recently read. I can't find a case past where I made such an err, and I wrote it as draw_mandelbrot in the source

Also, on the topic of standardized typos, obligatory HTTP REFERER
Title: Re: Rejoice
Post by: IsmAvatar on March 02, 2010, 05:17:07 PM
Quote
Blame Josh and the tendency to spell in the manner recently read. I can't find a case past where I made such an err, and I wrote it as draw_mandelbrot in the source
Not quite a spelling error, but more grammar: "err" is a verb. "error" is the noun form.
Title: Re: Rejoice
Post by: Josh @ Dreamland on March 02, 2010, 10:44:54 PM
Serp shortcuts. Makes a great optimizer.
Title: Re: Rejoice
Post by: serprex on March 07, 2010, 08:04:55 AM
One of the qualities I enjoy of English is how easily verbs and nouns can be interchanged. It's something that doesn't work well in French