ENIGMA Forums

General fluff => Announcements => Topic started by: Josh @ Dreamland on March 02, 2010, 11:21:36 pm

Title: Pride
Post by: Josh @ Dreamland on March 02, 2010, 11:21:36 pm
I'm so proud of this little community for not hanging on security considerations when looking to the efficiency ones. :3

Good news. Here's the result of parsing this code:

Code: [Select]
#include <map>
#include <string>
#include <vector>
#include <stack>
#include <queue>
#include <list>
#include <set>
#include <iostream>

#include <windows.h>

Quote
No error.
Parse time: 1509 milliseconds
++++++++++++++++++++++++++++++++++++++++++++++++

Macros (12684) [+ ]
Variables [+ ]
>>d
Define: CreateWindow
  #define CreateWindow CreateWindowA
>>d
Define: CreateWindowA
  #define CreateWindowA(a, b, c, d, e, f, g, h, i, j, k) CreateWindowExA(0,a,b,c,d,e,f,g,h,i,j,k)
>>d
Define: CreateWindowExA
  CreateWindowExA:  Function with 12 parameters, returning HWND__  Dereference path: (params = 786444)*</group>
>>

Since Windows includes everything, and I do mean everything, that anyone could ever need to make a game, there's really no point in assuming the parser needs further work. The parser no longer requires lists of functions and types.

Oh, and, if you noticed the odd parameter count when I asked it to define CWExA, don't worry, it stores that count in a jumbled short. It used to dissect the short and display the actual parameter bounds, but the code that does so got lost in one SVN commit or another.


Since you've all proven to be concerned with efficiency, here are some related tidbits of considerations:

1) The most pressing efficiency loss when treating EDL more like GML than C relates to the switch () statement. Those of you who are particularly well-informed with either GM or ENIGMA know that both engines treat a switch like a less-ugly chain of if()s. The complexity is the same; an == check for each label. A consideration, however, is to make a small, close-to-perfect hash function. Basically, this will allow for var case labels using some template magic. GM People: Stop reading after this next sentence.

I believe this can be done in O(1), maybe with two or three checks a number. Basically, a hash function will be used on the input to each case label, assuming these are constant. (Thanks to the C parser, checking can be done on this when they are not literals [such as 3.14 or "some string"]. How great is that?) Basically, the hash will take the sum of first and last byte of any strings along with its length, do some round() tricks on floats, and leave ints alone (C++ doesn't allow switching anything but int, and doesn't allow case labels of anything but constant int). Now, the thought of course is, what if two strings produce the same hash? That's why one check is done at the case label. You'll see the same idea being employed in hash maps. I owe it to Luda that I was enlightened to it. If anyone has a better idea, of course, by all means...

2) The next most annoying thing is compatibility garbage related to simple functions, like draw_primitive_begin(). From my reckonings, GM uses DirectX's idea of a Vertex Buffer Object. That's not always an option for GL... R3 and lower use glBegin(). This raises a compatibility issue when drawing two at once (this is allowed in GML, but not in GL). The reason I assume GM uses VBO is that it can in fact keep two separate primitives going at once. Long, long ago I decided to deal with the problem by allowing a statically allocated array to be used to hold the points for quick iteration... I made it static because I didn't want the overhead each function call to make sure we hadn't overflown anything and needed more space. However, this meant either an annoying amount of memory overhead, or an equally or more annoying setting to allow the maximum number of points to be set. (I thought 360 would be reasonable for most applications, but...) Anyway, in retrospect, it was an awful idea (good thing it was #if'd out by default). I think the plan should be to try for VBO, but if it doesn't exist, just offer glBegin(), no buffering. (This would break some compatibility for people with Intel cards barely meeting stone-age GL specifications). The good news is, no one who had any decent coding practice would notice the break.

I figured I'd get your opinions on those little things as well. If/when there is a DirectX port, that won't really be a concern (the nice thing is that with DirectX, we can make the same decisions as Yoyo did, good or bad, and no one can complain. Too bad I'm still using GL).


I'd throw in a third thing, but that's probably enough to swallow for one post. Those who would like to put their two cents in on security issues can do so in the previous thread. I guess we'll just keep doing this until I have a public opinion on the decisions that haunted me in early development stages (I want to get it all right this time. Or at least most of it).


While you people decide that, I'll be replacing lists of types and functions in the GML formatter with the new lookup calls. (Also, I think I need to make sure "using namespace" works in the global scope. (http://enigma-dev.org/forums/Smileys/josh/troll.png))
Title: Re: Pride
Post by: Josh @ Dreamland on March 02, 2010, 11:36:56 pm
Yeah, using namespace std works. I asked it to print the using scope... it took five seconds or so...
Asking it to print everything it had just parsed took 36 seconds. Heh...
Title: Re: Pride
Post by: Grundoko on March 03, 2010, 11:13:24 am
I hate to be a nag, and I know release dates are hard to give, especially for a programmer, but in what general time period do you expect R4 to be released?
Title: Re: Pride
Post by: Fede-lasse on March 03, 2010, 12:43:37 pm
Quote
(I thought 360 would be reasonable for most applications, but...)
Why 360? Primitives have nothing to do with circles 0_o.

Also, you use parenthesis too much. It confuses the hell out of me... well, for a while... but stop it anyway...

By the way, regarding Rejoice, then I think you should definitely implement a custom encryption feature like you and Ism suggested. Just thought I'd mention here instead of beauratically posting in two different topics.
Title: Re: Pride
Post by: Josh @ Dreamland on March 03, 2010, 01:49:58 pm
I do love my parentheses (and messing with you). Really, they're mostly for thoughts that might confuse the rest of the sentence; I separated them because they are confusing or because they are pieces of info that those who read them should probably know.

Grundoko:

Since if I even hint it's toward the end of the month, you'll guess what day it is, I may as well tell you I want it out on April Fool's day. I might release some things for testing around the 15th or 20th.
Title: Re: Pride
Post by: Grundoko on March 03, 2010, 04:55:03 pm
April 11th sounds like a good release date :p
Would be a good birthday present.
Title: Re: Pride
Post by: Josh @ Dreamland on March 03, 2010, 06:20:22 pm
*shrug*

Oh, and Fede: Many people use primitives for circle-related effects. Think ripples or rings. Especially awesome with surfaces.
Title: Re: Pride
Post by: freezway on March 03, 2010, 08:11:56 pm
I'll defiantly test!
Title: Re: Pride
Post by: Micah on March 03, 2010, 08:59:01 pm
What will you be defiant about?
Title: Re: Pride
Post by: retep998 on March 03, 2010, 09:30:01 pm
Hell yeah I'll be testing this piece of pure awesomeness.
As soon as Enigma is working with everything I need, I am ditching game maker and the gmc.
Today I realized that the gmc really is just full of flaming nitpicking idiot noobs.
Even the staff were joining in on the flaming.
It was 5 pages of pure crap.
People shouting their opinions and then attacking anyone who said the slightest thing different from them.
I really hope to have a better community here.
Title: Re: Pride
Post by: Micah on March 04, 2010, 12:48:13 am
The signal-to-noise ratio on a forum so large is bound to be much smaller. As Enigma gets more popular, these forums will devolve beyond what they already are to become just as bad.

In fact, in some ways, the GMC is preferable to these forums.
Title: Re: Pride
Post by: The 11th plague of Egypt on March 04, 2010, 10:44:15 am
There are many people in the GMC who helped me solving my coding problems. Kudos to them.

Back to us, will we be able to make games with R4, or just testing?

And what about the frontend? Is it usable?
Title: Re: Pride
Post by: Rusky on March 04, 2010, 11:11:42 am
You could make games with R3, sort of.
The frontend is LGM. It's been usable far longer than Enigma.
Title: Re: Pride
Post by: RetroX on March 04, 2010, 06:09:54 pm
With that hash, wouldn't "hello" and "herro" be considered two identical strings?  What does it check besides the length and first and last characters?
Title: Re: Pride
Post by: Josh @ Dreamland on March 04, 2010, 06:23:53 pm
Retro and other concerned individuals:

In the case that strings are involved, it'd be a hijacked switch. What I was hinting at is basically this (assume that the hash of "hello" and "herro" is 143):

Code: [Select]
case 143:
  if (enigma::sval == "hello") { ... }
  if (enigma::sval == "herro") { ... }

Note, however, a couple things:
1) Ideally, the hash function would be played with a little bit to try to achieve the fewest collisions possible
2) An actual hijacked switched statement would look more like this:
 
Code: [Select]
switch hash()
    case hash: if (...) goto ...
    case hash: if (...) goto ...
    break;
3) The most ideal option, but perhaps a more difficult one, is to do the hash map myself, storing pointers to goto labels in my own array (this is legal in C)
This would look like this:
goto enigma::hashmap_switch_1[sval];
Where operator[] computes the hash, starts raiding pre-defined buckets for the correct label, then returns it for jump. This is how switch() actually works, and how I hope to do it.
Title: Re: Pride
Post by: RetroX on March 04, 2010, 06:33:57 pm
To be honest, I've completely forgotten about goto. :/
Title: Re: Pride
Post by: The 11th plague of Egypt on March 05, 2010, 01:54:34 am
Why should you add such an hack? There is no switch for arrays in C++ and there is no switch in Gm at all,
so neither users will feel a need for it. Using a switch for this kind of things means writing bad code.
Don't even bother wasting time.
Title: Re: Pride
Post by: Josh @ Dreamland on March 05, 2010, 10:34:32 am
It's by no means a hack. GML does support switch(); it will let you switch either a real or string. Switching an array switches its first value.
Just because a code uses goto doesn't mean it's a hack. Goto was excommunicated from use in high level code due to it making said code difficult to read when misused. Case labels are just that: labels. The compiler writes the hash map instead of me doing it. The same will apply to ENIGMA.
Title: Re: Pride
Post by: retep998 on March 05, 2010, 05:05:01 pm
Quote from: Ace Ventura
Oh, and Fede Lasse: Many people use primitives for circle-related effects. Think ripples or rings. Especially awesome with surfaces.
Yeah, I concidered that after some time, but I was like "meh, really?". But then again, when you mention it, I guess people do.
But you'd have to remember that it's not 360 vertices, it's 361 since you need the center vertex.
Title: Re: Pride
Post by: The 11th plague of Egypt on March 05, 2010, 06:23:04 pm
It's by no means a hack. GML does support switch(); it will let you switch either a real or string. Switching an array switches its first value.
Just because a code uses goto doesn't mean it's a hack. Goto was excommunicated from use in high level code due to it making said code difficult to read when misused. Case labels are just that: labels. The compiler writes the hash map instead of me doing it. The same will apply to ENIGMA.
Hell, I didn't know GM had such a statement. Really, I saw no one using it in the GMC.
The first time I heard about it was studying C++

BTW I really hope you aren't using gotos in the rest of your code as well, unless you aim at being a sole developer
Title: Re: Pride
Post by: score_under on March 05, 2010, 06:44:04 pm
BTW I really hope you aren't using gotos in the rest of your code as well, unless you aim at being a sole developer
My goodness, it's like we haven't invented ctrl+f and multitasking yet.
Title: Re: Pride
Post by: Josh @ Dreamland on March 05, 2010, 07:24:18 pm
I use a goto roughly four times in the C parser. My labels are usually descriptive and located nearby. Usually.
Title: Re: Pride
Post by: Micah on March 06, 2010, 01:26:49 pm
Now if you had used recursive descent and operator precedence or some other decent method for writing a parser, it would actually be readable and you probably wouldn't ever have had felt the need to use gotos.

Not to mention that it would have been working and done quite a while ago.
Title: Re: Pride
Post by: Josh @ Dreamland on March 06, 2010, 01:39:35 pm
All right, that's finally starting to bother me. If any of those things were the case, where the hell was anyone that was going to prove that? Hm?
I was the only one doing fuck-anything on this project. I expressed countless times that I hate the recursive decent method. I don't care if you'd rather study one of those methods than learn mine; mine's faster. And no, using recursive-descent would not have made the project finish faster, because I would have given up. Or needed an entire fucking team to help me.

I'm getting sick of your cynical opposition at every turn. If you think you can do this job better than me, feel free. It's even open source. Until that time, stop pretending that because you can read about a method that you can understand it and replicate it faster than I can.

I scarcely think you understand some of the things my parser can read. Let me tell you, I wrote the parser, and there were things it parsed that I didn't understand.
Title: Re: Pride
Post by: Rusky on March 06, 2010, 02:39:10 pm
I have written recursive descent parsers. Yes, by myself, and yes, in less time than however many years you've been working on this one. I read about recursive descent after figuring them out, so I am not pretending anything. I believe you have a case of not-invented-here syndrome, not a magical new faster method for parsing.
Title: Re: Pride
Post by: Josh @ Dreamland on March 06, 2010, 02:43:50 pm
Feel free to read it over and discern what has been reinvented.

...Also, writing a parser for one language does not imply that you can write one for any language. Especially not C++.
Title: Re: Pride
Post by: Rusky on March 06, 2010, 03:54:02 pm
The reason I don't work on the Enigma parser is that I don't really care. I'm interested in how it turns out and I might use it, but I would rather spend my time on a better-designed system. I have no interest in torturing a good design into backwards compatibility with GM's.

GCC and Clang are for sure recursive descent. I'm sure other C++ compilers are recursive descent. It's not a question of how complicated it is, because recursive descent already has an established way to parse complicated things. Your GML parser needs new logic for every little change that recursive descent would handle the same way as everything else.
Title: Re: Pride
Post by: Josh @ Dreamland on March 06, 2010, 08:53:00 pm
What good design would you be torturing with GM's?

Also, the only thing which would require a complete change in the logic of my parsers is a complete change in the language itself. New keywords can be added to the GML parser by changing one line, and to the C++ parser (more comprehensive) by adding in roughly three places (The switch of keywords, the list of token codes, and the handler for each under the semicolon/comma handler).
Title: Re: Pride
Post by: notachair on March 06, 2010, 09:17:05 pm
The reason I don't work on the Enigma parser is that I don't really care. I'm interested in how it turns out and I might use it, but I would rather spend my time on a better-designed system. I have no interest in torturing a good design into backwards compatibility with GM's.

GCC and Clang are for sure recursive descent. I'm sure other C++ compilers are recursive descent. It's not a question of how complicated it is, because recursive descent already has an established way to parse complicated things. Your GML parser needs new logic for every little change that recursive descent would handle the same way as everything else.

If you don't care why are you caring about it
Title: Re: Pride
Post by: Micah on March 06, 2010, 09:28:46 pm
Good design = something much, much better than Game Maker. Although it's not perfect, Construct is an example.
Title: Re: Pride
Post by: Josh @ Dreamland on March 06, 2010, 09:32:16 pm
This forum isn't on or related to Construct. If you want to talk about construct (or any other really great examples of how a parser can go right), we can make you another board next to Shupaer's.

Until then, I don't want to hear how everything I've done could have gone better if I had taken X method from Y project.
Title: Re: Pride
Post by: Rusky on March 06, 2010, 09:39:07 pm
The good design I would use: A real language with types, closures and a good FFI for extensions. A way to extend objects or make simpler structures for things like particles. A good VM to support dynamic language extensions. A good compiler that goes directly to bytecode rather than through C++.

Half or more of that is what you're doing, yes. but you're shoving it into GM's design with doubles to store indexes to everything, with objects and instances in the same space, no infrastructure for extension, etc. and you're having to bend your internal design to match that and C++. For example, the var class. A lot of that work would be unnecessary if you were doing the compilation yourself.

Basically, Enigma has different goals than I do. If you don't want to hear our opinion, stop complaining about your problems.
Title: Re: Pride
Post by: notachair on March 06, 2010, 09:42:30 pm
:munch:
Title: Re: Pride
Post by: Josh @ Dreamland on March 06, 2010, 10:01:09 pm
EDL is the union of GML and C++, not the intersection. If you consider either of those a real language, then I don't see how EDL isn't. EDL has GM's only type, and any type from the C++ STL, as well as any types users care to define. I have no idea how you are defining closures, other than as found in the functional paradigm, and as far as foreign functions go, we have LibFFI for GML's external_define, as well as the ability to just add to your project with other C++ libraries (whether designed specifically for ENIGMA or not). If that's not good enough, feel free to make a suggestion.

Way to extend objects? Easy. Way to make simpler structures? Good thing I just wrote a C++ parser; now you can make as many as you want. I'm having a new script resource added for that purpose. As for "A good compiler that goes directly to bytecode rather than through C++," what the fuck is that? Oh, that's right: A synthetic axiom with the soul purpose of ruling out ENIGMA, not even attached to a valid complaint, such as "not doing it yourself is slow." Perhaps that's because all such complaints could so easily be stricken down.

As for your next paragraph of bitching, it's been an intention of mine to tier ENIGMA to allow for removal of indexing. The first public hint at that was back when I announced that a new instance system I've developed will allow using pointer instead of ID to reference instances. The same could be done with sprites for anyone brave enough (and well-practiced enough) to enable it. I like the idea of C++ being an option; it's a beautiful language, especially when you can create a new project and have the annoying parts done for you. That's all C was ever missing. I would want the language to be a part of the project even if I could create so good an optimizer as that employed by the GCC...

As for your last statement, "If you don't want to hear our opinion, stop complaining about your problems."... this is my damn forum. I paid for it; legally I own it. This is the place where I present the problems of my project, for which this site was founded. Just because you hang around here doesn't mean you are entitled to a say in anything, or that your goals should in any way be reflected by the proceedings of this forum or the project of its topic. I am all for constructive criticism. Making the same suggestion repeatedly as I respectfully decline it isn't constructive, nor is telling me how much better I could have done had I listened to you, especially offering no proof other than child's-play proofs-of-concept and examples of full-fledged teams that have pulled similar stunts.
Title: Re: Pride
Post by: Rusky on March 06, 2010, 11:16:37 pm
Closures are in non-functional languages.
external_define does not let the DLL deal with GM resources.
Bytecode would be for a VM to allow simpler language changes.
You missed the point- I said I would do these things differently.
It is naive to think C is that close to perfect.
Title: Re: Pride
Post by: Josh @ Dreamland on March 07, 2010, 12:07:37 am
Never heard it used that way, only in reference to Lisp (or maybe it was Haskell).
That can easily be fixed with a small API.
I like things compiled natively.
As opposed to simply better?
I'm not about to debate something as subjective as which language is perfect. C++ is perfect for me.
Title: Re: Pride
Post by: serprex on March 07, 2010, 07:40:56 am
I think GML is simple enough that it'd be better to compile it more like many ML/Haskell compilers do, native code with calls to a fat libc-esque library for the language

I've read that some prefer to hand roll their parsers. Josh is one such such. Rusky, you're complaining that Enigma is hard coded to GML? I wasn't aware it was suppose to be anything else

You're a lazy ass. You don't contribute anything to the project but weenieism. Complain with examples; I wrote up a toy (http://github.com/serprex/dysvk) implementation of GML's type system catered to GC and type tracing, something Josh hasn't quite explained to me how he plans to deal with. Nothing really came of it, because I was unsure how to integrate it into Enigma. I recall you wrote a bit of a grammar for GML, that could have been useful. Unfortunately, because you didn't implement a patch to integrate it with Enigma, it's shit for dicks
Title: Re: Pride
Post by: Rusky on March 07, 2010, 02:15:51 pm
Ruby, Java, C++0x, JavaScript, Lua, PHP 5.3 and more all have closures. They are fairly popular with scripting languages.

It is more complicated than just a "small API" to manage resources with a DLL, and to allow the DLL to do non-trivial things with those resources (including things GM doesn't call resources like the screen).

VMs can compile bytecode to native code at start time or JIT, which allows for much more aggressive optimization in addition to more language features. That's a better reason that just liking native code.

I would have the same features (more or less) making it "better," but the architecture would be different, which I would say is better but you wouldn't believe me, so I'll settle for different.

Josh can "hand roll" a parser without making it one big mess.

I may be lazy with respect to Enigma, but you're in no position to judge what I do otherwise. Besides, Josh refused to use my grammar because he didn't write it. I would rather not work on a project with these kinds of problems.
Title: Re: Pride
Post by: luiscubal on March 07, 2010, 04:04:25 pm
C# has delegates, which are basically closures and definitively useful. First class functions are among the greatest innovations in programming in a long time. Saying they are useless is basically like saying object orientation is useless(although the arguments are different, the core point is the same). It is not a coincidence that languages that didn't have them are adding them now.
And Java has ugly closures. One of the most requested features for Java 7 was proper closure support in the syntax.

Funny fact about VMs. Some VMs allow AOT. So why not love VMs? They simplify the code generation step and generating bytecode(just like generating machine code) is faster than generating+compiling C++ code.
Pretty soon, your big projects are going to take hours to compile.
Title: Re: Pride
Post by: score_under on March 07, 2010, 04:31:35 pm
OK, so what is the advantage of compiling something into one form of bytecode and then converting it again (bytecode AOT compilation), over just converting it to that in the first place?
Title: Re: Pride
Post by: Josh @ Dreamland on March 07, 2010, 04:35:56 pm
I don't see how a "'small API'" won't cut it. ENIGMA's resources are stored in an array; giving a DLL read and write access to that array wouldn't take any more than an additional pointer.

That VM idea sounds great when you assume everyone has it installed. I like native code because it runs quickly, doesn't need touched at load time, and doesn't require large VM's that no one has installed. In advance: I don't care about `this awesome language by Google' or `by the Ubuntu team' that Construct was thinking about using because it's so fast, cross platform, and great.

Working from architecture to architecture is nice. It's also not going to happen. Cross-compilation has been invented to come close to the ideals of such.

The parser is only a "big mess" in your opinion. Dedicate some of the time you would spend finding a "decent method" to studying mine before you start criticizing it.

...

It seems the difference between you and me is only that of a few years: In the beginning of the project, I had no direction. I'd see a shiny thing and think WOW THIS WOULD BE GREAT IN ENIGMA! Then I'd see another one and think WOW THIS WOULD BE GREAT IN ENIGMA! Eventually I came to realize that not all of the things that WOULD BE GREAT IN ENIGMA, WOW! can be added; that I have to choose between different systems. I decided that some systems could be added alongside others to be used one at a time when appropriate, such as OpenGL and DirectX, or Win32 and XLib.

I also realized that the foundational language for this project was NOT such a system. And so I chose one. I chose the language that will work on Windows, Mac, and Linux, without it being anyone's fault by my own if it misbehaved, and that would then work on the most non-computer systems (consoles, mainly). That language happened to be C, though I later revised to C++ wanting to provide more to users than GML. C/C++ were my best bet for getting ENIGMA running on the most systems and performing the fastest.

Also in advance: Please spare us your spiel about how C# (or perhaps something similar you may have found) is a beautiful language that half-ass works on Linux as long as you use Mono, and that works on Windows if you're patient, and that works on XBox if you have plenty of time and money, and that works on the Wii if you are X company (look, we have a Barby game written in C# to prove it!). I also don't want to know how many wonderful functional languages people have managed to write a C++-based interpreter for on different consoles. I hate it when people start naming off ten different languages that collaboratively, if magically used all at once, match the ability of C++.

You can, however, feel free to gravitate back to how I should be writing an assembler for each of these architectures myself.

Score_:

They want to pretend that by writing a VM for it, everyone will magically have said VM installed so code will be smaller and cross platform (as one module) and all-around happier for everyone, the end.
Title: Re: Pride
Post by: serprex on March 07, 2010, 05:38:59 pm
Generating code for an AOT based VM like LLVM is more work than generating C. It's designed for supplanting a machine code generator with a more generic case. C is similar. Haskell's GHC allows for GCC as a back end; albeit deprecated for a machine code generator, and now someone has devoted their PhD to implementing an LLVM backend. Multiple Scheme implementations (Bigloo, Gambit, Chicken, etc) use C as a back end. PHC, the php compiler, converts to C++. All these languages support closures. I love first class functions, my design of Fractaler shows it, but we're just trying to work out a translation of GML here, and GML's closure support is something along the lines of passing strings around to be executed with execute_string. Also notice how all the languages I listed have closures. Scheme is a dialect of Lisp, which introduced closures to the game

I'll agree, I believe LLVM could make a suitable backend for a well designed implementation of GM. Enigma isn't such an implementation though. To use LLVM, you'd have to change the manner by which you go about it. We're attacking this is a completely different manner than code generation

Rusky, I'm not judging anything you do outside of this project. I don't even know what you do
Don't pull Josh not accepting anything you do because he didn't do it, you never offered a working fork of Enigma that consists of your work. He's cooperating with the changes I've been making because I'm applying them to a copy of Enigma which can showcase the changes
Title: Re: Pride
Post by: Rusky on March 07, 2010, 08:20:18 pm
If you're planning on adding GC, you can't just hand the DLL a pointer.
Working from architecture to architecture has already happened and you're making it happen again, using compatibility layers.

A VM doesn't need to be a separate thing like the JVM or .NET/Mono. It also doesn't need to be large, especially if it's not a separate installation with all the libraries in the universe. Enigma's engine is linked to each game- a VM could be part of that engine. A VM gives you more portability and more aggressive optimization. You can write a VM in C++ and it will run on other platforms just as easily as if you have the user write in C++. VMs can actually be faster than AOT-compiled C/++, because they know more at optimization time.

I know you can compile a language with closures to C. It's hard to get the closed-over variables to work by-reference, though. I know Enigma is meant to compile GML; in fact, that's my point. I wouldn't try to improve GML by adding C++, I would create a new scripting language or use an existing one, designed to have GC, dynamic typing, closures, etc. You say Enigma is "attacking this in a completely different manner" and again, that's my point. I would make a new system rather than improve GM's. Josh asked-
Quote
What good design would you be torturing with GM's?

The reason I didn't offer a fork is that I don't want to work on a system like Enigma. I'm interested in how it turns out, and I offered a system that would make things easier, but Josh had different priorities. He wanted a hand-coded, unstructured code beautifier written and obfuscated by himself rather than a readable, extensible parser. If he had shown any kind of interest in my system I might have tried to merge it in.
Title: Re: Pride
Post by: Josh @ Dreamland on March 07, 2010, 08:43:12 pm
This is C++. There is no GC. There will not be one.

You would evidently have me include a VM for all concerned architectures (And believe me, there are multiple) in with the executable...? Or rather, the platform independent byte code? That would be ginormous. It'd be a 20MB job just like Yoyo's Mac runner.

And it's irrelevant if you wouldn't better GML by mixing in some C++, because I would. Serp would just as soon leave it as GML. And again, since presently we are the only two people doing anything on the project, it doesn't matter what anyone else would do.

If you are interested in how this project turns out, then by all means feel free to watch. Stop bombarding me with facts about languages you think are better than the one months of work have gone into using. Or have someone else fork it. I'm not just dropping all this in favor of a highly idealized VM to magically solve all our problems.

I'd have shown a lot more interest in your system if I thought it had a chance of doing what mine does, or performing as fast. It never would have made it through C++.


Your ideals on the subject seem to be just that. If you could find another C++ parser project to expediently obtain all the class and function definitions for users to take advantage of, that'd be a great start. I don't trust it, maybe you do. Seems all the projects you pointed out parse their own dialect and their own headers rather than that and those of the GCC. So, why do I want GCC's? Because it compiles for every platform I intend to target. Every. One of them.

There isn't a language nor a C++ compiler that you will pick out that can do all the GCC can. To be able to parse the same things as GCC ensures that it is always an option to include headers for the current system. It was my best option from the start, it still is.

Maybe you've noticed that I'm working from the broad to the specific, not vice-versa. Starting with GL rather than DirectX, and other such strategies where possible. If one day it becomes a decent consideration to write my own VM for ENIGMA, then maybe I will. I can't see that happening at this point; the project is young and a 20 MB RUNNER would only scare people away.
Title: Re: Pride
Post by: Rusky on March 07, 2010, 09:45:16 pm
You are only being "bombarded" with answers to your own questions. What I would do is completely relevant, because I am explaining why I don't work on Enigma in answer to your question.

This is Game Maker with C++ as an afterthought. How are you going to manage memory? Just let it leak? Have the user manage it and get rid of a major reason to use a system like Enigma?

You only need one VM in an executable, not one for every architecture. You need a separate executable for every architecture anyway. Thus, a 20Mb runner is a myth.

Compilers can be separated from their standard libraries. Clang is more standards-compliant than GCC. There exist compiler-neutral standard libraries.
Title: Re: Pride
Post by: Josh @ Dreamland on March 07, 2010, 09:52:38 pm
No one asked why you didn't work, we stated that you didn't. You defended yourself. We didn't ask for such.

C++ isn't an afterthought to me. Granted, it wasn't truly part of the plan from the beginning, but as far as I am concerned, it was. I will be treating it like it was.

If the users allocate things, it's up to them to free them. C++ has destructors; what more does it need? The only place I introduced anything resembling a GC was for instance_destroy, and that was only to prevent a delete this; followed by invalidating an iterator. Nothing else in the project is particularly prone to leak...

"You need a separate executable for every architecture anyway." Good, may as well make it all native and strip unused goodies as per the original plan.

"Clang is more standards-compliant than GCC." Again, I'm not using Clang. I don't care how standards compliant it is; I don't see it compiling for all the things the GCC compiles for.
Title: Re: Pride
Post by: Fede-lasse on March 08, 2010, 08:25:00 am
(http://enigma-dev.org/forums/Smileys/somethingawful/munch.gif)
Title: Re: Pride
Post by: Rusky on March 08, 2010, 10:22:23 am
Of course you didn't initially ask. But after I stated my reasons, you asked "What good design would you be torturing with GM's?" and that's when you were "bombarded with languages" or whatever you called it.

C++ isn't an afterthought for Enigma, but it is for GM. Enigma's design is GM + C++, it was created without C++ in mind.
The user doesn't allocate a the things that do get allocated- by the engine. Are you going to leave all the rooms with all the backgrounds and all the sprites loaded the entire time? Even if you use all stack-based allocation for user code, you have to have some way to deallocate stuff the user doesn't allocate.

You can't have a windows and linux executable in the same file. You wouldn't be putting the linux VM in a windows executable. Basically, size is not a reason to use a VM as you claimed. So your reason for using a VM is... you aren't already doing it? Size and speed aren't problems as you've seen.

LLVM supports x86 and PowerPC, 32 and 64 bit versions, Arm, and several other processors. What else are you planning to support? The fact that Clang is standards-complaint nullifies the advantage you claim GCC has in parsing the standard library. Clang is the one with the advantage there- it will parse it for you. With GCC you have to do it yourself, so it all gets parsed twice.
Title: Re: Pride
Post by: retep998 on March 08, 2010, 11:45:28 am
@Rusky - Kindly STFU and stop complaining. If you don't feel like adding stuff to it, then don't tell Josh to do it. He's put a lot of effort into this project, and he doesn't need someone complaining about his methods. If you think you can do better, do so. Otherwise, STFU.
Title: Re: Pride
Post by: Rusky on March 08, 2010, 01:23:52 pm
I'm not complaining. I'm explaining what I would do (and am doing) differently.
Title: Re: Pride
Post by: Josh @ Dreamland on March 08, 2010, 02:05:45 pm
"Enigma's design is GM + C++, it was created without C++ in mind." Assuming it is the fault of a typo that the sentence is contradictory, you are apparently trying to tell me what was on my own mind as I created the project...

"Are you going to leave all the rooms with all the backgrounds and all the sprites loaded the entire time?"
Considering that this is exactly what GM does, I don't see doing so as a problem. I intend to implement a system to use object grouping data to allow importing and exporting blocks of resources, maybe on a basis that won't need the game to freeze (reading a small amount each step so small cutscenes can run in the game in the meantime). However, this is irrelevant to the current argument. You are asking me to include a garbage collector for data the *engine* works with? Garbage collectors are for the lazy. They aren't very slow as of late, but they will never be as fast as handling the allocation and freeing linearly,except, maybe, with a well-optimized VM the likes of which will not be written for this project and thus is again irrelevant. I'm not writing a garbage collector for my own convenience, and I cannot make accurate guesses at which resources will be added when, due a lot to how GML is structured.

"So your reason for using a VM is... you aren't already doing it?"
Incorrect. The only reason to use a VM for this project is to have a once-compiled, cross-platform module. The necessary VM to run such a project on each system would need to be installed on each system, however, thus totally ruining that idea. And don't give me any shit about how many people have LLVM installed; there are people out there who don't even have Java installed, and that's giving me a hard enough time as it is.

Also, I didn't claim that GCC had an advantage in parsing *the* standard library. I implied it had an advantage in parsing *its* standard library. Which is better than Clang's because it comes with GCC, which is better than Clang because there is a version of GCC to cross compile for everything I ever wanted to port ENIGMA to (my favorite presently being the Wii). All the Wii Brew developers utilize GCC, last I checked. Feel free to move to their forums and convince them to switch to Clang; I'm sure none of them will laugh at you.

"Your search - site:wiibrew.org clang - did not match any documents."
No one there even advocates the noise the project was named after. :(
Title: Re: Pride
Post by: Micah on March 08, 2010, 04:28:45 pm
He was explaining that the bulk of Enigma is already designed, as Game Maker. Game Maker was designed completely separate from C++.

So your argument against including GC is that real programmers don't use GC? In that case, you should stop using C++, because real programmers use butterflies. It would be really stupid not to include some way to get rid of the crap that users allocate. That's one of the reasons GM takes up so much memory, and I thought a main goal of this project was to make a more efficient GM.

You're completely failing. A VM is not something that has to be installed on the user's computer. The Lua VM is very, very small, and it's almost always embedded in the programs that use it. Look at Löve2D. LLVM works the same way.

Clang parses the whole thing for you, so you would be done already with what you've been working on for... how many months?

This was mentioned a while ago, but I'll answer it now. Closures are not only in functional languages. C#, Ruby, and JavaScript all have closures, all of which are not functional languages at all.
Title: Re: Pride
Post by: Rusky on March 08, 2010, 05:01:17 pm
Sorry, GM was originally created without C++ in mind. "It" was rather ambiguous.

GM also uses a lot of memory; why would you use what it does as justification? Also, manually allocating and freeing requires a different kind of heap than what GC usually uses. It's not really faster, it's just got the load in a different place. Have you ever written malloc? The only difference is that GC puts the load on freeing while manual puts the work on allocation. If the collector ran at the right time or in a different thread, it could potentially be faster. Finally, I never suggested making Enigma's internals garbage collected, only the parts that the user interfaces with or writes.

Ack. Having a once-compiled module is one advantage, yes. But the user would not have to install the VM on their system. Just like GM games include the runner and Enigma links the engine to the game, it could be included with the game. Literally nobody has LLVM installed, because it's a library intended to work exactly this way. There are also several other advantages of a VM. The game module could be easily separated for use with different runners simultaneously like instant-play. Optimization can be more aggressive with JIT. The engine's API, the runner and the games would be decoupled.

Clang is GCC compatible. The platform it's built on supports PowerPC. There's no reason for it not to support Wii in the future (just like it will support the rest of C++ in the future). I did not suggest using Clang right now. However, Clang has a separate library that will parse C++ for you, which would save you the work of having to do it yourself as well as allow future versions to cooperate with the Clang compiler to get rid of all the unnecessary work.

Just so you know, again, I am not advocating Enigma using these tools or even techniques. I'm just giving you the design you asked for.
Title: Re: Pride
Post by: Josh @ Dreamland on March 08, 2010, 05:38:54 pm
Quote
So your argument against including GC is that real programmers don't use GC? In that case, you should stop using C++, because real programmers use butterflies. It would be really stupid not to include some way to get rid of the crap that users allocate. That's one of the reasons GM takes up so much memory, and I thought a main goal of this project was to make a more efficient GM.
I didn't say I wanted to act anything like a real programmer, I said that garbage collecting my own code was lazy and that GML doesn't have anything worth garbage collecting, especially being so spontaneous as a language. Again, GM users don't allocate anything. I've sometimes opened files and forgotten to close them, yes, but that can't be garbage collected. Nothing in GM can be, because everything is an integer. There is no reference, weak or strong. A reference can be pulled out of one's ass.

Quote
You're completely failing. A VM is not something that has to be installed on the user's computer. The Lua VM is very, very small, and it's almost always embedded in the programs that use it. Look at Löve2D.
I'm aware. I couldn't tell what in specific Rusky was promoting, probably due to the fact that he claims not to be advocating anything in specific.

Quote
Clang parses the whole thing for you, so you would be done already with what you've been working on for... how many months?
Almost half a year, thanks. Probably less than Clang spent on theirs, or has left before they say it supports all of C++.


Quote
GM also uses a lot of memory; why would you use what it does as justification? Also, manually allocating and freeing requires a different kind of heap than what GC usually uses. It's not really faster, it's just got the load in a different place. Have you ever written malloc? The only difference is that GC puts the load on freeing while manual puts the work on allocation. If the collector ran at the right time or in a different thread, it could potentially be faster. Finally, I never suggested making Enigma's internals garbage collected, only the parts that the user interfaces with or writes.
That wasn't meant to be justification in the sense that it was excuse; I should have been more specific. I can't just load and dump resources as they come in and out of need, simply because I never really know when they're needed. GM always keeps them in memory; when the user wants them, they are there instantly. The best I can do is provide functions to give the user more control of that.

Like I said, not much in the GML really needs freed. GM offers no method of freeing anything except through functions ending in _destroy. I suppose those could have threaded frees, but I'm not sure it'd be worth it to do so.

Quote
Ack. Having a once-compiled module is one advantage, yes. But the user would not have to install the VM on their system. Just like GM games include the runner and Enigma links the engine to the game, it could be included with the game. Literally nobody has LLVM installed, because it's a library intended to work exactly this way. There are also several other advantages of a VM. The game module could be easily separated for use with different runners simultaneously like instant-play. Optimization can be more aggressive with JIT. The engine's API, the runner and the games would be decoupled.
I know little about LLVM. Simply have no interest in it, especially seeing that the only advantage it seems to have is a potentially more aggressive optimization, which no one has backed up (in advance: "it's a fact" will not back this up). That separation doesn't seem to be an advantage. In fact, I take it for more of a problem; the same problem half the GMC was screaming about. ENIGMA games compress to fit inside 300 KB. Last I recall, they compress to fit in just over 100. This sounds like another attack on compression and step back toward decompilation-made-simple. Yes, obfuscation... Just like Yoyo's been promising but never delivered... Not good enough if they did.

Quote
Clang is GCC compatible. The platform it's built on supports PowerPC. There's no reason for it not to support Wii in the future (just like it will support the rest of C++ in the future). I did not suggest using Clang right now. However, Clang has a separate library that will parse C++ for you, which would save you the work of having to do it yourself as well as allow future versions to cooperate with the Clang compiler to get rid of all the unnecessary work.
GCC compatibility is a plus you'd not yet advertised. You did mention it supports PowerPC, but just because there is no reason for something not to happen does not mean it is going to happen. There's also no reason for it to support Wii in the future. Just because you are enamored with it, and maybe others are, doesn't mean someone will take the effort to do for it what they did for GCC. The work is done, and people are happy with it. Passing the parse work off to clang would have been nice if I knew I could trust it and I knew so from the beginning. Since my parser's done now, and in the format I like, I have no reason to switch to anything else from here on out. Unless something really stupid happens.

Again, I didn't ask for a design. I wrote my own, and it's done now. I don't want to hear about other techniques unless they are genuinely worth the work to replace what I have with them.
Title: Re: Pride
Post by: Rusky on March 08, 2010, 06:26:54 pm
So there's no strings, surfaces, data structures, sounds or images that you could ever possibly delete?

How could you not tell what I was promoting VM-wise?
You can't have a windows and linux executable in the same file. You wouldn't be putting the linux VM in a windows executable.
You only need one VM in an executable, not one for every architecture. You need a separate executable for every architecture anyway. Thus, a 20Mb runner is a myth.
A VM doesn't need to be a separate thing like the JVM or .NET/Mono. It also doesn't need to be large, especially if it's not a separate installation with all the libraries in the universe.
I think I made myself fairly clear on the nature of the VM I was describing.

Clang has been going for at least 2 and a half years. LLVM has been around far longer- nearly 10 years. Clang's C++ parser is done. What's not done is public/private/protected/friend checking and some code generation. This means that if you were to use Clang's parser you would have a complete (and not redundant) parser in the time it would take you to set up a library, rather than in the time it would take to write a whatever-you-have. That's a good thing.

LLVM has its website to back itself up. Check it out. Trying to prevent decompilation by using native code is just silly. What harm has come out of the decompiler other than the frightening of a few 10-year-olds? Size might be an argument, but any decent-sized game's resources are going to be much, much larger than its engine, and if you include execute_string/file then you have to have some kind of parser anyway and it may as well be the same (removable) one.

To support Wii, you need to 1) compile for its architecture and 2) link with its libraries. Clang can already do the first. The homebrew libraries are for GCC, so Clang could support Wii without the developers themselves even caring. I doubt the GCC developers do. There wouldn't be any work beyond possibly recompiling the Wii libraries.

Again, I didn't ask for a design. I wrote my own, and it's done now. I don't want to hear about other techniques unless they are genuinely worth the work to replace what I have with them.
I quote:
Quote from: Josh @ Dreamland
What good design would you be torturing with GM's?
Title: Re: Pride
Post by: Micah on March 08, 2010, 07:12:14 pm
Nothing in GM can be [garbage collected], because everything is an integer. There is no reference, weak or strong. A reference can be pulled out of one's ass.
So can pointers in C++. Does that mean that C++ can't be garbage collected?

Almost half a year, thanks. Probably less than Clang spent on theirs, or has left before they say it supports all of C++.
Yours has absolutely no way of being adapted to do anything but get information from template declarations. Theirs turns C(++) into LLVM bytecode. Bragging that yours took less time, even if it did, means nothing.

Clang has many advantages over GCC, one of which is incredibly better error messages. And since it can be used on anything with LLVM support, LLVM may very well be ported to the Wii. NOTE: I am not advocating that you switch to Clang anytime soon.

LLVM isn't much higher-level than normal assembly; it's just not machine-specific. It's nothing like JVM or .NET bytecode. It wouldn't really be any more decompilable than machine code from GCC.

Anyway, the whole decompilation and obfuscation issue is moronic. Who cares if people can decompile your game and see its source code? What are they going to do with it? Claim it's their own? How are they going to get away with that? Steal the ideas in the code? Don't flatter yourself.
Title: Re: Pride
Post by: Josh @ Dreamland on March 08, 2010, 09:37:58 pm
"So can pointers in C++. Does that mean that C++ can't be garbage collected?"
"So there's no strings, surfaces, data structures, sounds or images that you could ever possibly delete?"

C++ could be garbage collected if I replaced all calls to malloc with my own. It is because GM integers are only incremented by one, and are completely ambiguous. Case-in-point: object0 == sprite0, sprite0 == sound0, sound0 == path0. The only unique IDs are for instances, and, well, that's the only thing I come close to garbage collecting. People often shortcut and reference some sprites by integer instead of full identifier. I have also seen choose(spritex,spritey...), and with that, of course, random() used in the mix. There's no garbage collecting resources that are identified like that. My best bet is to assume they are in use until _destroy is called. Good garbage collectors work by tracking references rather than pointers, especially ambiguous ID's representing pointers in an array.

"I think I made myself fairly clear on the nature of the VM I was describing."
Yes, reading that over, now that my expectations for a good reason to use such a VM are zero, I see what you are describing. I was just looking for something to set it off as a better idea than native code. Granted that being able to discover the architecture the program is on and run some specific optimizations for it at load time is nice. However, I don't see it as necessary for the extra work on the platforms for which I would/could include it.

"LLVM has its website to back itself up"
I don't see a single statistic, though it is a nice site compared to others of its breed. They even invested in background textures.

"LLVM isn't much higher-level than normal assembly; it's just not machine-specific"
Which makes it a fraction of the work to write a decompiler for. Though I suppose that can be dropped as a concern since they do optimize it some at compile/link time.

"Anyway, the whole decompilation and obfuscation issue is moronic. Who cares if people can decompile your game and see its source code? What are they going to do with it? Claim it's their own? How are they going to get away with that? Steal the ideas in the code? Don't flatter yourself."
Everyone says that. I agree with most of it. Doesn't mean everyone will kiss ENIGMA's ass for using that same excuse. Some people just like feeling more secure.

Anyway, all this future talk. Clearly, the closest thing Clang has to what I need is its parser, assuming it can really tell me what can be used as a type, and verify if X is a member of Y with a decent complexity. I will add that to my list of considerations for if the one I wrote should ever fail to be enough. Until then, I don't need it, and other than that, the project is no use to me.
Title: Re: Pride
Post by: Rusky on March 08, 2010, 10:22:47 pm
Instances could easily be garbage collected- maybe not in the traditional sense, but at least freed when you leave the room. Resources could probably be left unloaded until they were referenced either by loading a room or using them in a function that actually needs them.

Clang stats galore (http://clang.llvm.org/features.html). How does being machine-agnostic make decompilation easier? Most native instructions have nothing to do with the architecture- move, jump, branch, arithmetic, call, etc.
Title: Re: Pride
Post by: Josh @ Dreamland on March 08, 2010, 10:26:55 pm
But the bytes coding for those may vary. Now we know it doesn't; to write a decompiler for one is to have one secured for them all. There is only one.
And by stats, I meant benchmarks of LLVM-JIT code versus native on different platforms and architectures. Not just a list of over-elaborated features.

Also, I don't want to do a check to see if the sprite or background has been loaded every time you draw it. The less such checks, the better.
If GM offers such (which I think it may), it will be replaced with a more hands-on system, driven by functions or user-defined events.
Title: Re: Pride
Post by: serprex on March 08, 2010, 10:44:09 pm
Those stats are for the compiler, not the output. Josh, you're still not getting that LLVM is a native code compiler. Here's some links by Dons comparing GHC's native, C, and LLVM backends
http://donsbot.wordpress.com/2010/02/21/smoking-fast-haskell-code-using-ghcs-new-llvm-codegen
http://donsbot.wordpress.com/2010/03/01/evolving-faster-haskell-programs-now-with-llvm

And no JITs on Xbox: http://lua-users.org/lists/lua-l/2009-11/msg00130.html
Title: Re: Pride
Post by: Rusky on March 08, 2010, 10:59:58 pm
...The bytes for different architectures are all well-documented, or GCC wouldn't target them. Yes, you have to do more work, but with a system like Enigma, what are the chances a game will be published on x86 even if it's on other platforms too?

That page I gave you had benchmarks for the compiler, not just a list of features. Also, LLVM compiles to native code. It can be JIT, which is how you would probably do it in a VM, but it could also be AOT. In fact, Clang is AOT. It's just a normal compiler. BTW, thanks for the links, serprex.

You wouldn't have to check every time you draw an image- just when the object responsible for it is created. It would be a tradeoff, though- more memory usage.
Title: Re: Pride
Post by: Josh @ Dreamland on March 08, 2010, 11:01:08 pm
I understood it to be machine-independent code that is optimized and loaded native, JIT.
I assume it should be at least somewhat faster than code compiled for a particular architecture but not optimized for a particular system. I'd like to know by how much on average it's faster, to see if it at all justifies its use.

Memory is cheap. I can't know which objects will use a certain resource, everything's an integer. Doing so by sprites I know will be used is begging for trouble.

Assuming Haskell and C's performance boosts are the same, that's not a bad gain, but I'm not sure it's worth the effort.