Pages: [1]
Author Topic: ENIGMA progress 28.01.2016  (Read 876 times)
Offline (Unknown gender) TheExDeus
Posted on: January 28, 2016, 09:56:05 AM

Joined: Apr 2008
Posts: 1886

View Profile
This is continuation of this topic (this was meant to be a post there, but I now understand it would be confusing). I wanted to make a new one several months ago, but didn't manage to do so. So this will show progress between July 26 2015 and today. Fixed at least 36 bugs (probably more, but depends on how you count) and added 53 functions to main ENIGMA and more than 60 in BGUI extension.

Custom vertex attributes
One thing that we lacked in GL3 was custom vertex attribute data for shaders. This meant we couldn't do a lot of custom calculations and effects. GM has vertex_buffer for that and vertex_format. I decided to remove code duplication so for ENIGMA it is done trough the model interface. An example of adding a group attribute to vertex:
Code: (EDL) [Select]
format = vertex_format_create();
vertex_format_add(format, vertex_type_float3, glsl_get_attribute_location(shr_group_modify_per_pixel, "in_Position"));
vertex_format_add(format, vertex_type_float3, glsl_get_attribute_location(shr_group_modify_per_pixel, "in_Normal"));
vertex_format_add(format, vertex_type_float, glsl_get_attribute_location(shr_group_modify_per_pixel, "in_Group"));

model = d3d_model_create();
d3d_model_format(model , format);
d3d_model_primitive_begin( model, pr_trianglelist );
for (int i=0; i<100; i++){
  //Fill with vertices like so
  d3d_model_add_float3( model, i*5, 0, 0);
  d3d_model_add_float3( model, 0, 0, 1);
  d3d_model_add_float( model, i ); //Group
And GLSL vertex shader:
Code: (EDL) [Select]
in vec3 in_Position;   // (x,y,z)
in vec3 in_Normal;           // (x,y,z)
in vec in_Group;

out vec4 v_Color;
void main()
  v_Color = vec4(in_Group,0.0,0.0,1.0); //Set red color based on group
Now when rendered the shader will have access to in_Group attribute and will be able to use it. This useful in animation (both morph targets and skeleton animation), dynamic terrain, particle systems etc.
For example, drawing groups on model and then using custom attributes to use them can be seen here:
If we also send weight (additional float) and another transformation matrix, then we can do this:
I will also try making examples to post here. Ones I have planned are morph target animation and shadowmapping.

More low level stencil buffer control
In previous update I shown stencil functions together with an example. Sadly it was a bad example and didn't show more "powerful" things you can do with it. Here is a CSG example where we do subtraction of two meshes using stencil buffers, depth buffer control and a custom shader to merge depth maps from a surface to screenbuffer.
Code example:
Code: (EDL) [Select]
//CSG test


// Draw furthest front face
//Draw the main object normally

// Count back-facing surfaces behind
d3d_stencil_function(rs_always, 0, ~0);
d3d_stencil_operator(rs_keep, rs_keep, rs_incr);

d3d_stencil_function(rs_equal, 1, ~0);
d3d_stencil_operator(rs_keep, rs_keep, rs_keep);

// Reset pixels where n != stencil
d3d_stencil_function(rs_always, 1, ~0);
d3d_stencil_operator(rs_keep, rs_keep, rs_replace);

d3d_stencil_function(rs_equal, 1, ~0);
d3d_stencil_operator(rs_zero, rs_zero, rs_zero);

// Draw RGB image


//Disable everything


And output if using two d3d_models:

Added a textbox widget to BGUI extension. Supports unlimited and limited length (both line count and line length), non-monospaced fonts (so you can use any font you like), mouse selection as well as most keyboard functions (copy ctrl+c, paste ctrl+v, move cursor with cursor keys, select with cursor keys while holding shift etc.). It has styles for the marker as well so when you select text you can make it look how you want.
Things to do:
1) Add unicode support (will be limited to the font selection you do in LGM of course).
2) More usability and bug fix changes for keyboard (some buttons like alt breaks it now).
3) Pasting multiline string into limited line textbox only partly works.
4) Make it independent of FPS using delta-time. It uses internal counters for key repeating and cursor blink, which needs to be changeable.

-Added optimizations in BGUI - if the text of the GUI element is empty, then a lot of code is now skipped. Extra useful with parenting like adding icons to buttons, where buttons could have no title text and the icons (gui_label) wouldn't have any text either. commit
-For some reason the texture atlas code threw a compile time error on GCC 4.8. Fixed that. commit
-Removed all warnings from the ENIGMA.exe source. commit
-When a glyph outside of font range was used then some functions would segfault. Now when these characters are used they will be substituted by a blank space. Note that _ext functions will not break lines on this blank space. The fixed functions are: string_width_ext, string_width_ext_line, string_width_ext_line_count, draw_text_ext, draw_text_ext_transformed, draw_text_ext_transformed_color and draw_text_ext_color. commit
-string_height_ext, string_width_ext_line and string_width_ext_line_count now properly uses get_space_width() to determine the width of space character instead of the height/3 backup. This is in case if the font actually defines a space character. commit Fixes 945
-Fixes a few warnings in lang_CPP.cpp. commit
-Added exists() to eYaml parser, so we can check if value exists.
-64bit compile mode on Windows now uses a unique directory name ("Target-platform" in .ey). This means that you can have compiled 32bit and 64bit ENIGMA in parallel. Uses the exists() to implement build-dir. If it doesn't exist the "Target-platform" is used like before, but if it is defined then it is used instead. commit
-Parser uses a lot of unsafe pointers everywhere. And it doesn't check for NULL anywhere, so there are many cases where a NULL pointer is cast to string and a segfault occurs. Here are some basic fixes in one case, but I'm sure this is where most of the problems lie. This fix made crashes somewhat fewer. commit
-Also casting is a lot better (using  C++11 functions), so no need to use unsafe functions like sprintf. commit
-Changed compiler makefile so it runs from Win cmd. commit
-font_get_ functions return int instead of unsigned int. This was especially bad because of this humorous line: "unsigned(-1)". commit
-string_ ... _line functions now return 0 if the line selected doesn't actually exist. Previously it returned the length of the last valid line. commit
-Now all returns in the string_width functions are "ceil()'ed", not only the last return. I'm not sure this is correct, but Robert back in June 2014 believed they should be (a81fea0). commit
-clipboard_set_text now replaces all "\n" with "\r\n" which is required for Windows clipboard to pass newlines. commit
-If model_begin(format) is done with format = -2 (which is the default) the format will not be changed. This is useful when use several _begin and _end functions in one model. commit
-Parser now uses std::to_string() to actually implement toString. Should be a lot safer. commit
-Fixed d3d_transform_stack_disGard to be d3d_transform_stack_disCard. Same with d3d_projection_stack_discard. This seems to be a typo done many years ago, but nobody has used that function. commit
-Better errors for shader uniforms (will print its name). commit
-Speed optimizations in gsmath by loop unrolling and other things. This should speed up rendering. commit
-d3d_projection_stack functions now push and pop view_matrix as well. Without it the projection stack didn't actually work. commit
-Fixed a potential SEGFAULT with surface_getpixel_ext. commit
-gui_slider_set_value now properly checks bounds. commit
-d3d_model_get_stride returns pre-calculated stride instead of recalculation. This should be a lot faster in many cases, as this is called every time a texture batch check happens. Basically every time a draw function is used. commit
-Fixed surface error in debug mode. commit
-get_texture in GL3textures is now a macro and shows an error in debug mode. commit
-Fixed a tile bug in GL3 that meant the wrong texture is used. Screwed up rendering with tiles. Thanks to rcobra's example for finding this bug. commit
-Enabled C++11 when compiling on Linux. commit
-Matrix4::init_camera_transform (used in d3d_set_projection_ext) is now slightly faster, because matrix mult was replace with 3 dot products. commit
-Math function lerp() now uses C++11 std::fma function. It is as fast (or faster) than what was done previously, but it returns correct results when t == 1 or when x == y. commit
-Added a GM compatible overload for draw_button that lacks the border size argument and is hardcoded to size 2. commit Fixes 951
-screen_init, screen_save and screen_save_part now properly end the shape batching, so you can actually see in the output image the same things you saw on the screen. commit
-texture_set_repeat_ and texture_set_wrap_ now uses GL_CLAMP_TO_EDGE instead of GL_CLAMP in GL3, as GL_CLAMP is deprecated. commit
-Dr.Memory reports some uninitialized read's and accesses in GLSLshader structs. Fixed by initializing it. The rest of the warnings seem to be in driver. commit
-Removed d3d_depth_clear as we have d3d_clear_depth and it actually didn't clear depth. Yeah, confusing. commit
-GL3ModelStruct Clear() now check if we have anything to clear (aka, stride is not 0). Potentially an optimization, as clear() is called on every batch flush. commit
-Shader functions which change state now call batch flush. This means that if something is drawn, then uniform is changed, then drawn some more. The two drawings will be separate as intended. commit
-texture_reset() now checks if we already don't have reset texture. Doesn't seem to have broken anything and is a potential optimization (as texture_reset is also called on every batch end). commit

Added or implemented
-Added window_set_maximized() and window_get_maximized() on Windows. Not sure why they weren't implemented. Also added them for xlib, but THEY ARE NOT TESTED!!! commit
-Added gui_continue_propagation() to BGUI. This allows the event propagation to continue even if it would be stopped. Like if you have a button over a window. Normally the window would not get an event, because the button is pressed. If you want the window to get the event you put "gui_continue_propagation" in the button callback. I use this
for things like double clicks on window titles. If you click once it works like regular window click, but if clicked twice the button does something different. commit
-Added gui_windows_group_update() which allows updating windows in groups. It also has a second "continueProp" argument, which allows the groups to be treated individually or together like gui_windows_update() would. They also return if the event propagation was stopped. That basically means the mouse was over a window or any widget bound to the window. Useful if you want to combine GUI with mouse events that are not tied to GUI, like RTS units shouldn't go to a waypoint if the mouse was actually clicked on a window.
Previously there wasn't an easy way to check this. commit  commit
-Added window_update_mouse() which allows calling an update for mouse_x and mouse_y variables. Useful if you depend on them (like the BGUI extension does), and you need to update them in the middle of a step or draw event. For example, you have a view in which you use mouse and GUI (which isn't a view) also uses mouse. Here you must update the mouse two times instead of once like ENIGMA does by default. commit
-Added window_update() which just calls ALL the events for all instances. Note that his can cause infinite loop if called in a place like step event. But useful in alarms, one shot events or places like the resize() event. commit
-Added draw_roundrect, draw_roundrect_color, draw_roundrect_ext and draw_roundrect_ext_color. Thanks Garo for reporting. commit
-Added font_height() that returns height of the maximum character. Previously string_width("M") was required or something similar, but this is O(1) as we already have that number.  commit
-Added d3d_model_add_float, d3d_model_add_float2, d3d_model_add_float3, d3d_model_add_float4 and d3d_model_add_ubyte4 for custom attributes. They are like vertex_buffer in GM, but here it is implemented as a part of model class, so there wouldn't be any code duplication. commit
-Added d3d_model_add_color, d3d_model_add_texcoord and d3d_model_add_normal. The only different from _float2 or _float3 is that they can be used without formats. That means the default shader supports them. commit
-Added vertex_format_ functions, but they are totally different from GM:S. Added vertex_format_create, vertex_format_destroy, vertex_format_exists, vertex_format_add. Removed the weird "_end returns the ID" thing because it is totally different from how the other systems work. So now we use ID's instead (with creation and destruction which GM:S doesn't allow). I didn't add vertex_format_add_color, vertex_format_add_position and so on because they don't make sense in a custom shader. vertex_format_add() is different than GM:S because the last argument is attribute to add. This can be found using glsl_get_attribute_location allowing you to use custom attributes (GM:S didn't actually allow that as far as I know). commit
-Added glsl_program_get() which returns the currently bound shader. Most useful to actually get the default shader. commit
-Added glsl_attribute_enable_all() which is used in the modelstruct and not much useful anywhere else. commit
-Added equal() function for floating point math comparisons.  commit
-Added ray_sphere_intersect() function which allows raytracing with rays and spheres. Useful for collision, vertex painting and more. commit
-Added d3d_transform_get_array, d3d_transform_set_array, d3d_transform_add_array, d3d_transform_get_array, 3d_projection_set_array, d3d_projection_add_array and d3d_projection_get_array. Much easier to use in external C++ than EDL because of broken pointers in parser. But still useful. commit
-Added d3d_transform_add_rotation/d3d_transform_set_rotation which just takes the three angles. commit
-Added execute_shell and execute_program to Linux (xlib). commit
-Surfaces now have an option to have a stencil buffer, as well as readable depth buffer. For a depth buffer to readable it had to be an FBO, as Renderbuffers cannot be sampled. But then the buffer is slower, so I ended up implementing it as a choice.  commit
-New function surface_get_depth_texture() to get surface depth texture if the surface has depth buffer and if its not write only. This can be used for things like shadowmapping.  commit
-Added d3d_set_color_mask(r,g,b,a) function to enable/disable writing to a color channel. commit
-Added d3d_stencil_enable, d3d_stencil_clear_value, d3d_stencil_mask,d3d_stencil_clear, d3d_stencil_function and d3d_stencil_operator to have more fine grain control of stencil buffers. commit
-d3d_model_ format can now be changed with  d3d_model_format. This allows the same model to be drawn with a different shader which has a different format. commit
-Added d3d_transform_set_look_at and d3d_transform_add_look_at in GL1 and GL3. This rotates an object to face a point. commit
-Added d3d_transformation_get_mv and d3d_transformation_get_mvp that returns model_view_matrix and model_view_projection_matrix respectively. Useful in cases like shadowmapping where we use projection function to set a camera and then we need MVP in a shader later. commit
-Added _duplicate_ functions to BGUI widgets. This returns an exact duplicate of the widget that can then be modified. Very useful for styles and skins. commit
-Added BGUI textbox widget consisting of 57 functions. It is a textbox with unlimited chars and lines (but can be limited in with functions). It supports mouse selection, non-uniform fonts, copy and pasting (ctrl+c and ctrl+v) and much more.

The End
This is just an update so people know that ENIGMA hasn't died. Sadly I haven't been able to compile some of the 3rd party stuff we use so I cannot make an installer now. I use ENIGMA in several of my projects (also at work) so it gets updated quite frequently, but I sadly don't have much time doing community things and uploading new installers.

Some of the still long standing problems:
1) LGM is unstable on Windows. Crashes 24/7.
2) We need extracted EGM format. I use Git as version control for my projects and zipped EGM is detected as binary (even though I have tried all the diff tricks with extracted zips).
3) EDL needs to be either fixed or replaced. I want to use pure C++11 for a long time now and I believe it would be a perfect language, as you can do a lot of "auto" things and ranged loops that would make C++11 easier than EDL.
Offline (Male) Garo
Reply #1 Posted on: January 29, 2016, 01:32:28 PM

Location: Szczecin, Poland
Joined: Sep 2014
Posts: 22

View Profile
Amazing work. Thank you for all your effort.

Question though, how much different would things be if EDL was replaced with C++11?
Offline (Male) Goombert
Reply #2 Posted on: January 29, 2016, 02:13:41 PM

Location: Cappuccino, CA
Joined: Jan 2013
Posts: 2983

View Profile
Looks really nice Harri, I like the implementation of the attributes, looks very clean.

At this point in time, I say just switch over to C++11, enough messing around. We ship the compiler anyway, there's absolutely nothing stopping us from switching.
Offline (Male) Josh @ Dreamland
Reply #3 Posted on: January 29, 2016, 06:17:03 PM

Prince of all Goldfish
Location: Ohio, United States
Joined: Feb 2008
Posts: 2925

View Profile Email
I love CSG. Why isn't this topic in Announcements?

That 3D work is very impressive, and truly fascinating.

As for replacing EDL: you'd have a much better time of that if you could declare local variables somewhere specific in LGM.
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble
"I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
Offline (Unknown gender) TheExDeus
Reply #4 Posted on: January 29, 2016, 07:53:50 PM

Joined: Apr 2008
Posts: 1886

View Profile
Question though, how much different would things be if EDL was replaced with C++11?
EDL, just like GML is quite permissive, so it depends on how you code right now. If you use GML like Pascal, then C++11 will seem foreign. Like this won't work:
Code: (EDL) [Select]
t = 5
if t = 4 then
t = 0
t = 1
Not sure if this code actually would run as I don't code that way, but I'm sure GM allowed it. So it doesn't require semicols at end of lines, proper scoping with braces and so on. On the other hand if you code very C/C++ style anyway, then it won't be a big change. Like I code in GML and EDL like this:
Code: (EDL) [Select]
t = 5;
if (t == 4){
   t = 0;
} else {
   t = 1;
And this is valid C++ (except the variable needs to be declared previously). In C++11 you can do things like this:
Code: (EDL) [Select]
array<int,50> myInts; //Create 50 integers
myInts.fill(3); //Fill with 3's
for (auto &integer : myInts){ //Go trough all integers in array
  integer += 1;
} //Now they are all 4
auto sum = 0;
auto myFunction = [&sum](auto myInt){ sum += myInt; }; //Create Lambda function - i.e. a "script" in EDL terms, but right here in the code
for_each( myInts.begin(), myInts.end(), myFunction ); //Apply that function to each of the elements
show_message(string(sum)); //This would print 200, i.e., sum of all 50 elements of integers 4
This might seem more complicated than pure EDL or GML, but it is not low-level pointer ridden C either. Like we don't need to declare variables explicitly and we can instead use "auto" in many places. Then we don't need to keep track of indexes in "for loop" and instead can iterate in a shorter way (and faster too). And in the end it is more useful for the person as well, as he actually learns C++. And if you didn't use any of the C++11 stuff then it would just be regular EDL. Like this would still work just fine, as it is also proper C++:
Code: (EDL) [Select]
var spr = sprite_add("sprite.png",1,0,0,0,0);

At this point in time, I say just switch over to C++11, enough messing around. We ship the compiler anyway, there's absolutely nothing stopping us from switching.
The thing stopping me is that we still have to maintain instancing system, so at least minimal parser is required to deal with "with(){}" and dot access.

I love CSG. Why isn't this topic in Announcements?
I am still trying to make a proper installer and examples. Then I will try make a large post.

As for replacing EDL: you'd have a much better time of that if you could declare local variables somewhere specific in LGM.
You mean instead of the parser trying to figure it out? Previously I posted about my "Pure ENIGMA" idea that would involve the most minimal parser possible. It would allow C++11 to be used without any real restrictions (as the parser wouldn't touch it), but it would find local variables (that would have to be defined as local and there would be no "uninitialized" variables allowed) and would deal with scoping - "with()" and dot access. If we ditch pure integer id's it would be easier to achieve it and I would love to see that too for all resources, as A LOT of functions would be overloadable then.
« Last Edit: January 29, 2016, 07:57:20 PM by TheExDeus » Logged
Pages: [1]