ENIGMA Development Environment

For a side project I am developing in Java I needed a good JavaScript parser but the publicly documented Nashorn interface is all about compiling when I only needed an intermediate representation. It is currently possible as of JDK8u40 to use the parser to get the AST as a JSON encoded string either from a JS file being executed by the ScriptEngine or from within Java using the non-public Nashorn API.

An abstract syntax tree is produced by a parser after the lexer phase breaks code into a stream of tokens. The below image should convey to you how a simple assignment statement is broken into a stream of tokens then an AST is generated as JSON on the right. This is why symbols like * + - are called binary operators because they take two operands, ! is the logical negation and an unary operator becaues it takes only one operand. The operands can also be expressions because expressions are defined recursively in terms of expressions which can be literals, terms, or other expressions. This is how we end up with tree's and these tree's coupled with semantic information such as keywords and identifiers help us do code generation which can let the whole process take in one language, say GML, and spit out a completely different one like C++ and if you don't already know this is exactly what ENIGMA's compiler does.



Wikipedia has additional information on abstract syntax trees if you would like to know more.
https://en.wikipedia.org/wiki/Abstract_syntax_tree
The following StackOverflow post provides clarification between an AST and a parse tree.
http://stackoverflow.com/questions/5026517/whats-the-difference-between-parse-tree-and-ast

My first example here is the standard example on the web of how you can get the JSON tree for any arbitrary JavaScript parsed by a JS script that is being actively executed in Nashorn. You can take this AST and print it or traverse it in JS or pass/return it up to Java through a binding. This example was taken from the following link.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/file/bfea11f8c8f2/samples/astviewer.js
#// Usage: jjs -scripting -fx astviewer.js -- <scriptfile>
/*
 * Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Oracle nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

if (!$OPTIONS._fx) {
    print("Usage: jjs -scripting -fx astviewer.js -- <.js file>");
    exit(1);
}

// Using JavaFX from Nashorn. See also:
// http://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/javafx.html
// This example shows AST of a script file as a JavaFX
// tree view in a window. If no file is specified, AST of
// this script file is shown. This script demonstrates
// 'load' function, JavaFX support by -fx, readFully function
// in scripting mode.

// JavaFX classes used
var StackPane = Java.type("javafx.scene.layout.StackPane");
var Scene     = Java.type("javafx.scene.Scene");
var TreeItem  = Java.type("javafx.scene.control.TreeItem");
var TreeView  = Java.type("javafx.scene.control.TreeView");

// Create a javafx TreeItem to view a AST node
function treeItemForASTNode(ast, name) {
    var item = new TreeItem(name);
    for (var prop in ast) {
       var node = ast[prop];
       if (typeof node == 'object') {
           if (node == null) {
               // skip nulls
               continue;
           }

           if (Array.isArray(node) && node.length == 0) {
               // skip empty arrays
               continue;
           }

           var subitem = treeItemForASTNode(node, prop);
       } else {
           var subitem = new TreeItem(prop + ": " + node);
       }
       item.children.add(subitem);
    }
    return item;
}


// do we have a script file passed? if not, use current script
var sourceName = arguments.length == 0? __FILE__ : arguments[0];


// load parser.js from nashorn resources
load("nashorn:parser.js");


// read the full content of the file and parse it
// to get AST of the script specified
var ast = parse(readFully(sourceName));


// JavaFX start method
function start(stage) {
    stage.title = "AST Viewer";
    var rootItem = treeItemForASTNode(ast, sourceName);
    var tree = new TreeView(rootItem);
    var root = new StackPane();
    root.children.add(tree);
    stage.scene = new Scene(root, 300, 450);
    stage.show();
}

This example shows you how to get the AST as JSON from Java. This was my own discovery from studying the Nashorn source code.
String code = "function a() { var b = 5; } function c() { }";

Options options = new Options("nashorn");
options.set("anon.functions", true);
options.set("parse.only", true);
options.set("scripting", true);

ErrorManager errors = new ErrorManager();
Context contextm = new Context(options, errors, Thread.currentThread().getContextClassLoader());
Context.setGlobal(contextm.createGlobal());
String json = ScriptUtils.parse(code, "<unknown>", false);
System.out.println(json);

Both of the above two examples should give the following JSON encoded AST. This JSON encoding provided by Nashorn is compliant with the community standard JavaScript JSON AST model popularized by Mozilla.
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API

{"type":"Program","body":[{"type":"FunctionDeclaration","id":{"type":"Identifier","name":"a"},"params":[],"defaults":[],"rest":null,"body":{"type":"BlockStatement","body":[{"type":"VariableDeclaration","declarations":[{"type":"VariableDeclarator","id":{"type":"Identifier","name":"b"},"init":{"type":"Literal","value":5}}]}]},"generator":false,"expression":false},{"type":"FunctionDeclaration","id":{"type":"Identifier","name":"c"},"params":[],"defaults":[],"rest":null,"body":{"type":"BlockStatement","body":[]},"generator":false,"expression":false}]}

This example code shows you how to get the AST as a Java object representation however the interface is poorly documented and I could not for the life of me figure out how to traverse the children of the function node. This solution is adapted from a StackOverflow post.
http://stackoverflow.com/questions/6511556/javascript-parser-for-java
String code = "function a() { var b = 5; } function c() { }";

// parser options including anonymous functions
final Options options = new Options("nashorn");
options.set("anon.functions", true);
options.set("parse.only", true);
options.set("scripting", true);

ErrorManager errors = new ErrorManager();
Context contextm = new Context(options, errors, Thread.currentThread().getContextClassLoader());
// get a source handle for arbitrary javascript code passed as a string
final Source source = Source.sourceFor("<unknown>", code);

// get the global function node to traverse the parsed AST
FunctionNode node = new Parser(contextm.getEnv(), source, errors).parse();

System.out.println(node.getKind().name());

for (Statement stmt : node.getBody().getStatements()) {
System.out.println(stmt.toString());
}

You should get the following output on the Java Console from the above code.

SCRIPT
function {U%}a = [<unknown>] function {U%}a()
function {U%}c = [<unknown>] function {U%}c()

It is important to note however that this interface may change because it's not well documented and is new to the JSE. Additionally the OpenJDK project is developing a public interface for Java 9 that allows AST traversal in a more standard and user friendly way.
http://openjdk.java.net/jeps/236

Limited documentation for the existing public Nashorn classes in Java 8 can be found below.
https://docs.oracle.com/javase/8/docs/jdk/api/nashorn/allclasses-noframe.html

The following link provides a list of all of the parser and compiler options that I set above. However it is important to note that the syntax is different when setting the options inside Java where - is replaced with a period.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/file/tip/docs/DEVELOPER_README

The Nashorn source code can be found on GitHub and also on BitBucket. I prefer the BitBucket version as the GitHub version seems to be missing some classes.
https://github.com/uditrugman/openjdk8-nashorn
https://bitbucket.org/adoptopenjdk/jdk8-nashorn/src/096dc407d310?at=default

After I fixed the primitive arrays with Josh's help I wanted to fix initializer lists, they are only broken because of the automatic semicolons, when this is disabled they work. Harri's matrix array initialization is primarily what inspired me.
http://enigma-dev.org/forums/index.php?topic=2397.msg24374#new

This was my attempt here:
https://github.com/enigma-dev/enigma-dev/pull/917

But it ultimately fails because we simply don't have sufficient information to do this yet and a number of games are failing to parse after the changes. A better work around for the time being would be to add an option to disable automatic semicolons all together, this means you would have to always put your ';' terminating semicolons where they belong. But it would stop some things like initializer lists from breaking.

This would make the following possible in ENIGMA if you choose to disable it.
int arr[2][3][4] = { { {1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4} },
                     { {1, 2, 3, 4}, {1, 2, 3, 4}, {1, 2, 3, 4} } };

for (int r = 0; r < 2; r++) {
    for (int c = 0; c < 3; c++) {
        for (int n = 0; n < 4; n++) {
            show_message(string(arr[r][c][n]));
        }
    }
}

I would appreciate feedback on this and whether you guys think a setting for this would be nice. It should be considered an advanced setting as advanced users are more likely to properly terminate statements.

We've had some really awesome compiler fixes lately and I wanted to get these fixes out to everyone for testing. I have not updated LateralGM or the plugin since the last Portable ZIP, this was just a quick releases for these compiler fixes.

You can update by downloading the new ZIP.
http://enigma-dev.org/docs/Wiki/Install:Windows

You can also get these changes by entering the following in a terminal or using git-bash.exe
cd enigma-dev
git fetch
git pull

1) Nested macros have been fixed, this is basically nested string() calls, they will now work.
https://github.com/enigma-dev/enigma-dev/pull/906
2) Fixed primitive arrays adding full multi-dimensional primitive array functionality.
https://github.com/enigma-dev/enigma-dev/pull/908
3) Finished implementing the modulus assignment (%=) operator and overloaded it for var
https://github.com/enigma-dev/enigma-dev/pull/902
4) Fixed alarm inheritance, though alarms may still fire out of order, they are supposed to be fired 0,1,2,3,etc. but currently ENIGMA fires the parents then the childs, this will be fixed in the future, but it is unlikely to cause a bug and it still works better than before. I can't really think of an example where someone would rely on alarm 1 to fire after alarm 0.
https://github.com/enigma-dev/enigma-dev/pull/894
https://github.com/enigma-dev/enigma-dev/issues/895
5) Instance activation/deactivation when used with inheritance fixed by sorlok
https://github.com/enigma-dev/enigma-dev/pull/898
6) Enabled Unicode RC files, so you can enter the copyright symbol into LGM's platform settings now and it will properly encode it in your game executable
https://github.com/enigma-dev/enigma-dev/pull/893
7) Fixes syntax checking output, fixes the line numbers and also actually formats the output for script, instance and room creation scope where it did not before.
https://github.com/enigma-dev/enigma-dev/pull/913

This is just a heads up I ran across the tutorial for GM6 over at the GMC and it works perfectly in ENIGMA with no changes.



You can follow the tutorial from start to finish with ENIGMA.
http://gmc.yoyogames.com/index.php?showtopic=645185

This has been in Studio for a while, you can use /// on the first line of code action to change its descriptive label. I just want to know what everyone thinks of this feature, maybe we could add it to LGM or not, would just like to know what people think.


From the GMC:
http://gmc.yoyogames.com/index.php?s=2dbd49a45df4db76813855b67a017983&showtopic=646815

Well well well. Windows 10 will have a package manager that will make Windows development just amazing, we may finally be able to create a proper Portable ZIP and allow MinGW installations. This is honestly the greatest news ever!

http://www.extremetech.com/computing/192950-windows-10-will-come-with-a-command-line-package-manager-much-to-the-lament-of-linux-users



Even those of you who want to go ahead and start using PowerShell can, I was only vaguely aware of its existence, it has rectangle select as well. The only downside is that it is built on the .NET framework, it opens at about 45 MB ram usage for me where as regular cmd opens at 0.3 MB.
https://en.wikipedia.org/wiki/Windows_PowerShell

No joke, it's been built into Windows since XP, and Microsoft has an installer for you to access the hidden feature, so it's not a rough hack.
http://www.tekrevue.com/tip/virtual-desktops-windows/

I've always been a big fan of the BlitzBasic engine and products, though never really having used them that much, I liked the environment much better over GM. It made it very easy to manage objects and everything from code and for novices to learn without having to create excessive GUI infrastructures. Well anyway since Mark Sibly is focused on Monkey X now, they've put Blitz Plus and Blitz 3D up for free. Blitz Max and the programming manual are still being commercially sold.

http://www.blitzbasic.com/Products/_index_.php

You can read the official announcements on the home page and the forum threads.
http://www.blitzbasic.com/Home/_index_.php
http://www.blitzbasic.com/Community/posts.php?topic=102907
http://www.blitzbasic.com/Community/posts.php?topic=102473



Anybody not already aware, Monkey X is an open source cross-platform game engine with the BASIC programming language as well, sort of based on BlitzBasic. For $100 it can export to numerous modules including Android, Playstation, Xbox, and other platforms. The IDE is built, quite evidently, with the Qt Framework, and it's a very nice IDE.
http://www.monkey-x.com/