ENIGMA Development Environment

For a side project I am developing in Java I needed a good JavaScript parser but the publicly documented Nashorn interface is all about compiling when I only needed an intermediate representation. It is currently possible as of JDK8u40 to use the parser to get the AST as a JSON encoded string either from a JS file being executed by the ScriptEngine or from within Java using the non-public Nashorn API.

An abstract syntax tree is produced by a parser after the lexer phase breaks code into a stream of tokens. The below image should convey to you how a simple assignment statement is broken into a stream of tokens then an AST is generated as JSON on the right. This is why symbols like * + - are called binary operators because they take two operands, ! is the logical negation and an unary operator becaues it takes only one operand. The operands can also be expressions because expressions are defined recursively in terms of expressions which can be literals, terms, or other expressions. This is how we end up with tree's and these tree's coupled with semantic information such as keywords and identifiers help us do code generation which can let the whole process take in one language, say GML, and spit out a completely different one like C++ and if you don't already know this is exactly what ENIGMA's compiler does.



Wikipedia has additional information on abstract syntax trees if you would like to know more.
https://en.wikipedia.org/wiki/Abstract_syntax_tree
The following StackOverflow post provides clarification between an AST and a parse tree.
http://stackoverflow.com/questions/5026517/whats-the-difference-between-parse-tree-and-ast

My first example here is the standard example on the web of how you can get the JSON tree for any arbitrary JavaScript parsed by a JS script that is being actively executed in Nashorn. You can take this AST and print it or traverse it in JS or pass/return it up to Java through a binding. This example was taken from the following link.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/file/bfea11f8c8f2/samples/astviewer.js
#// Usage: jjs -scripting -fx astviewer.js -- <scriptfile>
/*
 * Copyright (c) 2014, Oracle and/or its affiliates. All rights reserved.
 *
 * Redistribution and use in source and binary forms, with or without
 * modification, are permitted provided that the following conditions
 * are met:
 *
 *   - Redistributions of source code must retain the above copyright
 *     notice, this list of conditions and the following disclaimer.
 *
 *   - Redistributions in binary form must reproduce the above copyright
 *     notice, this list of conditions and the following disclaimer in the
 *     documentation and/or other materials provided with the distribution.
 *
 *   - Neither the name of Oracle nor the names of its
 *     contributors may be used to endorse or promote products derived
 *     from this software without specific prior written permission.
 *
 * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS
 * IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO
 * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
 * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR
 * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,
 * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,
 * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR
 * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF
 * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING
 * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
 * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
 */

if (!$OPTIONS._fx) {
    print("Usage: jjs -scripting -fx astviewer.js -- <.js file>");
    exit(1);
}

// Using JavaFX from Nashorn. See also:
// http://docs.oracle.com/javase/8/docs/technotes/guides/scripting/nashorn/javafx.html
// This example shows AST of a script file as a JavaFX
// tree view in a window. If no file is specified, AST of
// this script file is shown. This script demonstrates
// 'load' function, JavaFX support by -fx, readFully function
// in scripting mode.

// JavaFX classes used
var StackPane = Java.type("javafx.scene.layout.StackPane");
var Scene     = Java.type("javafx.scene.Scene");
var TreeItem  = Java.type("javafx.scene.control.TreeItem");
var TreeView  = Java.type("javafx.scene.control.TreeView");

// Create a javafx TreeItem to view a AST node
function treeItemForASTNode(ast, name) {
    var item = new TreeItem(name);
    for (var prop in ast) {
       var node = ast[prop];
       if (typeof node == 'object') {
           if (node == null) {
               // skip nulls
               continue;
           }

           if (Array.isArray(node) && node.length == 0) {
               // skip empty arrays
               continue;
           }

           var subitem = treeItemForASTNode(node, prop);
       } else {
           var subitem = new TreeItem(prop + ": " + node);
       }
       item.children.add(subitem);
    }
    return item;
}


// do we have a script file passed? if not, use current script
var sourceName = arguments.length == 0? __FILE__ : arguments[0];


// load parser.js from nashorn resources
load("nashorn:parser.js");


// read the full content of the file and parse it
// to get AST of the script specified
var ast = parse(readFully(sourceName));


// JavaFX start method
function start(stage) {
    stage.title = "AST Viewer";
    var rootItem = treeItemForASTNode(ast, sourceName);
    var tree = new TreeView(rootItem);
    var root = new StackPane();
    root.children.add(tree);
    stage.scene = new Scene(root, 300, 450);
    stage.show();
}

This example shows you how to get the AST as JSON from Java. This was my own discovery from studying the Nashorn source code.
String code = "function a() { var b = 5; } function c() { }";

Options options = new Options("nashorn");
options.set("anon.functions", true);
options.set("parse.only", true);
options.set("scripting", true);

ErrorManager errors = new ErrorManager();
Context contextm = new Context(options, errors, Thread.currentThread().getContextClassLoader());
Context.setGlobal(contextm.createGlobal());
String json = ScriptUtils.parse(code, "<unknown>", false);
System.out.println(json);

Both of the above two examples should give the following JSON encoded AST. This JSON encoding provided by Nashorn is compliant with the community standard JavaScript JSON AST model popularized by Mozilla.
https://developer.mozilla.org/en-US/docs/Mozilla/Projects/SpiderMonkey/Parser_API

{"type":"Program","body":[{"type":"FunctionDeclaration","id":{"type":"Identifier","name":"a"},"params":[],"defaults":[],"rest":null,"body":{"type":"BlockStatement","body":[{"type":"VariableDeclaration","declarations":[{"type":"VariableDeclarator","id":{"type":"Identifier","name":"b"},"init":{"type":"Literal","value":5}}]}]},"generator":false,"expression":false},{"type":"FunctionDeclaration","id":{"type":"Identifier","name":"c"},"params":[],"defaults":[],"rest":null,"body":{"type":"BlockStatement","body":[]},"generator":false,"expression":false}]}

This example code shows you how to get the AST as a Java object representation however the interface is poorly documented and I could not for the life of me figure out how to traverse the children of the function node. This solution is adapted from a StackOverflow post.
http://stackoverflow.com/questions/6511556/javascript-parser-for-java
String code = "function a() { var b = 5; } function c() { }";

// parser options including anonymous functions
final Options options = new Options("nashorn");
options.set("anon.functions", true);
options.set("parse.only", true);
options.set("scripting", true);

ErrorManager errors = new ErrorManager();
Context contextm = new Context(options, errors, Thread.currentThread().getContextClassLoader());
// get a source handle for arbitrary javascript code passed as a string
final Source source = Source.sourceFor("<unknown>", code);

// get the global function node to traverse the parsed AST
FunctionNode node = new Parser(contextm.getEnv(), source, errors).parse();

System.out.println(node.getKind().name());

for (Statement stmt : node.getBody().getStatements()) {
System.out.println(stmt.toString());
}

You should get the following output on the Java Console from the above code.

SCRIPT
function {U%}a = [<unknown>] function {U%}a()
function {U%}c = [<unknown>] function {U%}c()

It is important to note however that this interface may change because it's not well documented and is new to the JSE. Additionally the OpenJDK project is developing a public interface for Java 9 that allows AST traversal in a more standard and user friendly way.
http://openjdk.java.net/jeps/236

Limited documentation for the existing public Nashorn classes in Java 8 can be found below.
https://docs.oracle.com/javase/8/docs/jdk/api/nashorn/allclasses-noframe.html

The following link provides a list of all of the parser and compiler options that I set above. However it is important to note that the syntax is different when setting the options inside Java where - is replaced with a period.
http://hg.openjdk.java.net/jdk8u/jdk8u-dev/nashorn/file/tip/docs/DEVELOPER_README

The Nashorn source code can be found on GitHub and also on BitBucket. I prefer the BitBucket version as the GitHub version seems to be missing some classes.
https://github.com/uditrugman/openjdk8-nashorn
https://bitbucket.org/adoptopenjdk/jdk8-nashorn/src/096dc407d310?at=default

Was this merged into the main repo?

I've moved this temporarily down to Function Peer Review as I am not quite sure where it belongs on the board.

Yeah basically what Josh said, though I am kind of diverging myself in a lot of different directions. I've so overloaded with school work but I have a really amazing prototype of some of my new ideas and a lot of great ones that I am not going to drop on this forum because I want to retain them. Some people I've been keeping in the loop about some of these things I've been working on it, but it is too early for me to say much. But I am basically behind what you are saying Harri and we're pretty much on all the same page. I am hoping that I may actually come up with something for the interim that will put us back on the right track until Josh solidifies his ideas and plans.

That's a possibility and yes it would be relatively straight forward Harri.

I think I will just recompile the libs for 64bit and get a 64bit Portable out first and then we can discuss restructuring the ZIP's.

On side a note I had to push another fix to make it build for Mingw32 again, so it builds for both now, it should work for both 32bit and 64bit GCC on Linux now as well. In addition the compiler now builds with C++11 support.
https://github.com/enigma-dev/enigma-dev/commit/5b99151173b1dad9acb059f9e99a75974801d42b
This was to fix the issues reported in the other topic.
http://enigma-dev.org/forums/index.php?topic=2422

Edit: Just a quick update here I managed to get the basic engine compiling with Mingw64.

Thanks to Rusky for making a 64bit zlib and me finding a 64bit FFI lib, the basic engine and graphics now build. What you see in the following screenshot is the first 64bit ENIGMA game built with Mingw64 with LateralGM running in a 64bit JVM and a 4GB memory capacity. It can build at least Project Mario with DirectSound so far.

Note: If you try setting -xmx to larger than 1GB in the regular Portable the OS will probably not allow LGM to launch, I've tried it myself.



There are only 3 things left to do, build a 64bit OpenAL, Box2D, and Bullet and we can release a 64bit Portable. OpenAL will be the bulk of the work because we have to rebuild vorbis and a bunch of other libs to support all the audio formats.

This isn't new this is exactly what you reported last time and it hasn't changed.

Most reliable source I can find says that it could be related to using AppData, though I didn't think our exe's use that at all, only the compiler.
http://greatis.com/cleanvirus/remove-malware/behaveslike-win32-pwszbot-fh-usubucip-exe.htm

It could also be some of the extensions because those load DLL's, one of the reasons the old GM games still get flagged is because of DirectPlay, which is obsolete. Studio also does not handle resources the way the old GM did, its executables are essentially self-extracting 7zips, you can actually use 7zip to extract the audio from any standalone exe made with Studio, just extract it like a 7zip, though I haven't tested on the standalone installer. This is what my attempted decompiler originally did besides scanning for png and bmp files.

I'll run the test myself before and after disabling some extensions. Either way I am not really concerned with what some silly virus scanner on the internet says, we know for a fact it's not a virus.

Edit: Oddly enough switching off all systems and disabling every extension except paths gets it flagged by a different scanner, but still 1/57
Qihoo-360    Malware.QVM20.Gen    20150123

Using only Direct3D9 the same way but instead of OpenGL with no extensions flags it twice.
CMC    Packed.Win32.Katusha.1!O    20150120
McAfee-GW-Edition    BehavesLike.Win32.PWSZbot.ch    20150122

For why you shouldn't care what this scanner thinks either, read the following:
http://www.cplusplus.com/forum/beginner/67634/

Do you have the PNG? I'll test myself, Project Mario uses extensive BMP and PNG loading. Also the code that you are trying to load it with.

No problem, I had to fix it because it was actually me that just broke it lol. And because now I accomplished what I wanted, 32bit and 64bit support for JDI and the compiler. As usual let us know if you have any more problems!

Don't worry I fixed it and this time made sure to build for both Mingw64 and 32.
https://github.com/enigma-dev/enigma-dev/commit/5b99151173b1dad9acb059f9e99a75974801d42b

Just undo all your changes, cd to enigma-dev git reset --hard and then git fetch and git pull. Then it should build, you don't need to run python install.py