ENIGMA Development Environment

I am currently fixing a couple of issues, but the integration testing branch should be ready for merging into master. I have tested that a test report can be generated on both Linux and Windows. I have also created a new version of the plugin based on the current plugin source, which contains the changes needed to run the tests. It can be found here: https://www.dropbox.com/s/hoxxljr9zfnwaxv/enigma.jar. Note that it has been built for Java 1.6, and that it can only be used together with the latest LGM1.8 by Robert.

Robert: I made it work by using the LGM1.8 jar instead of the repository's source for the LGM library.

Josh: Yes, a server is needed for running Jenkins (http://jenkins-ci.org/) on. I don't currently have a server set up myself, so if you are interested in handling that aspect, it would be very nice. My (somewhat limited) experience is that once Jenkins itself has been set up, most things can be handled through its GUI, including installing plugins for it and setting the plugins up. It has a Git plugin and a GitHub plugin (instructions for getting it to pull automatically from GitHub when a new commit has been made can be found here: https://wiki.jenkins-ci.org/display/JENKINS/GitHub+Plugin), and it comes built-in with a system for handling test results.

In regards to setting it up with the integration tests, the tests should be run as the last step of building (using the command [snip]python3 integration_tests/run_and_record_tests.py[/snip]). To publish the test results, a post-build action must be used to retrieve the test results. While it expects the results to be in a JUnit test result report format, the current testing system supports that format, and reports in that format are automatically generated in "integration_tests/output_temporary/testreport_jenkins.xml" whenever tests are run.

In regards to the image, it seems like a modified version from someone describing how to set up an X-box controller on Linux to play Cave Story (http://linuxtidbits.wordpress.com/2009/11/11/doukutsu-aka-cave-story-with-an-xbox-controller/). If it turns out to be a problem, we can always take it down and replace it with another image. YoYoGames are using a different image in any case.

In regards to the controller support, nicely done .

TheExDeus: Running the tests manually is just a temporary option until we get them automated. Since it is somewhat bothersome to run the tests every time, I don't personally expect anyone to run them manually as well (though they would be welcome to).
In regards to the end-goal, the idea is indeed to have them fully automated. One way to do that would be to have multiple development branches like Josh previously proposed; developers commit new changes to the testing branch, the tests are run automatically for each tests and the results reported online, and if it looks good, the changes are then ported to the master branch (where building and testing is likewise done).
The nice thing about this is that automated building + testing is relatively easy to set up with a continuous integration system. I looked into using Jenkins for it, and it seems that it is really easy to get Jenkins to automatically detect whenever any changes have happened to a branch in a GitHub repository, and then build + run all tests and report the results (just install the GitHub plugin for it and configure it in the dashboard). Some of the nice things about the automated testing in Jenkins is that it shows an overview over tests, how many commits it is since a failed test first failed, and graphs over the count of failed/passed tests from build to build. Here is the Jenkins dashboard for one of Apache's Hadoop build tasks: https://builds.apache.org/job/Hadoop-branch-1/lastCompletedBuild/testReport/.

Robert: That sucks, I hope it gets sorted out soon. I might be able to build it some other way, don't worry about it too much.

2) In my opinion, ENIGMA is not quite there in regards to deployment, reliability and stability, especially for computer novices (which I get the impression your users are, given that you will introduce them to Office and the like). While we are slowly improving those issues*, I think it will be at least half a year or a year before ENIGMA gets to a level where its stability is acceptable for use by computer novices. That said, if the odd bug and the challenge of getting ENIGMA set up is not an issue, ENIGMA can definitely be used to create games with.

(*we are continually doing general improvements and bug fixes, and we are currently working on getting automated regression testing up and running, which should help quality and stability considerably over time).

On the thread talking about the GMC forums, it seems that they had an open beta for a long period before they announced the price for the compiler. Another issue they talked about is the lack of upgrade options for those who bought versions of GM:Studio on Steam: The upgrade discount options that customers who bought directly from YoYoGames have, aren't available to those customers who bought over Steam. This is partially explained by the decreased price of the products due to sales on Steam, but some of the users on Steam bought the products for the full price, and they are not offered the upgrade discounts.

Combined with the other issues (the pirate sprite overwriting, selling of unsupported software like GM4Mac without informing potential buyers that it is unsupported, etc.), it seems clear to me that YoYoGames' business model does not rely on building a good reputation amongst its customers. I don't know if that will work out for them in the long run. I get the impression that they are relying on two things: the momentum that Game Maker helped build over the years, as well as using the lack of real alternatives to squeeze money out of their customer base - GM is still a very good product, and people will likely bite the bullet if they don't have other options.

Of course, this business model just helps ENIGMA. I think the turning point will come when two things happens: ENIGMA becomes easy, reliable and stable enough for most users, and we get close to feature parity on important areas with GM. We have all worked hard on the latter point, and I think continuous integration and automated regression testing will be the missing piece of the puzzle to fix the former point.

In regards to the issues with approximate results, I think using wide bounds checking for most cases instead of checking for precise values will be a decent solution. While the test will not detect small deviations, using wide bounds should avoid any false positives, and we still get a lot of the advantages of testing it, including if it compiles. For those cases where we can figure out how to do precise checking without any false positives, we of course ought to do that. I agree that things like data structures and strings should work very well.

It would also be very nice if we could get the installer for Windows to be automatically created as part of the continuous integration. For instance, we could set up a job that creates a new installer once a week/when we request it, with the latest commit which built successfully.

On a side-note, the testing system depends on the latest version of the plugin, which has not been built and uploaded yet. As far as I can tell, I cannot do that by myself at the moment, unless I modify the install.py to load package information from another place. So, I would like to kindly request that it is built and uploaded .

Thanks, I am happy to help .

Josh, that reminds me of another thing I have thought about: I think the automated regression testing would work really well together with the idea you previously proposed about having multiple branches here. By doing that, development could go like this: Developing stuff, commit the stuff to the testing branch, wait for the tests on the testing branch to finish running on multiple platforms by the continuous integration system, check that there are no regressions, and merge the changes into master. That way, it would require almost no work to run all tests on multiple platforms before committing to master. What do you think?