ENIGMA Forums

Contributing to ENIGMA => Developing ENIGMA => Topic started by: daz on July 21, 2014, 01:17:02 PM

Title: Android
Post by: daz on July 21, 2014, 01:17:02 PM
I am trying to get Android working on Windows.

I copied the Android.ey from MacOS to Windows and changed it as follows:
Code: [Select]
%e-yaml
---
Name: Android NDK
Native: No
Maintainer: TGMG,IsmAvatar
Target-platform: Windows

Now upon loading LGM I can select "Android NDK" from the compilers drop down. The problem is if I change "Target-platform" to "Android", I can no longer select "Android NDK" from the Compiler combobox and "Android" does not show up in the Platform combobox either.

So I think it's a bug with LGM's enigma settings window.

The problem with leaving the Platform combobox as Windows is that it tries to compile as platform Win32 in the Windows/Windows compile path.
Title: Re: Android
Post by: time-killer-games on July 21, 2014, 02:11:10 PM
Are you actually going to expand what ENIGMA's android can do atm or are you just bug reporting so you can fiddle with the capability of the android port's current state?
Title: Re: Android
Post by: Darkstar2 on July 21, 2014, 03:16:03 PM
I'm confused about this, you are adding an export option for ENIGMA ? Doesn't that require GLES implemented ? to my knowledge it's not completed, or is it something else?

The day that ENIGMA will be able to spit out store compliant android apps will be amazing. That will really boost its popularity as more developers will use it instead of having to spend $199 for the Android export + another $299 for the YYC, which is sheer rip-off IMO.
Title: Re: Android
Post by: TheExDeus on July 21, 2014, 03:49:34 PM
He probably wants to get the dropbox in the settings panel working now. For Android to work he would have to redo the whole "ENIGMAsystem\SHELL\Platforms\Android" folder together with getting "ENIGMAsystem\SHELL\Graphics_Systems\OpenGLES" to work. I think GLES could actually be a smaller issue now, as GL3 graphics system should work as GLES with very little changes.

So the hard part is to get AndroidNDK working. It should be easier now that it was 3 years ago, because then you actually needed SDK (Java) and NDK (native, or C++) as they were separate. As far as I know now the SDK actually allows C++ to be compiled.
Title: Re: Android
Post by: Darkstar2 on July 21, 2014, 04:50:37 PM
There are more to be done than just the graphics system, there are specific functions that need to be added, virtual keys, in app advertising, tilt and other mobile/tablet specific features, and of course the apps would have to be signed/app store compliant.  Not placing any bets on ENIGMA ever being on par with other game making tools in terms of android exports.

Title: Re: Android
Post by: daz on July 21, 2014, 06:37:25 PM
He probably wants to get the dropbox in the settings panel working now.
Yes this is what I was originally asking. How else can I even try to get Enigma to attempt to start a compile process?

I plan to work on a lot of Android if I can. I don't know much of Enigma's architecture so there's a lot I will need help for (e.g. from Josh).

TGMG had a long time ago started some basic functionality. A lot of it is not relevant any more, and functions oddly. Currently it looks like he just had Enigma compile a .so. The user would then have to compile the Android project and launch it himself. This is where I will start. In the future it would be nice to configure Enigma to automatically copy an Android project to a staging area, include/exclude Android features from there as well as change the App manifest / icons, then copy in the compiled .so and create an APK and load it to a device. You can maybe see how much work this would be all from the start.

So the hard part is to get AndroidNDK working. It should be easier now that it was 3 years ago, because then you actually needed SDK (Java) and NDK (native, or C++) as they were separate. As far as I know now the SDK actually allows C++ to be compiled.
The SDK and NDK are still separate. My understanding is that before it was unofficial, and now Google provides an official NDK.

There are more to be done than just the graphics system, there are specific functions that need to be added, virtual keys, in app advertising, tilt and other mobile/tablet specific features, and of course the apps would have to be signed/app store compliant.  Not placing any bets on ENIGMA ever being on par with other game making tools in terms of android exports.
This would be a long way off, but you have to start somewhere.

I have my own little dummy Android application loading a custom written C++ library. What the first step would be is getting Enigma to compile with the NDK.
Title: Re: Android
Post by: TheExDeus on July 22, 2014, 05:46:37 AM
Quote
there are specific functions that need to be added, virtual keys, in app advertising, tilt and other mobile/tablet specific features, and of course the apps would have to be signed/app store compliant.
None of which is actually required to run a game on android. I was thinking what is ESSENTIAL for it to compile and run.

edit: I think the hello-gl2 example is the one that should be taken as a starting point in creating a window, rendering context and drawing. For me the only thing required is for it to create these things. Then I am willing to code drawing functions to work (which involves changing the GL3 shader to run in GLES2.0).
Title: Re: Android
Post by: daz on July 22, 2014, 08:49:20 PM
I was working out my problems with Josh in IRC. We got it to the point where Android could be seen in the drop down and ENIGMA would attempt to compile with the ndk.

The problem is that the NDK has 30 of its own makefiles

Quote from: Josh
(A) I have no idea how to retrofit our source list to their makefiles
(B) I have no idea which makefile to include, even if I knew how to feed in our sources

Unless Josh or someone fairly experienced with ENIGMA and makefile hell comes along and integrates this crap, we're at a standstill.

If anyone is interested in how to get Android to even show up as an option under Windows, I can help you with that. (But I don't think it's worth putting out a pull request with those changes when Android still does not compile anything)
Title: Re: Android
Post by: Rusky on July 22, 2014, 10:06:26 PM
From what I understand, all you need is a single makefile with some variables set, and then you run ndk-build. See http://www.kandroid.org/ndk/docs/ANDROID-MK.html
Title: Re: Android
Post by: daz on July 22, 2014, 11:43:57 PM
The single makefile is a problem. Enigma currently uses multiple makefiles to deal with dependencies and defines to specify different subsystems to be included in the final compile. So I haven't a clue how to get the whole of the engine to compile happily like that, but I'm not a C++ guy.
Title: Re: Android
Post by: Rusky on July 23, 2014, 02:01:57 AM
Hmm, a bit of digging turned this up: http://www.kandroid.org/ndk/docs/STANDALONE-TOOLCHAIN.html
Looks like you should be able to drop that in as a normal compiler. You will need to tweak the command line parameters a bit to get the standard library to link, but otherwise it should work out of the box.
Title: Re: Android
Post by: daz on July 23, 2014, 10:47:11 AM
The Android tools are currently GCC 4.6 while Enigma uses 4.8, and this also might be causing conflicts. The standalone toolchain looks promising but I don't know how to attempt to utilize it. I installed Cygwin so I could run the make-standalone-toolchain script (and even this might not be necessary). The problem I seem to have is that it's trying to compile with the mingw32 include files instead of the Android ones.

I will probably just end up waiting for Josh again tonight to see if we can make any further progress with a standalone-toolchain idea.
Title: Re: Android
Post by: Josh @ Dreamland on July 23, 2014, 10:11:42 PM
So, the point of ENIGMA's multiple makefiles is just to let systems specify the source files they include and the libraries they depend on. They operate under two assumptions:

1. You will be compiling some C++ sources into object files in order to build the C++ engine.
2. You will be linking the objects you have compiled into a standalone executable when finished.

Presently, (2) also assumes that the binary you use for linking is the same one you use for compiling. That can be changed. You can also inject additional dependencies and provide targets for them in Android-specific makefiles.

The big issue is, from what we discussed, that Android seems to compile sources into object files, but then there's some nebulous form of magic used to link those objects into a shared object to be loaded by the operating system (namely, through JNI). Unfortunately, I can't make you any recommendations until I can figure out (or someone tells me) what the hell that operation is.
Title: Re: Android
Post by: sorlok_reaves on July 23, 2014, 11:17:18 PM
Every Android project requires a small amount of Java boilerplate code. You'll typically create a normal Android project (using, say, the Eclipse plugin) with a "java" folder that contains Java code (Android Ad services, Network code, and the main window must go here) and then a "jni" folder (all your C++ code goes here). The "Android.mk" file in the jni folder controls the building of the JNI code (although there are some annoying caveats).

In terms of Platform code, the most stable library is, to my knowledge, SDL. Android ports of SDL have been around forever, and it's part of upstream as of SDL 2.0. I think you'll have more luck adding an SDL backend then, say, fiddling with the Xlib code (I have seen Irrlicht running on Android, so it's not impossible to spin your own solution, but SDL has all the easy stuff like keymappings and touch gestures done).

At that point, you'll basically build the entire project with the generic "ndk-build" script that is distributed with the NDK. This handles all the compiler irregularities and environment variables, and I've never seen an Android project built any other way (i.e., using GCC directly). You will never have to edit "ndk-build".

Here is an example (http://rpg.hamsterrepublic.com/ohrrpgce/Compiling_for_Android) of a project that I know to be working on Android that uses SDL and native code; unfortunately, a lot of the linked page is explaining how to deal with Free Basic (which that project is using, but they also have some C code).

I'm not really an expert on the NDK (it's a huge pain, in my opinion), but let me know if you run into any particularly difficult issues and I'll do my best to help.
Title: Re: Android
Post by: Goombert on July 24, 2014, 03:34:00 AM
Yeah the other big problem is spaces in the file paths which we still haven't fixed.
Title: Re: Android
Post by: TheExDeus on July 24, 2014, 04:43:24 AM
I think it should be now possible to compile without Java boilerplate. Just like the link Rusky gave mentions. You can use GCC directly to compile for android.
Title: Re: Android
Post by: sorlok_reaves on July 24, 2014, 12:45:21 PM
I think it should be now possible to compile without Java boilerplate. Just like the link Rusky gave mentions. You can use GCC directly to compile for android.

That's only to compile the JNI part. To actually distribute an APK, you'll definitely need some Java source. Even the article mentions it, in the section "Simple Example"

Quote
Here, we can see:

  - The 'src' directory containing the Java sources for the sample Android project.

  - The 'jni' directory containing the native source for the sample, i.e. 'jni/hello-jni.c'

I am nearly certain that networking can't be done in JNI code (only Java), and I think Android Ad Services have to run in Java too. Also, the main JFrame must be created in Java.

Edit: This might be a better example; it's a tutorial on using SDL on Android. They basically provide all the boilerplate for you so that you can minimize the Java code you have to write (you just define an sdl_main method, which is in JNI).
http://blog.stuff-o-matic.com/post/2013/10/20/ASGP-s-Android-Port-Part-IV%3A-building-the-APK
Title: Re: Android
Post by: daz on July 27, 2014, 12:25:13 PM
I'm just gonna throw this out there: would it be better to try to get the ndk-build script to compile all of Enigma? I don't know if it's possible to get the Android.mk file to (a) Compile other makefiles, or (b) compile source code that is not inside of a "jni" directory.

I'll look into this more myself once I'm feeling a bit better, but this might be a better solution than trying to deduce by what black magic the linker links.
Title: Re: Android
Post by: Josh @ Dreamland on July 28, 2014, 09:48:35 PM
Isn't there a guide anywhere on building manually? Or by any other mechanism than .mk? I mean, I believe I remember you reporting that you have it building all the object files, except some that had libstdc conflicts. Did you get that resolved? That's *most* of the battle. Linking should be the easy part, and whatever goes into it can be accomplished in the ENIGMA makefile using a custom rule, as I described earlier.
Title: Re: Android
Post by: daz on August 18, 2014, 01:16:56 AM
There really isn't a guide on building manually, and the documentation included in the NDK is very sparse.

The only thing I can think of right now (after some more toying with this) is that SYSROOT might have to be defined earlier in the compile process than the Android makefile. I have no other idea how mingw's libstdc++ files are being included.

My only other guess is that the make.exe I am calling is not using the correct GCC. It smells suspiciously like it's using mingw instead of the ndk... but I have no idea how to prove or fix that.
Title: Re: Android
Post by: Josh @ Dreamland on August 19, 2014, 10:40:57 PM
A quick way to test that is to call it with -v. :P
Title: Re: Android
Post by: sorlok_reaves on August 20, 2014, 02:27:07 AM
You can also temporarily rename your MinGW directory (or unlink it from the path) and see if make.exe complains.
Title: Re: Android
Post by: daz on August 21, 2014, 01:32:39 AM
You can also temporarily rename your MinGW directory (or unlink it from the path) and see if make.exe complains.
The Enigma plugin does not load if it cannot find make.exe.

A quick way to test that is to call it with -v. :P
Ah well apparently it is using the right one when attempting an Android compile (by the way interesting the way Enigma works in this regard!)
Quote
C:/android-ndk-r8e/prebuilt/windows-x86_64/bin/make -v

Any other ideas?
Title: Re: Android
Post by: Josh @ Dreamland on August 21, 2014, 08:48:04 PM
Actually, no, and none of the people I work with actually work on Android, so even after hours I can't solicit their help for you. The -v was for gcc, so you can tell if it's calling the right one. But since you already know it is, there's no point. What are you stuck on now, apart from linking? I thought that's where the big issue is, and that's what I don't know how to do. Google results for "link android app" are monumentally unhelpful; they seem to assume that you're writing in Java, so GWS ends up thinking "link" means "hyperlink."
Title: Re: Android
Post by: sorlok_reaves on August 22, 2014, 12:28:48 AM
Since everyone's out of ideas, here's a far-flung one (this is probably what I'd do if I was totally lost).

Compile one of the NDK example projects in Eclipse with verbose output on. Then, find the line (in the Console) that does the linking and try copying that for your own project.
Title: Re: Android
Post by: Josh @ Dreamland on August 23, 2014, 04:59:41 PM
Unfortunately, as I understand it, all the linking is done internally by some Android build tool that works with .mk files. The example mk files are, as usual, a labyrinthy maze of short yet impossible-to-read files of use to fucking no one. And the tool doesn't have a "verbose output" mode. So we have no idea what's going on.

Conceivably, someone could just grep over all mk files for anything resembling a link call, but I haven't tried.
Title: Re: Android
Post by: daz on August 25, 2014, 01:23:06 AM
An update for those interested, I determined Enigma was calling the wrong compiler when compiling for Android and with the help of Josh we tracked it down to my broken Android.ey file.

Now the problem we have is that apparently the Android g++ fails to invoke cc1plus when compiling Enigma. Even trying to manually compile 1 file fails. I will try manually compiling one of the NDK samples later in this manner to see if it results in the same error. If it does... there is some serious black magic going on behind the scenes.

EDIT:
Further update... the gcc binaries being in a non-standard place did not know where to look for cc1plus. I switched to using the standard bin directory with ugly-named gcc binaries, but compilation started!

The new problems I have are compilation problems. Mainly, OpenGLES is just screwed. It hadn't been updated in who knows how long, and a lot of Enigma's structure has changed in that time. I have been plowing away it it, but I fear I am in way over my head at this point. Fixing compilation errors results in other compilation errors! I will torture Josh if he is on IRC tomorrow (today?).

In the mean time, such fun errors include...
Code: [Select]
./Graphics_Systems/OpenGLES/../General/GSfont.h:36:42: error: 'variant' has not been declared
./Graphics_Systems/OpenGLES/../General/actions.h:47:9: error: 'object_planar' is not a member of 'enigma'
./Graphics_Systems/OpenGLES/../General/actions.h:249:69: error: 'background_replace' was not declared in this scope
./Collision_Systems/actions.h:28:103: error: 'struct enigma::inst_iter' has no member named 'inst'
./Graphics_Systems/OpenGLES/../General/GSscreen.h:29:8: error: ambiguates old declaration 'void enigma_user::screen_redraw()'
c:/androidtoolchain/lib/gcc/../../include/c++/4.6/bits/basic_string.tcc:221:5: note: std::basic_string<_CharT, _Traits, _Alloc>::basic_string(std::basic_string<_CharT, _Traits, _Alloc>::size_type, _CharT, const _Alloc&) [with _CharT = char, _Traits = std::char_traits<char>, _Alloc = std::allocator<char>, std::basic_string<_CharT, _Traits, _Alloc>::size_type = unsigned int]
c:/androidtoolchain/lib/gcc/../../include/c++/4.6/bits/basic_string.tcc:221:5: note:   candidate expects 3 arguments, 1 provided

And so much more! OpenGLES may have to be rewritten from the ground up! I also propose the starting of OpenGLES2 as a separate and shiny new Graphics subsystem! (Hell, maybe we ditch GLES1 entirely!)
Title: Re: Android
Post by: TheExDeus on August 26, 2014, 05:34:55 AM
GLES never worked, so it will have to be rewritten anyway. But the new GL3.3 code is 95% compatible with GLES2.0. So I would suggest you to do this:
1) Make /platforms work with Android (this is probably the cause of the errors you are seeing. They are not connected to the rendering system).
2) Make simple GLES implementation just to render an empty screen. This would mean that all other systems are working fine and that the only thing to do is rendering.
3) Then copy GL3.3 to GLES2.0 folder and fix all the rest of the warnings/errors that come up. Debug context might help.

I sadly still haven't fixed and committed my GL3 changes. I fixed a lot of stuff and removed a lot of incompatible functions, but it crashed when I merged and I didn't have time to figure out why.

But the fact that you actually got it to start compile is a great step forward. You should document the steps you took so others can replicate. Either in this topic or wiki.
Title: Re: Android
Post by: daz on August 26, 2014, 11:24:37 AM
I would like to push a commit first. After I can get a lib successfully compiled and run it on Android with a blank screen I'll send in a pull request and update the wiki. If anyone is serious about developing Android before then I'm willing to outline the steps here.

EDIT:
Wrote up the process on the Wiki ( http://enigma-dev.org/docs/Wiki/Android ). This process assumes that you are working with ENIGMA which has support for Android. As I have not yet pushed a commit, nothing will happen yet!

Also last night Josh and I got the whole thing compiling happily after much bashing of heads against desks. It seems the linking part fails though. Homestretch? I've also basically deleted the entire OpenGLES and plan to work on it from scratch. After it successfully builds a .so and I can verify it works with the Android project file, I will update the wiki with more information and create a pull request.

ENIGMA will still not be able to create complete Android games for some time, but progress is progress.
Title: Re: Android
Post by: sorlok_reaves on August 30, 2014, 06:50:01 PM
Wrote up the process on the Wiki ( http://enigma-dev.org/docs/Wiki/Android ). This process assumes that you are working with ENIGMA which has support for Android. As I have not yet pushed a commit, nothing will happen yet!

Thanks for doing this. I'll go through it at some point, as I hope to eventually get Iji working on Android. Good work so far, and good luck with the rest!
Title: Re: Android
Post by: daz on September 01, 2014, 11:48:51 PM
Thanks, we got quite close to getting it to generate an Android binary.

Anyway for anyone curious...
Code: [Select]
c:/androidtoolchain/bin/../lib/gcc/arm-linux-androideabi/4.6/../../../../arm-linux-androideabi/bin/ld.exe: fatal error: C:/Users/Matt/AppData/Local/Temp/egm7307289762881344537.exe C:/ProgramData/ENIGMA/.eobjs/Windows/Android/Run/libEGMstd.o: open: Invalid argument
Linker be failing. This is modified in my attempt to fix things... normally it would output as a .so in the ENIGMA directory, but hey who knows where it might be failing.

Anyway, the only two arguments passed to LD from what I can see are the temp file which will be the finalized binary, and the object file. Both the object file and the temp file exist so I'm not entirely sure what might be the problem. I have a hunch the linker might not realize we're trying to build a dynamic library instead of an executable and the error is just vague, but that's just a guess. If on the other hand it doesn't like ".eobjs" for some reason we're in for real trouble...
Title: Re: Android
Post by: Rusky on September 02, 2014, 09:05:52 AM
I'd double check the file types. .eobjs is highly unlikely to be a problem, but missing dynamic library flags, bad file types for the exe or object files, or not having enough information in the exe to link with anything could all be issues as well.
Title: Re: Android
Post by: daz on September 05, 2014, 07:09:19 PM
Well my current guess is LD is being given two parameters and it doesn't like it (the .exe and .o are given at the same time). From running file sniffing tools, when compiling for Windows this situation literally never arises (unless it's well hidden). It's trying to create a file named "C:/Users/Matt/AppData/Local/Temp/egm7307289762881344537.exe C:/ProgramData/ENIGMA/.eobjs/Windows/Android/Run/libEGMstd.o" it seems, where as the .o should be passed to the next call of LD I think. It's weird.
Title: Re: Android
Post by: Josh @ Dreamland on September 08, 2014, 10:37:40 AM
It seems that to work around problems of the C:/Documents and Settings/John Doe/My Documents/My Games/super cool game/game.exe ilk, ld just takes EVERYTHING after -o to be the output name. So that's one issue.

I can't tell from what you told me over IRC if it's finding all the objects correctly or not. It'd help to have a dump of the OBJECTS variable.
Title: Re: Android
Post by: daz on September 10, 2014, 05:31:44 PM
I have no idea what I'm doing, so the next time you're on IRC and have some free time we can see if we can progress any farther.