Goombert
|
|
Posted on: February 03, 2014, 03:44:40 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
I wanted to make part of LGM's 1.8.4 development about optimizing GMX loading and saving. Many of us have already discussed this optimization technique, and everybody is already aware. I have started by running a simple test of a way I think I can add it to LGM pretty easily. The idea is pretty simple, you only load the resource tree when somebody opens a project and only load each resource when they open it in an editor, and only write it when they actually make changes. This can effectively bring loading times down to instant even for very big projects. The only downside is that conversion from GMX to another format will be twice as long. The reason is because loading was postponed and when it comes time to write to the new format you now have to load all resources and then write all resources. There are still a few faults with this however, one of them being compile time. When it comes time to compile and run the game all resources need to be in memory. So this would then make the first time you hit the run button twice as long as well. This could also explain why Studio does not do something similar. Here is an image of loading Son of Blagger with my new prototype class below. Note that I have not added the preview icon just yet. I added the following prototype class to the GMX reader. This class is used to postpone a ResNode's resource reference to the first time that it is accessed. Testing with it on egofree's Son of Blagger brought loading time from 2,500ms to 1802ms. public static class GMXSprite extends ResNode { private boolean loaded = false; private String classpath = ""; public GMXSprite(String name,byte status) { super(name.substring(name.lastIndexOf("\\") + 1, name.length()),status,Sprite.class,null); // TODO Auto-generated constructor stub classpath = name; } @Override public ResourceReference<? extends Resource<?,?>> getRes() { if (!loaded && status == ResNode.STATUS_SECONDARY) { loaded = true; Sprite spr = LGM.currentFile.resMap.getList(Sprite.class).add(); res = spr.reference; LGM.currentFile.resMap.getList(Sprite.class).lastId++; spr.setNode(this); String fileName = new File(getUnixPath(classpath)).getName(); spr.setName(fileName);
String path = LGM.currentFile.getPath(); path = path.substring(0, path.lastIndexOf('/')+1) + getUnixPath(classpath); Document sprdoc = null; try { sprdoc = documentBuilder.parse(path + ".sprite.gmx"); } catch (SAXException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } spr.put(PSprite.ORIGIN_X, Integer.parseInt(sprdoc.getElementsByTagName("xorig").item(0).getTextContent())); spr.put(PSprite.ORIGIN_Y, Integer.parseInt(sprdoc.getElementsByTagName("yorigin").item(0).getTextContent())); spr.put(PSprite.SHAPE, ProjectFile.SPRITE_MASK_SHAPE[Integer.parseInt(sprdoc.getElementsByTagName("colkind").item(0).getTextContent())]); spr.put(PSprite.SEPARATE_MASK, Integer.parseInt(sprdoc.getElementsByTagName("sepmasks").item(0).getTextContent()) < 0); spr.put(PSprite.BB_MODE, ProjectFile.SPRITE_BB_MODE[Integer.parseInt(sprdoc.getElementsByTagName("bboxmode").item(0).getTextContent())]); spr.put(PSprite.BB_LEFT, Integer.parseInt(sprdoc.getElementsByTagName("bbox_left").item(0).getTextContent())); spr.put(PSprite.BB_RIGHT, Integer.parseInt(sprdoc.getElementsByTagName("bbox_right").item(0).getTextContent())); spr.put(PSprite.BB_TOP, Integer.parseInt(sprdoc.getElementsByTagName("bbox_top").item(0).getTextContent())); spr.put(PSprite.BB_BOTTOM, Integer.parseInt(sprdoc.getElementsByTagName("bbox_bottom").item(0).getTextContent())); spr.put(PSprite.ALPHA_TOLERANCE, Integer.parseInt(sprdoc.getElementsByTagName("coltolerance").item(0).getTextContent()));
//TODO: Just extra shit stored in the GMX by studio //int width = Integer.parseInt(sprdoc.getElementsByTagName("width").item(0).getTextContent()); //int height = Integer.parseInt(sprdoc.getElementsByTagName("height").item(0).getTextContent());
// iterate and load the sprites subimages NodeList frList = sprdoc.getElementsByTagName("frame"); path = LGM.currentFile.getPath(); path = path.substring(0, path.lastIndexOf('/')+1) + "/sprites/"; for (int ii = 0; ii < frList.getLength(); ii++) { Node fnode = frList.item(ii); BufferedImage img = null; try { img = ImageIO.read(new File(path + getUnixPath(fnode.getTextContent()))); } catch (DOMException e) { // TODO Auto-generated catch block e.printStackTrace(); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } spr.subImages.add(img); } } return super.getRes(); } }
|
|
« Last Edit: February 03, 2014, 04:07:36 am by Robert B Colton »
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
|
Goombert
|
|
Reply #2 Posted on: February 03, 2014, 08:01:43 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
This morning I actually went and implemented a progress dialog so you no longer just sit there and wait but actually have some feedback. GMK and GMX are the only formats that do it right now though as the EGM reader and writer is so complex I don't think you can track its progress easily. But I guess if the compile is slower only once, then it is not that big of a problem. You see this is interesting, and I'll tell you why, because it gives YoYoGames a slight advantage over us. If they did this they could use GMX as their internal serialized format at all times and when it comes time to compile only save the files that were changed. Then the compiler itself would load the resources from this serialized format, which if we could do would bypass LGM populating resources one at a time, and would eliminate loading the whole project and passing it to ENIGMA. I would like to see what Josh has to say about this. Though I don't use gmx at all... Will this change also impact other packed formats like EGM? I was only proposing implementing special tree nodes for GMX that override the basic ones for now. But GMK and EGM can theoretically do this, GMK would just have to move to the correct stream position in order to read only a single resource, not exactly sure about the practicality of it. But all formats in theory can do this.
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
|
Goombert
|
|
Reply #4 Posted on: February 03, 2014, 08:47:01 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
I always wondered why couldn't it only send resources that changed? That actually gives me an idea, if ENIGMA implemented a callback for dumping the old resources, kind of like Studio's clean assets cache option. Then LGM could control when this cache is cleared, and only clear it if you load or create a new project, and otherwise only send new resources to ENIGMA when they are added or when old ones are changed. We could pull that off, and like I said that is probably what Studio is doing now that I think about it. Josh should give some input here. Like create a .dat file or something inside the C:\ProgramData\ENIGMA if need be and just reload/modify that. That would be the same as just making ENIGMA only work with EGM or something. This is only useful if we could implement a command line interface which there has been effort to do before. This is again why Studio has the advantage since it only ever works with GMX internally but import GMK or GMZ and export GMZ. Another solution I thought could be good would just to thread the whole thing. Like if the thing is done in memory now (instead of disk), then couldn't you just make it populate several resources in parallel? Sounds like what Josh was suggesting and telling polygonz how to add. But I was skeptical because Josh never clarified it to me, I could do it probably but I'd like an explanation and reasoning as to how that could improve the performance but I really don't see how since compiling is already threaded. Which also brings me to the implementation of threaded loading. The reason we never threaded loading and saving projects before was because you really shouldn't be dicking around with anything resources or anything while that is occuring since it will all be undone anyway once it finishes and leaves the door wide open for a host of exceptions. Also the reasoning behind the implementation of the modal dialog. Now most IDE's like Visual Studio or Qt or Eclipse will do the GMX suggestion and only load a file once you open it, they just load the tree like the suggestion in my OP. This goes back to what I was saying about ENIGMA's CLI as well, the reason these programs can utilize that effectively is because the respective compilers have a standard format eg. "GNU make" like how Studio only uses GMX internally. All of that is speculation, as I don't know how LGM does that (I guess the plugin does that?) and I don't really plan to Java anytime soon. Yes that is the plugins Job, LGM is not made for a particular purpose. Java isn't scary, don't be afraid of it, C# is just a ripoff of it, down to every single core package. Also Harri in case you didn't realize I only tested on Sprites there with GMX, so once every resource had the abstract tree node you would essentially load a GMX even if it was several Gigabytes in less than a second.
|
|
« Last Edit: February 03, 2014, 08:52:42 am by Robert B Colton »
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
Josh @ Dreamland
|
|
Reply #5 Posted on: February 03, 2014, 12:04:12 pm |
|
|
Prince of all Goldfish
Location: Pittsburgh, PA, USA Joined: Feb 2008
Posts: 2950
|
We could cache resources, yes, but the better idea is to just have the game load from the EGM directly. Nothing needs sent at all, then. When building a standalone, the source code would simply be stripped, and optionally, the EGM appended to the executable.
|
|
|
Logged
|
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble "I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
|
|
|
Goombert
|
|
Reply #6 Posted on: February 03, 2014, 12:25:31 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
I am not exactly following Josh, that would not help to improve GMX compiling times though. If we were going to do that we should do what Studio does and force all projects when loaded to auto-convert to EGM, which I don't particularly care for.
Basically, are you saying if I was working on a GMX and hit compile, the plugin would write out an EGM and the compiler would load that? And that would serve as communication buffer population?
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
Josh @ Dreamland
|
|
Reply #7 Posted on: February 03, 2014, 12:31:16 pm |
|
|
Prince of all Goldfish
Location: Pittsburgh, PA, USA Joined: Feb 2008
Posts: 2950
|
We don't support GMX; we offer loading it and saving to it. If you're trying to compile anything other than EGM, it's not unreasonable that you should expect overhead. Of course, modules can be constructed to allow loading other formats, but those are to be considered third-party; our focus should be on making sure that using our technologies produces the best results that could possibly be expected.
|
|
|
Logged
|
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble "I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
|
|
|
Goombert
|
|
Reply #8 Posted on: February 03, 2014, 12:51:00 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
Alright so you are ok if I restructure the abstract base ResNode class to work this way with the various readers and writers?
If so we'll at least have extremely responsive times working with files, and compile time will speed up anyway because LGM's current resource population is slower than just saving an EGM.
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
|
Josh @ Dreamland
|
|
Reply #10 Posted on: February 03, 2014, 06:56:48 pm |
|
|
Prince of all Goldfish
Location: Pittsburgh, PA, USA Joined: Feb 2008
Posts: 2950
|
ENIGMA's compiler does not presently support any formats natively, including EGM. LGM has all the resource readers/writers, so from the get-go, we just let LGM do that work for us. However, formatting the data into C-friendly structures is apparently an insurmountable task for the JVM. Possibly due to the fact that Java has the opposite endianness.
|
|
|
Logged
|
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble "I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
|
|
|
Goombert
|
|
Reply #11 Posted on: February 04, 2014, 09:36:38 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
The first part I decided to go ahead and do is moving all the glyph and font metric shit out of the plugin and into LateralGM. This is for several reasons, but the base reason is GMX requires a texture stored with every font as well as the glyph metrics in the properties file. This will also make it easier to simply pass an EGM to the compiler by dumping the glyph metric information into the EGM. I guess this is why GM also dumps it to the GMX. Now we can add proper support for multiple character ranged fonts. This however leaves me with some questions for Josh. 1) The current approach I made was to store the ranges and glyphs into a data Eef writer alongside the properties file. They look like this. Ranges: - [32, 127] - [32, 127] - [32, 127] Glyphs: - [132, 0, 0, 12, 12, 0] - [133, 12, 0, 12, 12, 0]
Now, currently I do not statically type out the parameter order for each property array, and because of ISO not guaranteeing parameter order if this would be an issue with YAML and Java? Also because the following occurs with all GMX projects exported by LGM. <glyph character="113" h="18" offset="1" shift="9" w="7" x="123" y="22"/> <glyph character="87" h="15" offset="0" shift="15" w="15" x="19" y="2"/>
Notice h is before offset and shift and not after w? 2) To make it possible for EGM to just be passed to the compiler, I will also need to store the texture into the EGM. I can do two things, I can tack it onto the end of glyph/ranges data file as just pure binary preceded by a 4 byte size header and just make that part of the data file illegible when opened in say notepad. The other option is to store a second data file, eg. "font_0.tex" and add a property to the range/glyph data file that tells us what format it is in, kind of like I did with shaders where I had two data files. Which do you like better? 3) So you like the simple way of sub-classing ResNode in order to accomplish responsive file editing? And also, do you think it is worth it to try and implement this behavior for GMK as well, or only EGM and GMX?
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
IsmAvatar
|
|
Reply #12 Posted on: February 05, 2014, 01:56:56 pm |
|
|
LateralGM Developer
Location: Pennsylvania/USA Joined: Apr 2008
Posts: 877
|
Notice h is before offset and shift and not after w? The properties are alphabetized. c h o s w x y ENIGMA's compiler does not presently support any formats natively, including EGM. LGM has all the resource readers/writers, so from the get-go, we just let LGM do that work for us. However, formatting the data into C-friendly structures is apparently an insurmountable task for the JVM. Possibly due to the fact that Java has the opposite endianness. It hasn't always been this way. We've dabbled with a few options, eventually having LGM export the data to a file, and calling ENIGMA as an EXE - simply because I couldn't figure out how to get dlls to work in Java (using our own format, EGF, I think, so CLI still wasn't a viable option C-side only). It wasn't until I finally broke down and agreed on a JNA dependency that we were able to do what seemed more natural. Unfortunately, as Josh pointed out, transferring data this way is still a struggle on the JVM. Sorry for skimming - I might be repeating what you already said, but what you might consider doing is some kind of hybrid where ENIGMA has a native EGM format, and ENIGMA itself will load any unloaded resources from file - while the loaded resources will be passed over through JNA->C-structures. This could further be optimized by only keeping track of which resources are changed, so LGM doesn't have to populate structures for unchanged resources. While we're doing this, as Josh suggested, we might even consider not bothering passing the more binary resources, and rather keep them in the EGM - especially if ENIGMA is just going to turn around and write them to a file anyways - it's better if we save the overhead of JNA->C-structures for binary data (which are, measurably and unarguably, the biggest cycle offender) and just write it to the file ourselves.
|
|
|
Logged
|
|
|
|
Goombert
|
|
Reply #13 Posted on: February 05, 2014, 03:33:37 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
c h o s w x y Oh wow you're clever I did not notice that, I wonder if that is what is behind the ISO file_text_read() thing, perhaps it behaves the same way. GM traditionally did however guarantee the order, which we established with Project Mario, hence the reason for me re-releasing the game ISO/C compliant. So basically, I just want to clarify, you and Josh are not against ENIGMA being able to load EGM's without the plugin? If so that is awesome, I would have done much more work on the Command Line Interface a long time ago. And to further clarify, IsmAvatar, what do you specifically think of delaying the loading of each resource to the first time it is accessed?
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
IsmAvatar
|
|
Reply #14 Posted on: February 06, 2014, 09:50:11 am |
|
|
LateralGM Developer
Location: Pennsylvania/USA Joined: Apr 2008
Posts: 877
|
I am not against ENIGMA natively supporting reading EGMs. From my point of view, ENIGMA is and always has been intended as independent from any IDE, and having a single input format that allows it to run standalone should be a goal. The fact that this could potentially ease the strain on communicating information from the IDE to ENIGMA is a bonus. I think if there's any single main reason this was delayed, it was because EGM was never formally finalized (since LGM was the testing ground and I never really finished with it). what do you specifically think of delaying the loading of each resource to the first time it is accessed? I'm in favor of it. It was one of the main selling points of EGM. Much of the modularization of LGM has been towards that end, so hopefully you will at least find hooking it up to be fairly straightforward. Unfortunately there are a large number of aspects of the interface that were never designed with such a use case in mind, so we will run into unexpected bugs that will need to be figured out and kinked out - the resource preview icons being the most obvious (and we were making plans on how to handle that). Less obvious things will be resource references, such as the Room editor being essentially useless without all the dependent objects, sprites, and backgrounds loading unless you can resolve some sort of independent render storage system - which is pretty silly since loading the renders of those things is the bulk of the loading time - the rest is small data. And finally, and perhaps most significantly, this will alter the "Save" concept, which pretty much nothing supports changing this, so I expect it will fight you the whole way.
|
|
« Last Edit: February 06, 2014, 09:52:28 am by IsmAvatar »
|
Logged
|
|
|
|
|