Darkstar2
|
|
Posted on: February 01, 2014, 01:18:42 am |
|
|
Joined: Jan 2014
Posts: 1238
|
I have been pondering on this for quite some time now. At the current stage I don't like the way GameMaker handles resources. It is better suited for smaller games that can load all resources at startup but can become a memory hog and problematic for those with a bit more ambition who want to make big games and use large sprites and other resources.
For a while I have been thinking maybe I should code my own engine (scripts actually) to handle this with ease. Here is an example: (this is just what I was visioning, nothing was made yet) so this serves as an example:
Level complete, everything uneeded is freed from RAM, new level resources to be loaded from single encrypted resource files, the script would automatically handle locating the individual file within the single file, decrypting it and storing it, it would use an encrypted pointer file / index file to do its trick as well. Only one command would be used to extract from the single resource file, be it sound, music, background and video.
res_load (music.pak,act2.mp3,act2_snd)
So in this example res_load will pass along MUSIC.PAK which is the single large file containing all game music and extract act2.mp3 from it and store it in act2_snd.
This would be quite practical for large adventure games whether point and click, scrolling, etc, only loading the background / scenery, music, sound and video cut scenes as needed.
Now I can get this accomplished by coding a script to handle all this work and could plug it into any game I need this for. One advantage to that is that the resources are not stored individually and easily accessed by someone, they are PACKED inside one file and encrypted. That is an easy part to make a script for, the trickier is the custom encryption and proprietary nature of my idea.
Now as I was thinking of that, I asked myself, wouldn't it be simpler to just use 7z which is an open source, to pack my resources into single files and simply use 7z to EXTRACT whatever file I want and use it then delete it when I don't need it ? That was possible with the shell execute command in earlier GM versions but this option was obsoleted and not available in STUDIO. I don't believe this is possible in ENIGMA either due to the compiled nature, so there is no way to access programs externally from a compiled enigma project. Of course the script method, I would have to extract what I want to disk then later load it to memory using built in functions inside my script such as sound_add, video_add, background_add, sprite_add, etc......So what I would be doing is using the pointer file seeking to the right spot on the resource file and binary reading bytes for the entire file to be extracted, but I would have to save the resulting file......Unless there is a way to append these bytes to a string for example res_loaded is the string containing all the read bytes off of the mp3 file, then I could use the sound play and refer to the string ? I don't think this can be done. The sound_add, video_add, etc......DIRECTLY load into RAM, but they load from individual files, which is great, I would like to apply the same by directly extracting from a resource file into RAM.
Any suggestions ? The direct to memory extraction from a large resource file would avoid needing a disk cache and save an extra step (freeing up CPU and speeding up loading).
Using my method everything would be encrypted so it would not be possible for people to RIP resources using one of many hack tools out there.
|
|
« Last Edit: February 01, 2014, 01:20:59 am by Darkstar2 »
|
Logged
|
|
|
|
Josh @ Dreamland
|
|
Reply #1 Posted on: February 01, 2014, 02:15:26 am |
|
|
Prince of all Goldfish
Location: Pittsburgh, PA, USA Joined: Feb 2008
Posts: 2950
|
You might be interested in reading my remarks on the matter from the Wiki To-do page. I'd consider .pak files to be a viable bundling mechanism. You can easily call 7zip using execute_shell. If that function is not implemented, you can instead use C++'s system() function, which is a bit hacky, but viable. Using 7zip externally, however, will cost you additional disk operations, so I wouldn't personally recommend it.
|
|
|
Logged
|
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble "I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
|
|
|
Darkstar2
|
|
Reply #2 Posted on: February 01, 2014, 02:59:52 am |
|
|
Joined: Jan 2014
Posts: 1238
|
Thanks but I am confused, what do you mean it will cost me disk operations ? If I use 7z called externally through C++ I will be essentially doing the same thing as my proprietary method, except with 7z the resources would be compressed. 7z would extract and save the result to disk, then later called by *_add to load them in memory. With my script it would do the same, extract bytes from x to y saving the result to disk and calling it in memory using *_add, in which case perhaps it would be quicker to use 7z right ? Think I will experiment with my own packing/unpacking engine for now and skip 7z, see where it leads me. Yes PAK files. or files can be renamed anything basically. From what I recall in some games I have, PAK files where actually ZIP files renamed ! I am looking at making a proprietary format that cannot be opened or ripped from using tools. If ENIGMA could support this natively and handle all this dynamically that would be so amazing, but I understand that would require some time to get implemented right ? BTW I recall someone saying Yoyo does use something similar in studio, I do know they use a resource file but don't think it is handled properly. So to make the proprietary format I would have to make 2 programs, one would be used to PACK and ENCRYPT resources, which would be a stand alone, and the 2nd would be the script to handle the extraction. So if I have to use C++ inside my script would I need to do it like this ? #define PROG "(whatever filename).exe" int main() system(PROG); return 0; } Thanks, i'm not fluent in C++ so please guide me if I have something wrong BTW isn't it bad practice to be calling external programs from within a program this way ? Wouldn't that be detected as a suspicious program by an AV ?
|
|
« Last Edit: February 01, 2014, 03:10:17 am by Darkstar2 »
|
Logged
|
|
|
|
Josh @ Dreamland
|
|
Reply #3 Posted on: February 01, 2014, 03:32:41 am |
|
|
Prince of all Goldfish
Location: Pittsburgh, PA, USA Joined: Feb 2008
Posts: 2950
|
7z would extract and save the result to disk, then later called by *_add to load them in memory That's exactly the problem. Using 7zip externally, your resources have to be read from disk (compressed, by 7z), written to disk (uncompressed, by 7z), then read from disk (uncompressed, by you). Using a 7zip library instead would allow you to read the resources once, compressed, and unpack them in memory. I had not intended to allow users to specify their own formats for automated resource loading. A long time ago, I was going to allow users to specify their own encryption routines. There isn't much point to it, as anyone with a debugger can call your own load method to unpack your files; it will just make modding your games a science instead of an art. If your game can read it, so can your users. A 7zip extension for ENIGMA would not take much effort to code. There hasn't been any interest in the proposition. For a while, ENIGMA used zlib on its resources. That's since been removed in favor of just letting users zip their modules, which produces better file sizes. You can call C++ functions in ENIGMA the same way you call any other function. Just [snip=EDL]system("7zip your-options your-file")[/snip] will suffice. I don't know of any antivirus that targets all programs that make shell calls. They'd probably not sell well. Game Maker was targeted for using ASProtect to unload an encrypted executable into memory—that's the kind of sneaky behavior antivirus programs look for.
|
|
|
Logged
|
"That is the single most cryptic piece of code I have ever seen." -Master PobbleWobble "I disapprove of what you say, but I will defend to the death your right to say it." -Evelyn Beatrice Hall, Friends of Voltaire
|
|
|
Darkstar2
|
|
Reply #4 Posted on: February 01, 2014, 12:11:26 pm |
|
|
Joined: Jan 2014
Posts: 1238
|
That's exactly the problem. Using 7zip externally, your resources have to be read from disk (compressed, by 7z), written to disk (uncompressed, by 7z), then read from disk (uncompressed, by you).
Agreed, but so would my own engine (read byte range x to y from resource pak save to disk, read back using *_add. So you have the same read save read sequence right ? Unless there was a function that works the same way as for example background_add, but instead of reading from a single file it would read bytes from range x to y from a given file and load to memory as with the *_add functions. Combined with the GM string encrypt functions (don't think those are supported yet in ENIGMA). I guess for now I will play with my own (experimenting a little). BTW, does ENIGMA support DLL inside my projects ?
|
|
« Last Edit: February 01, 2014, 12:17:30 pm by Darkstar2 »
|
Logged
|
|
|
|
time-killer-games
|
|
Reply #5 Posted on: February 02, 2014, 12:38:29 pm |
|
|
"Guest"
|
It does support DLL's I've tested this a while back and it worked.
I personally find the idea of loading external resources a great idea. But encrypted ones? Not so much. It would be better if we had a regular zip file and have it password protected and extracted to a well hidden temp directory at runtime with files deleted when no longer in use or when the game is closed. This is exactly how GMStudio protects the html5 runner from being pirated. In the GMS directory there is the html5 runner in a password protected zip. When you compile to html5 the runner is quickly extracted, read, then deleted all rather quickly and behind the scenes in a hidden temp directory. If this method of protection is good enough for YYG there's no reason why we can't do this too for protecting include files and external resources.
But if this does ever happen, there needs to be a way to specify within ENIGMA whether to encrypt or protect each file individually because there are times you'll need those files accessaable to players, I. E. customizable sprite sheets, etc.
|
|
« Last Edit: February 02, 2014, 12:40:38 pm by time-killer-games »
|
Logged
|
|
|
|
Goombert
|
|
Reply #6 Posted on: February 02, 2014, 12:48:28 pm |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
Sorry I didn't get a chance to read this yet, let me write a reply. I have been pondering on this for quite some time now. At the current stage I don't like the way GameMaker handles resources. It is better suited for smaller games that can load all resources at startup but can become a memory hog and problematic for those with a bit more ambition who want to make big games and use large sprites and other resources. This is actually one of my main gripes, although they claim to be making all these new advancements, they are forgetting functions like audio_add, which does not exist, at all, well except in ENIGMA. They seem to be wanting to do away with external resources. So in this example res_load will pass along MUSIC.PAK which is the single large file containing all game music and extract act2.mp3 from it and store it in act2_snd. What you are suggesting is similar to Grand Theft Auto's IMG format. http://www.gtamodding.com/index.php?title=IMG_archivedon't believe this is possible in ENIGMA either due to the compiled nature, execute_shell/execute_program is implemented in ENIGMA I even went and documented the functions for you. http://enigma-dev.org/docs/Wiki/Execute_shellhttp://enigma-dev.org/docs/Wiki/Execute_programIf ENIGMA could support this natively and handle all this dynamically that would be so amazing, but I understand that would require some time to get implemented right ? It's very possible somebody just needs to take a popular ZIP API or 7ZIP's and write an extension for it. I would if I had a lot more time on my hands. It does support DLL's I've tested this a while back and it worked. You do realize all the cool extensions like GMOgre, 3D particles, Ultimate 3D, etc. etc. Are all still broke, right? They all need rewritten because every GM extension relied on GMAPI which is defunct with Studio. Edit: Sorry, I thought you were talking about GM. It would be better if we had a regular zip file and have it password protected and extracted to a well hidden temp directory at runtime with files deleted when no longer in use or when the game is closed. Yes, but the point here is to not force this on everyone like what YYG tries to do. The better idea would be to simply implement file_temp_* functions and write a zip_* extension.
|
|
« Last Edit: February 02, 2014, 01:02:15 pm by Robert B Colton »
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
Darkstar2
|
|
Reply #7 Posted on: February 02, 2014, 10:57:07 pm |
|
|
Joined: Jan 2014
Posts: 1238
|
It does support DLL's I've tested this a while back and it worked.
Good to know thanks, although the majority of the DLLs in the GM community are outdated and go way back, most are broken, not documented and buggy. Luckily I will not be needing DLL. I think I want to start expanding my knowledge and learning some C++. I really like the fact one can use C++ inside my projects alongside what I already know. Great program. BTW I heard mention of a new compiler in the works, what will this do or intend to do ? I personally find the idea of loading external resources a great idea.
A great idea and a standard practice, as you see now commercial games take multi gigabytes and are massive, they all use external resources. The days of the games that pack everything inside the EXE and load all in memory are long gone. As far as encryption yes I do realize that anything can be ripped using tools, in RAM, but the whole idea is to make it difficult for most people. When one works so hard to make a game with original content, it is a good to want to protect said work. But encrypted ones? Not so much. It would be better if we had a regular zip file and have it password protected and extracted to a well hidden temp directory at runtime with files deleted when no longer in use or when the game is closed.
That can easily be hacked using a tool to monitor registry, disk access, RAM, etc. One could have the files immediately deleted once the game loses focus and / or exited. However, again it does not stop the most persistent of hackers which can use tools to undelete files. But it will stop the average person (most people) playing your game to say "WOW this is amazing, and one of the best games I played in my life, I think I'm gonna rip off the developer and steal his work " that's why I mention encryption. directory. If this method of protection is good enough for YYG there's no reason why we can't do this too for protecting include files and external resources.
In my opinion anything of the sort that gets implemented would have to be optional and flexible. The use of external resources and such method might not be needed for all games.
|
|
|
Logged
|
|
|
|
Darkstar2
|
|
Reply #8 Posted on: February 02, 2014, 11:07:27 pm |
|
|
Joined: Jan 2014
Posts: 1238
|
This is actually one of my main gripes, although they claim to be making all these new advancements, they are forgetting functions like audio_add, which does not exist,
really!?!?!? I thought it was a function that did exist in GM, at least in 8.1, and I recall in early versions of GM:S...hmm. I recall something that GM:S already uses external resource files, in my opinion it is poorly implemented. at all, well except in ENIGMA. They seem to be wanting to do away with external resources.
Weird, and so how are you supposed to make big games then ? If I wanted to make an adventure game with tons of hi-res scenery and sprites, how would I do that without external ? I think they use external handling only for certain things, but from what I saw it is not done well and you have no control over it. execute_shell/execute_program is implemented in ENIGMA
Interesting, so I don't have to use system(). I even went and documented the functions for you.
Great ! Thanks! Quite an amazing and helpful community here, I'm impressed. I completely hate the theme of their forum there. It does support DLL's I've tested this a while back and it worked. You do realize all the cool extensions like GMOgre, 3D particles, Ultimate 3D, etc. etc. Are all still broke, right? They all need rewritten because every GM extension relied on GMAPI which is defunct with Studio. [/quote] Indeed, lot of the DLLs are outdated and non functional. Once I asked YoYo about implementing some functions and their reply was "we won't...... use a DLL" LOL. The only DLLs that did what I was looking for did not work in Studio. Nice isn't it ?
|
|
|
Logged
|
|
|
|
Goombert
|
|
Reply #9 Posted on: February 03, 2014, 12:28:31 am |
|
|
Location: Cappuccino, CA Joined: Jan 2013
Posts: 2993
|
really!?!?!? I thought it was a function that did exist in GM, at least in 8.1, and I recall in early versions of GM:S...hmm. I recall something that GM:S already uses external resource files, in my opinion it is poorly implemented. sound_add exists in 8.1 and lower, Studio uses a new 3D positional audio engine, or basically just OpenAL, they also butchered the original implementation pretty bad, they just deprecated half of the functions too. They have never added audio_add. Weird, and so how are you supposed to make big games then ? If I wanted to make an adventure game with tons of hi-res scenery and sprites, how would I do that without external ? I think they use external handling only for certain things, but from what I saw it is not done well and you have no control over it. Right, they really don't care about that, because it is primarily a 2D game engine, and all the attention is on mobile development. In fact you no longer even have access direct access to your projects working_directory, a working directory for an application is usually the path where the executable is run. https://en.wikipedia.org/wiki/Working_directoryInteresting, so I don't have to use system(). The problem with those two functions however is they are unlikely to work for mobile devices and you'll probably need to check os_type to run a different command on Linux/Mac. Nice isn't it ? I have a conspiracy theory they broke all those extensions on purpose so people would upgrade :\ Anyway it is important to note, things like our Box2D extension for physics uses a dll, though not actually. Our extensions build the library directly inside the game executable because all of the extensions I wrote are static linked.
|
|
|
Logged
|
I think it was Leonardo da Vinci who once said something along the lines of "If you build the robots, they will make games." or something to that effect.
|
|
|
Darkstar2
|
|
Reply #10 Posted on: February 03, 2014, 02:05:20 am |
|
|
Joined: Jan 2014
Posts: 1238
|
sound_add exists in 8.1 and lower, Studio uses a new 3D positional audio engine, or basically just OpenAL, they also
Ok now I remember it was for 8.1 indeed, it's strange but I thought they had included that in GM:S 1.0, oh well, from what I recall they use FFMPEG that is externally called each time a sound is played:P I remember on the forum some people complained they saw hundreds of instances of FFMPEG in their tray LOL. I have not used GM:S much at all sine11.2 due to time constraints, are they still using FFMPEG? butchered the original implementation pretty bad, they just deprecated half of the functions too. They have never added audio_add.
So is that how they work ? Ooops we fucked something up bad let's just remove it.... ?Why does this sound so familiar lol, does it remind you of other companies that make software ? Right, they really don't care about that, because it is primarily a 2D game engine, and all the attention is on mobile development. In fact you no longer even have
What ? So a 2D engine, but people don't have more ambition ? high definition sprites, scenes, heavy content ? I think they figured out that serious developers who want to make heavy games will not bother touching GM and use competition - but it's sad, because their competition is doing so much better in so many areas. Personally I have always been polite and I try to distance myself from flame wars and public bashing on the GMC forums, so I am on good terms however, one thing I witnessed many times that really gets to me is how rude some people get replied to, particularly in the mantis, paying members requesting features or help etc.......rudeness from yoyo staff, mods and other members, I have observed this since YoYo took over. What a shame really. It's one thing to bash but many (including you and others) have made factual, valid, constructive criticism. I am also well aware now that some people got banned too! access direct access to your projects working_directory, a working directory for an application is usually the path where the executable is run. https://en.wikipedia.org/wiki/Working_directory
yes things changed a lot when they started adding platform exports, I guess they have to abide by some restrictions for apps (security restrictions) according to them ! LOL when you see all those apps in the store practically want access to your underwear I wonder how valid their reasoning is ? Interesting, so I don't have to use system(). The problem with those two functions however is they are unlikely to work for mobile devices and you'll probably need to check os_type to run a different command on Linux/Mac.
I won't have to worry about this on mobile for smaller projects, obviously they won't be needing external calls And I'm using Windows. I have a conspiracy theory they broke all those extensions on purpose so people would upgrade :\
Ohhhh thank you ! Confirmation that I am not the only one to make conspiracy theories abut the evolution of game maker. I guess that means I agree 100%. But again, in all fairness, that's business, and many software companies do that.......*cough*Microsoft*cough*cough* Anyway it is important to note, things like our Box2D extension for physics uses a dll, though not actually. Our extensions build the library directly inside the game executable because all of the extensions I wrote are static linked.
BTW has anybody done some tests to compare GM:S compiled games side by side with ENIGMA ? (meaning YYC Compiled Windows games vs. ENIGMA ?) It would be nice to have a site with some extensive tests... GPU intensive GM vs.ENIGMA CPU intensive GM vs. ENIGMA CPU intensive GM YYCvs. ENIGMA etc. Also do you think that in the next official version of GameMaker (codename GM:NEXT) but probably going to be called Studio 2.0, they will allow C++ code and GML to coexist, meaning you can use both GML and C++ I wonder what they have in store.
|
|
|
Logged
|
|
|
|
|