ENIGMA Forums

Outsourcing saves money => Programming Help => Topic started by: Darkstar2 on July 23, 2014, 08:56:25 pm

Title: C++ short delay when using CIN + file access discussion
Post by: Darkstar2 on July 23, 2014, 08:56:25 pm
This question is for C++ users, I noticed when using cin in my projects there is a noticeable pause of 1 second before the cursor appears.

Example, when making a program that displays a menu followed by a cin, when running the menu would display instantly, but there is a delay before the cursor appears, it's as if the cin takes time to "start".  Why is this ?

Code: (cpp) [Select]
#include <iostream>

using namespace std;

int x = 0;

int main()
{
    cout << "-Options-" << endl;
    cout << "1. Pack with encryption" << endl;
    cout << "2. Pack without encryption" << endl;
    cin >> x;
    return 0;
}

Yes even on the release compile, same thing.  I mean 1 second ? WTH!  When I coded BASIC and used INPUT it was instant.
Title: Re: C++ short delay when using CIN
Post by: edsquare on July 23, 2014, 09:49:50 pm
Don't know why it happens to you, I tried it and I see no delay  ???

Code: [Select]
#include <stdio.h>
#include <string.h>
#include <iostream>

using namespace std;

int main()

{

int x = 0;

{
    cout << "-Options-" << endl;
    cout << "1. Pack with encryption" << endl;
    cout << "2. Pack without encryption" << endl;
    cin >> x;
    return 0;
}
   
getchar();

return 0;

}
Title: Re: C++ short delay when using CIN
Post by: Josh @ Dreamland on July 23, 2014, 10:00:50 pm
He's on Windows. The console is GOING to be terribly slow.
Title: Re: C++ short delay when using CIN
Post by: edsquare on July 23, 2014, 10:02:51 pm
He's on Windows. The console is GOING to be terribly slow.

I compiled it as an exe and executed it under wine, shouldn't that be just as slow?

EDIT: Just noticed I placed the variable in a different place, changed it so it would be where Darkstar2 put it but still no delay.
Title: Re: C++ short delay when using CIN
Post by: Josh @ Dreamland on July 23, 2014, 10:13:19 pm
Write a custom terminal emulator that, upon printing a character, deletes the entire content of the terminal by filling it with spaces, then re-prints everything, including the new character. That might get it to be as slow. Probably not worth the effort, though.

Disclaimer: Not implying that's what Windows does. I can't conceive of why the Windows command prompt is that damn slow. Especially since it takes so many disgusting shortcuts.
Title: Re: C++ short delay when using CIN
Post by: edsquare on July 23, 2014, 10:21:12 pm
Write a custom terminal emulator that, upon printing a character, deletes the entire content of the terminal by filling it with spaces, then re-prints everything, including the new character. That might get it to be as slow. Probably not worth the effort, though.

Disclaimer: Not implying that's what Windows does. I can't conceive of why the Windows command prompt is that damn slow. Especially since it takes so many disgusting shortcuts.

Damn I'm glad I started using Linux before trying to relearn to program.
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 23, 2014, 10:37:04 pm
Why did you change my code there ?
Why 2 returns ? The example I gave is fine.

Could it have to do with the fact I am running Windows 7 64bit ?

I've made tons of console apps in my days, in QB, and the input prompt was instant.  Shocked that it takes 1 second ! LOL.

BTW this was just a question out of curiosity.
Majority of my upcoming projects won't use CIN, but will take commands directly from command line.  Though it's kinda odd, I also remember working with batch files back in the days and it was instant too....
I have a PC with Windows XP on it I will go try running it there, if CIN prompt is instant (no delay) then it's a windows7 or windows7 64bit thing. :D

Title: Re: C++ short delay when using CIN
Post by: edsquare on July 23, 2014, 10:43:19 pm
Why did you change my code there ?
Why 2 returns ? The example I gave is fine.

Could it have to do with the fact I am running Windows 7 64bit ?

I've made tons of console apps in my days, in QB, and the input prompt was instant.  Shocked that it takes 1 second ! LOL.

BTW this was just a question out of curiosity.
Majority of my upcoming projects won't use CIN, but will take commands directly from command line.  Though it's kinda odd, I also remember working with batch files back in the days and it was instant too....
I have a PC with Windows XP on it I will go try running it there, if CIN prompt is instant (no delay) then it's a windows7 or windows7 64bit thing. :D

Didn't changed it on purpose, just copied the output/input part over a small program I already had (a homework of my son), so that's why the two returns, I didn't notice it  :(

And since I only copied your output and input and left everything else as it was, that's also why I wait for the return key to be pressed.  :(

I also don't remember GWBASIC nor QBASIC being slow on windows, but I don't remember if it was XP or windows98  :o
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 23, 2014, 11:30:03 pm
I am not going to open another topic,  but I have a slightly O/T question :D

it concerns dynamic memory allocation,

what happens if you use NEW but don't use DELETE and you exit your program.  Will the OS automatically free any memory allocated by your CPP, or will the memory not be freed.

Let's say for example

Code: (cpp) [Select]
char* resMem = new char[5000];

What happens if I let the program exit or forcibly exit and don't use delete ? Will 5000 bytes of RAM be lost in space unless I restart windows or use a RAM freeing tool or will the OS automatically free it upon exit ?

Again this is a question out of curiosity, as I know better than to not free RAM I allocate :P
BTW, I'm assuming that once the process ends the memory allocated along with it is freed, but just want to make sure :D

2) (yes it's a 2 part question).   What's the command so I find out HOW MUCH RAM was allocated ?  In the example above I know that 5000 bytes were allocated as char = 1 byte, right ?  for an int it would have been 4 * 5000, etc.  so how do I find out, in the example above, how much RAM was allocated.

Thanks :)

and eds don't worry about it, I was wondering why my code was modified was afraid maybe I switched to a parallel universe :D
Title: Re: C++ short delay when using CIN
Post by: edsquare on July 23, 2014, 11:50:37 pm
I am not going to open another topic,  but I have a slightly O/T question :D

it concerns dynamic memory allocation,

what happens if you use NEW but don't use DELETE and you exit your program.  Will the OS automatically free any memory allocated by your CPP, or will the memory not be freed.

Let's say for example

Code: (cpp) [Select]
char* resMem = new char[5000];

What happens if I let the program exit or forcibly exit and don't use delete ? Will 5000 bytes of RAM be lost in space unless I restart windows or use a RAM freeing tool or will the OS automatically free it upon exit ?

Again this is a question out of curiosity, as I know better than to not free RAM I allocate :P

2) (yes it's a 2 part question).   What's the command so I find out HOW MUCH RAM was allocated ?  In the example above I know that 5000 bytes were allocated as char = 1 byte, right ?  for an int it would have been 4 * 5000, etc.  so how do I find out, in the example above, how much RAM was allocated.

Thanks :)

and eds don't worry about it, I was wondering why my code was modified was afraid maybe I switched to a parallel universe :D

Memory allocation... No idea how that works... Do you eat it with a fork?  :D

No problem Ds2, copy paste was the easiest way and I did'nt check for consistency with your code, and I should have, it is good practice to do so.  >:(
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 23, 2014, 11:56:51 pm
I am not going to open another topic,  but I have a slightly O/T question :D

it concerns dynamic memory allocation,

what happens if you use NEW but don't use DELETE and you exit your program.  Will the OS automatically free any memory allocated by your CPP, or will the memory not be freed.

Let's say for example

Code: (cpp) [Select]
char* resMem = new char[5000];

What happens if I let the program exit or forcibly exit and don't use delete ? Will 5000 bytes of RAM be lost in space unless I restart windows or use a RAM freeing tool or will the OS automatically free it upon exit ?

Again this is a question out of curiosity, as I know better than to not free RAM I allocate :P

2) (yes it's a 2 part question).   What's the command so I find out HOW MUCH RAM was allocated ?  In the example above I know that 5000 bytes were allocated as char = 1 byte, right ?  for an int it would have been 4 * 5000, etc.  so how do I find out, in the example above, how much RAM was allocated.

Thanks :)

and eds don't worry about it, I was wondering why my code was modified was afraid maybe I switched to a parallel universe :D

Memory allocation... No idea how that works... Do you eat it with a fork?  :D

No problem Ds2, copy paste was the easiest way and I did'nt check for consistency with your code, and I should have, it is good practice to do so.  >:(

You can eat it with your hands if you wish :D Memory allocation is not that hard, basically they teach you that in kindergarten nowadays :D  so you mashed my code, I forgive you :P BTW that would not compile :P

Tomorrow I will try the CIN on an XP machine. :D

As far as memory I'm assuming memory does get freed by the OS, and that the delete would be avoiding any memory leak/hogs within the program itself / process running, unnecessarily.  :)

Title: Re: C++ short delay when using CIN
Post by: edsquare on July 24, 2014, 12:04:52 am
Quote
and eds don't worry about it, I was wondering why my code was modified was afraid maybe I switched to a parallel universe :D

Memory allocation... No idea how that works... Do you eat it with a fork?  :D

No problem Ds2, copy paste was the easiest way and I did'nt check for consistency with your code, and I should have, it is good practice to do so.  >:(
You can eat it with your hands if you wish :D Memory allocation is not that hard, basically they teach you that in kindergarten nowadays :D  so you mashed my code, I forgive you :P BTW that would not compile :P

BTW you mean my code would not compile? but it did compile without a hitch! and it ran without a hitch!  ???

It didn't on your end?  ???

Then windows is a worse pile of crap than even I thought it was.  :o

Tomorrow I will try the CIN on an XP machine. :D

As far as memory I'm assuming memory does get freed by the OS, and that the delete would be avoiding any memory leak/hogs within the program itself / process running, unnecessarily.  :)

What I've read so far as to memory allocation says you need to free it yourself, except no languages with garbage recollection like Java, Python and C#, not sure if it's so though.
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 12:52:14 am
did not try compiling it, but assuming it won't compile because of the 2 returns, but again I can be wrong on that one.....Been a windows user forever and never had any problem.  Done film, video, audio visual production, mutlimedia both hobby and professionally and never had issue. but again this is not a linux vs. windows debate really let's leave that for another topic :D

As to what your read, yes you have to free memory yourself to avoid memory leak within your app, memory not freed could be potential memory for other process/applications running, and you avoid the risk of stability/crashing or running out of memory, that is what they probably mean by that......as some would for example allocate new memory in a function but exit function without freeing it (creating a leak) etc.

I'll be using memory allocation (more likely dynamic allocation) for some of the stuff I have planned with my resource packer / reader and some of the stuff I will add to ENIGMA soon, hopefully if I have some time, :P

Title: Re: C++ short delay when using CIN
Post by: edsquare on July 24, 2014, 01:04:52 am
did not try compiling it, but assuming it won't compile because of the 2 returns, but again I can be wrong on that one.....Been a windows user forever and never had any problem.  Done film, video, audio visual production, mutlimedia both hobby and professionally and never had issue. but again this is not a linux vs. windows debate really let's leave that for another topic :D

Not being an expert on C++ and since Lazarus helps you a lot with memory management, not sure of what I'm about to say but here it goes:

You can have as many returns as you like, your functions need to return a value and your procedures return an int to check if they worked correctly (at least on pascal they do), but even on C++ you can have more than one return if you need it, to check that something did x or y you can then pass the integer for error management, I think.

I don't know how to explain it, hope you can understand it, because I managed to not understand what I just wrote. :(

About the Windows vs Linux, just name the hour the place and I get to choose the weapons  ;D (Beer and steaks okey with you?) ;)

As to what your read, yes you have to free memory yourself to avoid memory leak within your app, memory not freed could be potential memory for other process/applications running, and you avoid the risk of stability/crashing or running out of memory, that is what they probably mean by that......as some would for example allocate new memory in a function but exit function without freeing it (creating a leak) etc.

I'll be using memory allocation (more likely dynamic allocation) for some of the stuff I have planned with my resource packer / reader and some of the stuff I will add to ENIGMA soon, hopefully if I have some time, :P

Dynamic allocation on C++? Is that possible?
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 01:17:06 am
worked correctly (at least on pascal they do), but even on C++ you can have more than one return if you need it, to check that something did x or y you can then pass the integer for error management, I think.

Gotcha.  and BTW I learned Pascal in school :P  But I have not used Pascal a lot actually, so I forgot lots, I still remember readln and writeln lol.

Right now i'm sticking to C++, EDL, GML for a while should keep me busy and out of trouble lol!

Quote
Dynamic allocation on C++? Is that possible?

Let's say you want to declare an array but you don't know in advance the size, you can't do this

Code: (cpp) [Select]
int x = 100;
int array[x] ; //... etc, it needs a constant. so that won't work....

the new allows this.........

Here is an example,

Code: (cpp) [Select]
int i = 200;
char* p; p = new int[i];

Another example is you want to read a file and store it in memory,
you would get the file size first, store it in a variable and then declare the pointer and allocate RAM based on the file size (dynamic allocation).

There are many uses for that, such as creating data base reading into memory, reading files, using arrays of variable or unknown sizes, etc.

Hope I have this right :P

I'm going to be using this in my new dynamic resource engine :D


Title: Re: C++ short delay when using CIN
Post by: edsquare on July 24, 2014, 01:26:15 am
worked correctly (at least on pascal they do), but even on C++ you can have more than one return if you need it, to check that something did x or y you can then pass the integer for error management, I think.

Gotcha.  and BTW I learned Pascal in school :P  But I have not used Pascal a lot actually, so I forgot lots, I still remember readln and writeln lol.

Right now i'm sticking to C++, EDL, GML for a while should keep me busy and out of trouble lol!

Didn't mean to say you should switch to pascal, I reference it because it's fresh in my memory and it's compiled like C++

Quote
Dynamic allocation on C++? Is that possible?

Let's say you want to declare an array but you don't know in advance the size, you can't do this

Code: (cpp) [Select]
int x = 100;
int array[x] ; //... etc, it needs a constant. so that won't work....

the new allows this.........

Here is an example,

Code: (cpp) [Select]
int i = 200;
char* p; p = new int[i];

Another example is you want to read a file and store it in memory,
you would get the file size first, store it in a variable and then declare the pointer and allocate RAM based on the file size (dynamic allocation).

There are many uses for that, such as creating data base reading into memory, reading files, using arrays of variable or unknown sizes, etc.

Hope I have this right :P

I'm going to be using this in my new dynamic resource engine :D

Gotcha, I thought you meant dynamic memory allocation, which I believe it's not possible unless you use an interpreted language. (the garbage collector)  :D
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 01:32:52 am
Didn't mean to say you should switch to pascal, I reference it because it's fresh in my memory and it's compiled like C++

I know that :D I was just mentioning this in passing.  I remember acing the Pascal exam and made a big project and think it was one of the highest marked (bragging :D)  But oddly enough I know far more about C++ than Pascal as I have since not touched Pascal !

Quote
Gotcha, I thought you meant dynamic memory allocation, which I believe it's not possible unless you use an interpreted language. (the garbage collector)  :D

:D  I meant a dynamic way of allocating memory as shown above.

I'm not an "EXPERT" in C++ yet, though know enough about it to make full applications, console, some GUI, but to the level I can contribute BIG stuff to ENIGMA - and one thing I am a complete newbie with is graphics, so won't be making an enigma like engine in C++ anytime soon lol!

unless some E.T. puts probes in my brain and feeds me the knowledge, that's the only way I will ever reach that point given my luck :D
Title: Re: C++ short delay when using CIN
Post by: Goombert on July 24, 2014, 03:33:13 am
It does take significantly longer to print to a command prompt than a Linux terminal, like we're talking 10 times slower if LGM outputted to a CMD when compiling than running all the JNA callbacks it does.
Title: Re: C++ short delay when using CIN
Post by: TheExDeus on July 24, 2014, 04:28:47 am
1) Yes, memory is freed at the end of the program if it either crashes or if you didn't free the resources. It's done by C++ runtime (in MinGW case it's libstdc++.dll). It sometimes won't release resource handles though, so if you open a file, but don't close it, then it's possible that over time you won't be able to open new files, as the maximum open files at once has a limit per process. At least it did have a limit in the past. This freeing of memory is true for most devices and OS's. Only some embedded stuff still doesn't do it explicitly.
2) To get size you can use sizeof(). So sizeof(resMem) will return the number of bytes used. It's the same as doing sizeof(char)*5000. Note that you won't be able to use sizeof(array) way in functions, because if you pass the array via function parameter, then it's treated as a pointer. So it will return the size of the pointer, not the array.
3) For dynamic arrays (or actually arrays in general) you should use stl::containers. There is very little use in using the old C arrays anymore. The containers are fast, very optimized and releases you of many stupid tasks. Like freeing the memory, as it's done automatically. In your array case I would suggest using a std::vector. So you could write this instead:
Code: (edl) [Select]
#include <vector>

//in main(){}
std::vector<char> resMem(5000,'a'); //This will create a 5000 element vector of chars (filled with char 'a' specificly)
//Use regular semantics to modify and access them like this
resMem[0] = 'b';
resMem[4999] = 'c';
//You can get size of array
printf("resMem size is %i and in bytes = %i\n", resMem.size(), sizeof(resMem));
//When calculating size in bytes you would have to take vector.capacity() into account as well, but I won't go into that here

//You can add elements
resMem.push_back('d');
//Now resMem.size() is 5001.

So pointers and "new"+"delete" is the bain of C. In C++ there are very few cases you actually have to use them. The biggest thing I have written for myself in pure C++ is 10k line program for research purpose and it only used pointers in about 3 places, because of Abstract Base Class (ABS) usage.
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 12:26:46 pm
It does take significantly longer to print to a command prompt than a Linux terminal, like we're talking 10 times slower if LGM outputted to a CMD when compiling than running all the JNA callbacks it does.

As I mentioned in my examples, the couts display quickly, when I run the executable the text is displayed right away, it is the "CIN" that takes time to be executed, meaning the cursor to take input takes 1 second to appear.
I've never had issues with displaying text otherwise :P
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 12:45:21 pm
1) Yes, memory is freed at the end of the program if it either crashes or if you didn't free the resources. It's done by C++ runtime (in MinGW case it's libstdc++.dll). It sometimes won't release resource handles though, so if you open a file, but don't close it, then it's possible that over time you won't be able to open new files, as the maximum open files at once has a limit per process.

Interesting, ok so about memory what I knew was correct, but in regards to file handles, I was under the impression that those got automatically closed as well.  As far as the maximum per process, it's 32 right ???  So what happens if file handles don't get closed in such case, does it require a window restart ?

Quote
This freeing of memory is true for most devices and OS's. Only some embedded stuff still doesn't do it explicitly.

Well I'm using Windows :P and my target will be windows and maybe soon both windows and linux.

Quote
2) To get size you can use sizeof(). So sizeof(resMem) will return the number of bytes used. It's the same as doing sizeof(char)*5000.

ah yes sizeof forgot about that one. didn't think of using it that way.

Quote
3) For dynamic arrays (or actually arrays in general) you should use stl::containers. There is very little use in using the old C arrays anymore. The containers are fast, very optimized and releases you of many stupid tasks. Like freeing the memory,

So for storing reading/writing files, and in general you mean it's FASTER that way ? If I were to bench both methods one would be significantly faster ? Remember I am not entering any coding competition, so I would use what I know about and what works, but if one method is significantly faster than the other I would definitely use that.

Quote
as it's done automatically.

How would it know WHEN to free the memory ? Only at exit ?  What if I wanted it to free earlier ?

Quote
In your array case I would suggest using a std::vector. So you could write this instead:

Do I have to use std::, I see that a lot in code, can't I just use using namespace std; ?

Quote
So pointers and "new"+"delete" is the bain of C. In C++ there are very few

I thought it was the opposite, that malloc and free are used in C, and that new and delete are C++.

Quote
The biggest thing I have written for myself in pure C++ is 10k line program for research purpose and it only used pointers in about 3 places, because of Abstract Base Class (ABS) usage.

I agree, not much user of pointers, but I intend to use it mostly for file I/O in my project since I will be handling larger data handling that is more than the stack can handle and for allocating memory size I don't know in advance I would have to use pointers in conjunction with new.  But I see pointers, new and delete used in many places in ENIGMA's source :)
Title: Re: C++ short delay when using CIN
Post by: TheExDeus on July 24, 2014, 01:56:31 pm
Quote
Interesting, ok so about memory what I knew was correct, but in regards to file handles, I was under the impression that those got automatically closed as well.  As far as the maximum per process, it's 32 right ???  So what happens if file handles don't get closed in such case, does it require a window restart ?
I think it's safe to say that it closes file handles too. It was very long time ago when I heard that it didn't. Maximum amount of open though is limited if using POSIX C. It's 512 by default, but can be changed. So that is not a big issue (http://msdn.microsoft.com/en-us/library/kdfaxaay%28vs.71%29.aspx).

Quote
So for storing reading/writing files, and in general you mean it's FASTER that way ? If I were to bench both methods one would be significantly faster ? Remember I am not entering any coding competition, so I would use what I know about and what works, but if one method is significantly faster than the other I would definitely use that.
The speed difference if used properly is about the same. Arrays are as low level as you can get. Vector on the other hand is class, which has some memory overhead, but it usually is as fast as an array. But vectors are dynamic, and so when you use it for a dynamic array instead of coding special routines for your own custom array, then it's probably going to be faster. Those containers are also not really used for reading or writing files. Your char array was used for that? Because you will have to use other specific functions to actually write to files, that are not tied to containers the data is stored in.

Quote
How would it know WHEN to free the memory ? Only at exit ?  What if I wanted it to free earlier ?
It's freed at the end of scope. If you create a vector inside the function, then it is freed when the function ends. If it's inside main(){}, then it's freed when the program ends. You can always delete the contents of a vector via vector.clear().

Quote
Do I have to use std::, I see that a lot in code, can't I just use using namespace std; ?
You can use the latter. But it's usually a good coding convention not to pollute namespace with "using namespace", because if it's used in a header, then you effectively make everything in std namespace. That can cause conflicts and hard to find bugs, like the one we had in ENIGMA a few months ago. ENIGMA would not compile a game, because LGM showed an error in valid code, and it took like 3 weeks to fix that. Turned out Robert had included namespace in a header, and it broke LGM. If you want to use only something specific, like vectors, then you can write "using std::vector;". This will allow you not put std in front of vector, but nothing else will be changed.

Quote
I thought it was the opposite, that malloc and free are used in C, and that new and delete are C++.
Correct, my mistake. But what I wanted to say is that you would almost never need to use them. Even in the ABS situation I had, I could of used smart pointers (which is basically wrapping a pointer around a class so it can automatically be destroyed when not in use). Using these things in a modern code is not that great of a practice and while many would disagree, I personally don't like it. I had so many problems with them, then it has scarred me for life.

Quote
But I see pointers, new and delete used in many places in ENIGMA's source :)
Those are in underlying resource structures and in some hack'd classes, like "variant". In most decent code (like the graphics systems) they are not much used.
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 03:27:43 pm
I think it's safe to say that it closes file handles too. It was very long time ago when I heard that it didn't. Maximum amount of open though is limited if using POSIX C. It's 512 by default, but can be changed. So that is not a big issue

512 ok quite far from the 32, I guess 32 I was taking from GML lol!  But anyhow I probably won't be needing that many file handles !  And yes better be safe and always free memory and close files, in practice, something I would always do anyway.

Quote
Your char array was used for that? Because you will have to use other specific functions to actually write to files, that are not tied to containers the data is stored in.

I know how to read and write blocks or bytes of data, but how else am I supposed to store read files in memory ? Let's say I want to read 100 bytes from a binary file, I would use char and store it as an array, easier to have individual access, for bigger files, can't use strings, so I'd have to allocate memory on the heap.   What I want to do is not only read and store file in memory but have access to individual bytes and manipulate them (encryption, etc.). 

Title: Re: C++ short delay when using CIN
Post by: TheExDeus on July 24, 2014, 05:28:21 pm
All of those containers will be allocated on the heap, not that it really matters. But you can just use a char vector. Like this little example I found:
Code: (edl) [Select]
#include <iostream>
#include <iterator>
#include <fstream>
#include <vector>

int main()
{
    // open the file:
    std::ifstream file(filename, std::ios::binary);

    // read the data:
    std::vector<unsigned char> myData((std::istreambuf_iterator<unsigned char>(file)), std::istreambuf_iterator<unsigned char>());

    //Now myData holds the file, so you can access them however you like 
   if (myData[0] == 'a') return 2;
  return 0;
}
There are MANY other ways to load a file into some kind of array. Some are slower (usually the C++ variants) and some are faster (for me C variants are usually faster). Like check this: http://stackoverflow.com/questions/15138353/reading-the-binary-file-into-the-vector-of-unsigned-chars
Title: Re: C++ short delay when using CIN
Post by: Goombert on July 24, 2014, 06:10:51 pm
What container is allocated on the heap? Those sure look like they are being allocated on the stack, doesn't automatic storage duration apply to most of STL?
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 24, 2014, 10:42:55 pm
All of those containers will be allocated on the heap, not that it really matters.

Actually it does matter, there is limit on the size of an array you can use on the stack before you crash your program - I've found out the hard way when trying to use big arrays.  If I will be manipulating large files 50MB, 100Mb, +++ the stack won't cut it :D

Quote
But you can just use a char vector. Like this little example I found:
Code: (edl) [Select]
#include <iostream>
#include <iterator>
#include <fstream>
#include <vector>

int main()
{
    // open the file:
    std::ifstream file(filename, std::ios::binary);

    // read the data:
    std::vector<unsigned char> myData((std::istreambuf_iterator<unsigned char>(file)), std::istreambuf_iterator<unsigned char>());

    //Now myData holds the file, so you can access them however you like 
   if (myData[0] == 'a') return 2;
  return 0;
}
There are MANY other ways to load a file into some kind of array. Some are slower (usually the C++ variants) and some are faster (for me C variants are usually faster). Like check this:

You don't use stdio for files ? Doesn't ENIGMA use stdio ? Isn't it faster ?
I would use that instead, not familiar with the above :P sounds more complicated for nothing.  I've heard that with stdio and memory allocation someone managed to load a file at FULL speed (100MB/s) instead of the regular slower method of 10MB/s :P

Title: Re: C++ short delay when using CIN
Post by: TheExDeus on July 25, 2014, 04:40:44 am
Quote
What container is allocated on the heap? Those sure look like they are being allocated on the stack, doesn't automatic storage duration apply to most of STL?
As far as I know the difference between stack and heap is that stack has limited size, it's not dynamic (so the sizes must be known at compile time) and has limited scope (so it gets deleted when it goes out of scope). stl::containers are dynamic. Their size is usually not known at compile time, as you can always resize them and add/delete as much as you want. This means they are not allocated on the stack. What could be allocated on the stack is the class instance itself, which allows it to exist only in scope. But the memory for data internally probably uses new/delete, to allocate on the heap. That is why you can make a vector as big as you want, and if it fits in RAM, then you are okay.

Quote
Actually it does matter, there is limit on the size of an array you can use on the stack before you crash your program - I've found out the hard way when trying to use big arrays.  If I will be manipulating large files 50MB, 100Mb, +++ the stack won't cut it :D
What I meant is that if you use stl functions and don't use new/delete or regular static arrays, then you don't even have to know what heap and stack is. The whole distinction disappears, like in C# and Java where virtual machines have no such distinction.

Quote
You don't use stdio for files ? Doesn't ENIGMA use stdio ? Isn't it faster ?
I would use that instead, not familiar with the above :P sounds more complicated for nothing.  I've heard that with stdio and memory allocation someone managed to load a file at FULL speed (100MB/s) instead of the regular slower method of 10MB/s :P
As I said, there are many ways to do it. You could try the one I posted, I will try it too, and see how fast it is. I remember I tried many ways to make the fastest file loading possible and the fastest I was able to get was this:
Code: (edl) [Select]
    int load_csv(string fname, int kind, double radius)
    {
        model_primitive_begin(kind);
        ifstream  data(fname.c_str());
        string str_line;
        int line_num = 0;
        float v[3];
        int col;
        //vector<vector<unsigned int> > benchmark_frames;
        //benchmark_frames.reserve(1000);
        while(getline(data,str_line))
        {
            stringstream  lineStream(str_line);
            string        cell;
            int cell_num = 0;
            while(getline(lineStream,cell,' '))
            {
                if (cell_num<3){
                    v[cell_num] = atof(cell.c_str());
                }else{
                    col = atoi(cell.c_str());
                }
                cell_num+=1;
            }
            if (v[2]*v[2]<radius*radius){
                model_vertex_color(v,col + (col << 8) + (col << 16),1.0);
            }
            line_num += 1;
        }
        model_primitive_end();
        return line_num;
    }
If regular ENIGMA takes like 20 seconds to load the file, then this does it in less than 1. But I didn't try loading using the vector copy. That might be faster.
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 25, 2014, 01:53:02 pm
As far as I know the difference between stack and heap is that stack has limited size, it's not dynamic

Stack uses continuous memory, and by default is limited, in theory I could have the linker allocate more space, though I never touched that and never will, when using multiple threads, you are using stacks on each, and potentially wasting space that way. even as a non expert I know better :D

  Perhaps for most people and their application stack will be enough, but bigger games, apps that process large files and memory, you will use the heap, as its size is limited by available RAM (including virtual), and is dynamic.
For what I am going to achieve I will use the heap, or rather I will use your vector suggestion I guess it takes care of more for me :P

Quote
(so the sizes must be known at compile time) and has limited scope (so it gets deleted when it goes out of scope). stl::containers are dynamic. Their size is usually not known at compile time, as you can always resize them and add/delete as much as you want.

stl::containers is definitely something I need then, since I would need to pass the objects between functions, as I like to keep it clean and not do everything in main :P  say for example a file reading function, then passing the read large block to an encryption function and so forth, I know that the non dynamic, regular ways of doing things, it gets lost outside the function (scope).  Something I definitely do not want at this point for my specific needs :D

Quote
As I said, there are many ways to do it. You could try the one I posted, I will try it too, and see how fast it is. I remember I tried many ways to make the fastest file loading possible and the fastest

I discovered the hard way back when I using GM that file functions were HORRIBLY, RIDICULOUSLY slow, but YYG must have assumed most people would use small files and never notice...... But try reading a larger binary and it's a PAIN.......  They read 1 byte at a time, and I believe ENIGMA also, since it is compatible to its GML, also uses the same method.......

I believe with vector / container / new / methods file access should be near native speed though I have yet to try this I'm not at expert level yet and learning new things as I go along :P

In ENIGMA using current functions the most I've gotten was near 10MB/S binary.  LOL !  The slowest P.O.S. drive I used back the days did no less than 30MB/s......  My native sustained read speed is 106-110MB/s, if not more, so with the right C++ code I should be able to achieve this.  Now imagine making a MYST like game or some big adventure games using large assets, and someone has a SSD (they are becoming cheaper) I'm not sure it would be too convenient to load stuff at 10MB/s....it's a disgrace even for old generation PATA drives LOL.

Quote
I was able to get was this:
Code: (edl) [Select]
    int load_csv(string fname, int kind, double radius)
    {
        model_primitive_begin(kind);
        ifstream  data(fname.c_str());
        string str_line;
        int line_num = 0;
        float v[3];
        int col;
        //vector<vector<unsigned int> > benchmark_frames;


IN your example you are reading lines of text from a text file.  I'm guessing, model information :D :P

Quote
If regular ENIGMA takes like 20 seconds to load the file, then this does it in less than 1. But I didn't try loading using the vector copy. That might be faster.

Imagine how long it takes in GM :P Probably close too.

When I get some time I will try experimenting a little, and see where it brings me :D
One thing I am never using is the built in file functions in ENIGMA, I will code my own. I already know how to add new functions and extensions to ENIGMA.

BTW speaking of that, if I were to code a function that uses containers, buckets, vectors, or whatever they are called, and stuff dynamically allocated (heap), in functions, can I use this in my ENIGMA projects.  Example, if I use stl containers and vector and store a large 100 MB file into for example ResRead, and do something like return ResRead, can I pass back that 100MB back the function that was called in my enigma project ?  If that is the case, then I could easily revamp the file functions and add faster more efficient ones for advanced game developers. :)

[/code]
Title: Re: C++ short delay when using CIN
Post by: TheExDeus on July 25, 2014, 02:00:58 pm
Quote
stl::containers is definitely something I need then, since I would need to pass the objects between functions, as I like to keep it clean and not do everything in main :P  say for example a file reading function, then passing the read large block to an encryption function and so forth, I know that the non dynamic, regular ways of doing things, it gets lost outside the function (scope).  Something I definitely do not want at this point for my specific needs :D
Yeah, it's useful when passing to functions, because regular arrays don't hold it's size. So if you pass an array to a function, you cannot know how large it was (so you cannot iterate over it). That requires ugly fixes like passing size as a separate argument. With containers, you always have that information. Just remember passing stuff by reference, so you don't create a copy. I had a project to do with computer vision - I passed images to functions to extract data from them.
Code: (edl) [Select]
double awesomeFunction(std::vector<std::vector<unsigned char> > img){
//Calculate something awesome and return
}
I had like 20FPS, because I was passing data by value. Then I just had to pass it by reference (add &) and it jumps to 60FPS:
Code: (edl) [Select]
double awesomeFunction(std::vector<std::vector<unsigned char> > &img){
//Calculate something awesome and return
}
It's something people learn early, but still something worth reminding. Especially when working with large amount of data.

Quote
I discovered the hard way back when I using GM that file functions were HORRIBLY, RIDICULOUSLY slow, but YYG must have assumed most people would use small files and never notice...... But try reading a larger binary and it's a PAIN.......  They read 1 byte at a time, and I believe ENIGMA also, since it is compatible to its GML, also uses the same method.......
Yeah, we need to address that. One suggestion I have is to have an option to load the whole file (file_open function) and then abstract that fact, so you can still use file_text_readln() and so on, but instead of doing that on the HDD, you do that in RAM. So in essence you would have a much larger speed, but still be compatible.

Quote
IN your example you are reading lines of text from a text file.  I'm guessing, model information :D :P
As the name implies, I loaded data from a CSV file. I saved point clouds in that format (as it's very easy) and used this function to load it in ENIGMA.

Quote
BTW speaking of that, if I were to code a function that uses containers, buckets, vectors, or whatever they are called, and stuff dynamically allocated (heap), in functions, can I use this in my ENIGMA projects.  Example, if I use stl containers and vector and store a large 100 MB file into for example ResRead, and do something like return ResRead, can I pass back that 100MB back the function that was called in my enigma project ?  If that is the case, then I could easily revamp the file functions and add faster more efficient ones for advanced game developers. :)
Well the idea is that you can use C++ in ENIGMA. So you should be able to just call "vector<double> myVector;" straight in ENIGMA. I do make calls like "glEnable(...)" in my ENIGMA projects, so I know it's possible to use stuff that is not just in ENIGMA functions. Problem is that they break GML/EDL compatibility, as you know are using classes instead of ID's. But I personally think we should end using them.
Title: Re: C++ short delay when using CIN + file access discussion
Post by: Goombert on July 25, 2014, 03:22:38 pm
Quote
As far as I know the difference between stack and heap is that stack has limited size, it's not dynamic (so the sizes must be known at compile time) and has limited scope (so it gets deleted when it goes out of scope). stl::containers are dynamic. Their size is usually not known at compile time, as you can always resize them and add/delete as much as you want. This means they are not allocated on the stack. What could be allocated on the stack is the class instance itself, which allows it to exist only in scope. But the memory for data internally probably uses new/delete, to allocate on the heap. That is why you can make a vector as big as you want, and if it fits in RAM, then you are okay.
Ah so we are both right somewhat. According to the link below you can control that.
http://stackoverflow.com/questions/783944/how-do-i-allocate-a-stdstring-on-the-stack-using-glibcs-string-implementation
Title: Re: C++ short delay when using CIN
Post by: Darkstar2 on July 25, 2014, 03:52:09 pm
Quote
stl::containers is definitely something I need then, since I would need to pass the objects between functions, as I like to keep it clean and not do everything in main :P  say for example a file reading function, then passing the read large block to an encryption function and so forth, I know that the non dynamic, regular ways of doing things, it gets lost outside the function (scope).  Something I definitely do not want at this point for my specific needs :D
Yeah, it's useful when passing to functions, because regular arrays don't hold it's size. So if you pass an array to a function, you cannot know how large it was (so you cannot iterate over it). That requires ugly fixes like passing size as a separate argument. With containers, you always have that information. Just remember passing stuff by reference, so you don't create a copy. I had a project to do with computer vision - I passed images to functions to extract data from them.
Code: (edl) [Select]
double awesomeFunction(std::vector<std::vector<unsigned char> > img){
//Calculate something awesome and return
}
I had like 20FPS, because I was passing data by value. Then I just had to pass it by reference (add &) and it jumps to 60FPS:
Code: (edl) [Select]
double awesomeFunction(std::vector<std::vector<unsigned char> > &img){
//Calculate something awesome and return
}
It's something people learn early, but still something worth reminding. Especially when working with large amount of data.

Quote
I discovered the hard way back when I using GM that file functions were HORRIBLY, RIDICULOUSLY slow, but YYG must have assumed most people would use small files and never notice...... But try reading a larger binary and it's a PAIN.......  They read 1 byte at a time, and I believe ENIGMA also, since it is compatible to its GML, also uses the same method.......
Yeah, we need to address that. One suggestion I have is to have an option to load the whole file (file_open function) and then abstract that fact, so you can still use file_text_readln() and so on, but instead of doing that on the HDD, you do that in RAM. So in essence you would have a much larger speed, but still be compatible.

Quote
IN your example you are reading lines of text from a text file.  I'm guessing, model information :D :P
As the name implies, I loaded data from a CSV file. I saved point clouds in that format (as it's very easy) and used this function to load it in ENIGMA.

Quote
BTW speaking of that, if I were to code a function that uses containers, buckets, vectors, or whatever they are called, and stuff dynamically allocated (heap), in functions, can I use this in my ENIGMA projects.  Example, if I use stl containers and vector and store a large 100 MB file into for example ResRead, and do something like return ResRead, can I pass back that 100MB back the function that was called in my enigma project ?  If that is the case, then I could easily revamp the file functions and add faster more efficient ones for advanced game developers. :)
Well the idea is that you can use C++ in ENIGMA. So you should be able to just call "vector<double> myVector;" straight in ENIGMA. I do make calls like "glEnable(...)" in my ENIGMA projects, so I know it's possible to use stuff that is not just in ENIGMA functions. Problem is that they break GML/EDL compatibility, as you know are using classes instead of ID's. But I personally think we should end using them.

So you are telling me I could use those in ENIGMA, but I thought we could not do that inside ENIGMA, meaning can't use STL containers with this parser... I would much rather make it into a C++ function and call it from my ENIGMA project :)
More and more I realise how LAZY and UGLY many parts of GM were implemented...... But that idea of yours is a great one, keeping compatibility with file functions in G**Studio but reading the whole file in memory and reading from the buffer instead of the HDD.  However, keep in mind this has its pros and cons......

#1) More memory consumption
#2) counter productive, especially if you need to open really small files which most people do, so you won't gain any noticeable speed, and in some cases the opposite.

So in my opinion adding NEW functions would be ideal.  People can't have their cake and eat it, there is a hefty price to pay for being compatible to a highly flawed product.  :)

There are times when you DON'T want to load the entire file in memory and read straight from HDD, particularly when you need to read a small chunk of info....... There are times when you would want to read by blocks as opposed to the entire file.......and there are times when you want the entire file in RAM.

So it's nice of YoYoGames to have assumed and DECIDED that 1 byte binary reads was ideal LOL.
:P



Title: Re: C++ short delay when using CIN + file access discussion
Post by: TheExDeus on July 25, 2014, 04:19:01 pm
Quote
#1) More memory consumption
#2) counter productive, especially if you need to open really small files which most people do, so you won't gain any noticeable speed, and in some cases the opposite.
That is what I proposed to be optional. Like an additional argument to file_open functions which has a "bool preload = false" flag, so by default nothing changes. But if you want to load it in RAM, then you can. This way you won't have to make a duplicate set of functions just for this. I also don't think it will be slower for small files. You have to load the file in the ram always anyway. You will be able to load only parts of the file from HDD just like before, but I don't know how reading part of a file to RAM would work here. Maybe another two arguments would be required (read start and size to read).

I just don't want 100 file reading functions in ENIGMA, just for one reading in RAM, reading from HDD, reading from ASCII text, reading from UNICODE text, reading small endian binary, large endian binary etc. We should be able to do it with what we got, but expand to allow more. Like when you load in RAM, you should have an option to have access the buffer and then use buffer_ functions on them. So you can easily send the file via network. We basically need to tie all this together and not have a million different functions. Although that would allow the additional set to be as an extension, thus simplifying management.,