Joined: Aug 2017
Posts: 23

This handy function set allows you to convert a 2d screen coordinate (such as the mouse position) into 3d coordinates  basically the 3d coordinates that it's pointing at.
This is sometimes called unproject, because it reverses the projection process (which converts 3d coordinates into 2d screen coordinates). It's also sometimes called raycasting, since you are effectively casting a geometric ray from the camera/viewport into 3d space, and then in this case colliding it with the zdepth of whatever pixel it is (e.g. z=0 if you click on the floor).
4 functions are provided: d3d_unproject returns a Vector3, if you are ok with OO programming. d3d_unproject_x/y/z returns the respective x, y, and z coordinates, in GM's regular functional programming (where functions return a single number, rather than an object). All functions take an x and a y coordinate, which refer to the location respective to the viewport. I.e. d3d_unroject(mouse_x,mouse_y) Or if you're weird and actually use views in conjunction with 3d: d3d_unproject(mouse_x  view_xview[view_current], mouse_y  view_yview[view_current])
These functions are a little expensive due to complex math going on behind the scenes, so try not to use them more than a few times every step. I.e. if you're using them for mouse coordinates, consider storing the results in a global variable.
NOTE this only works in OpenGL1, as it depends on a call to gluUnproject. I tried porting the code to OpenGL3, but I'm terrible with matrices and ENIGMA uses an awkward matrix format. I've included my attempted OpenGL3 code below if you want to try and fix it.
enigma::Vector3 d3d_unproject(gs_scalar x, gs_scalar y) { GLdouble model[16]; GLdouble proj[16]; GLint view[4]; GLdouble retX, retY, retZ; GLfloat gly, glz;
glGetDoublev(GL_MODELVIEW_MATRIX,model); glGetDoublev(GL_PROJECTION_MATRIX,proj); glGetIntegerv(GL_VIEWPORT,view);
//invert mouse y into openGL y gly = (float)view[3]  (float)y; //Read the depth of the moused pixel as the z //this stops the raycast at the first pixel collision glReadPixels(x,int(gly),1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&glz);
gluUnProject(x,gly,glz,model,proj,view,&retX,&retY,&retZ); //TODO: Provide a raycast_to_floor, stopping at a desired z rather than the first pixel collision //I believe the way to do this is to call gluUnProject twice at glz=0 and glz=1 to raycast, //then somehow collide that ray with the desired z.
return enigma::Vector3(retX, retY, retZ); }
gs_scalar d3d_unproject_x(gs_scalar x, gs_scalar y) { return d3d_unproject(x,y).x; } gs_scalar d3d_unproject_y(gs_scalar x, gs_scalar y) { return d3d_unproject(x,y).y; } gs_scalar d3d_unproject_z(gs_scalar x, gs_scalar y) { return d3d_unproject(x,y).z; }
OpenGL3 BROKEN code, which I created by researching a few places explaining how gluUnproject works:
//This code does not work. Please fix it. enigma::Vector3 d3d_unproject(gs_scalar x, gs_scalar y) { GLint view[4]; GLfloat gly, glz;
oglmgr>Transformation(); glGetIntegerv(GL_VIEWPORT,view); //invert mouse y into openGL y gly = (float)view[3]  (float)y; //Read the depth of the moused pixel as the z //this stops the raycast at the first pixel collision glReadPixels(x,y,1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&glz); //Not sure but I think this is broken
//Now perform the unproject. //create an unprojection matrix by inverting the MVP. This is probably the part I screwed up. enigma::Matrix4 inv = enigma::view_matrix * enigma::projection_matrix; inv.Inverse(); //Create the normalized ray in GLspace [1,1] enigma::Vector4 tmp = enigma::Vector4((float)x / (float)view[2],gly / (float)view[3],glz,1.0f); tmp.x = tmp.x * 2.0f  1.0f; tmp.y = tmp.y * 2.0f  1.0f; tmp.z = tmp.z * 2.0f  1.0f; //Cast it enigma::Vector4 obj = inv * tmp;
//Unnormalize the result (the magnitude is 1/obj.w) return enigma::Vector3(obj.x / obj.w,obj.y / obj.w,glz / obj.w); }
