ENIGMA Forums

Contributing to ENIGMA => Function Peer Review => Topic started by: Solitudinal on November 07, 2017, 03:27:49 pm

Title: d3d_unproject
Post by: Solitudinal on November 07, 2017, 03:27:49 pm
This handy function set allows you to convert a 2d screen coordinate (such as the mouse position) into 3d coordinates - basically the 3d coordinates that it's pointing at.

This is sometimes called unproject, because it reverses the projection process (which converts 3d coordinates into 2d screen coordinates). It's also sometimes called raycasting, since you are effectively casting a geometric ray from the camera/viewport into 3d space, and then in this case colliding it with the z-depth of whatever pixel it is (e.g. z=0 if you click on the floor).

4 functions are provided:
d3d_unproject returns a Vector3, if you are ok with OO programming.
d3d_unproject_x/y/z returns the respective x, y, and z coordinates, in GM's regular functional programming (where functions return a single number, rather than an object).
All functions take an x and a y coordinate, which refer to the location respective to the viewport. I.e. d3d_unroject(mouse_x,mouse_y)
Or if you're weird and actually use views in conjunction with 3d:
d3d_unproject(mouse_x - view_xview[view_current], mouse_y - view_yview[view_current])

These functions are a little expensive due to complex math going on behind the scenes, so try not to use them more than a few times every step. I.e. if you're using them for mouse coordinates, consider storing the results in a global variable.

NOTE this only works in OpenGL1, as it depends on a call to gluUnproject. I tried porting the code to OpenGL3, but I'm terrible with matrices and ENIGMA uses an awkward matrix format. I've included my attempted OpenGL3 code below if you want to try and fix it.

Code: [Select]
enigma::Vector3 d3d_unproject(gs_scalar x, gs_scalar y)
{
  GLdouble model[16];
  GLdouble proj[16];
  GLint view[4];
  GLdouble retX, retY, retZ;
  GLfloat gly, glz;

  glGetDoublev(GL_MODELVIEW_MATRIX,model);
  glGetDoublev(GL_PROJECTION_MATRIX,proj);
  glGetIntegerv(GL_VIEWPORT,view);

  //invert mouse y into openGL y
  gly = (float)view[3] - (float)y;
  //Read the depth of the moused pixel as the z
  //this stops the raycast at the first pixel collision
  glReadPixels(x,int(gly),1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&glz);

  gluUnProject(x,gly,glz,model,proj,view,&retX,&retY,&retZ);
  //TODO: Provide a raycast_to_floor, stopping at a desired z rather than the first pixel collision
  //I believe the way to do this is to call gluUnProject twice at glz=0 and glz=1 to raycast,
  //then somehow collide that ray with the desired z.

  return enigma::Vector3(retX, retY, retZ);
}

gs_scalar d3d_unproject_x(gs_scalar x, gs_scalar y)
{
  return d3d_unproject(x,y).x;
}
gs_scalar d3d_unproject_y(gs_scalar x, gs_scalar y)
{
  return d3d_unproject(x,y).y;
}
gs_scalar d3d_unproject_z(gs_scalar x, gs_scalar y)
{
  return d3d_unproject(x,y).z;
}


OpenGL3 BROKEN code, which I created by researching a few places explaining how gluUnproject works:
Code: [Select]
//This code does not work. Please fix it.
enigma::Vector3 d3d_unproject(gs_scalar x, gs_scalar y)
{
  GLint view[4];
  GLfloat gly, glz;

  oglmgr->Transformation();
  glGetIntegerv(GL_VIEWPORT,view);
  //invert mouse y into openGL y
  gly = (float)view[3] - (float)y;
  //Read the depth of the moused pixel as the z
  //this stops the raycast at the first pixel collision
  glReadPixels(x,y,1,1,GL_DEPTH_COMPONENT,GL_FLOAT,&glz); //Not sure but I think this is broken

  //Now perform the unproject.
  //create an unprojection matrix by inverting the MVP. This is probably the part I screwed up.
  enigma::Matrix4 inv = enigma::view_matrix * enigma::projection_matrix;
  inv.Inverse();
  //Create the normalized ray in GL-space [-1,1]
  enigma::Vector4 tmp = enigma::Vector4((float)x / (float)view[2],gly / (float)view[3],glz,-1.0f);
  tmp.x = tmp.x * 2.0f - 1.0f;
  tmp.y = tmp.y * 2.0f - 1.0f;
  tmp.z = tmp.z * 2.0f - 1.0f;
  //Cast it
  enigma::Vector4 obj = inv * tmp;

  //Un-normalize the result (the magnitude is 1/obj.w)
  return enigma::Vector3(obj.x / obj.w,obj.y / obj.w,glz / obj.w);
}