I have an OpenGL app where a player (just an AABB with an eye-height, look and move vectors), can walk along a 3D grid made of a square cells, essentially like a player walking along a Minecraft level. The cells can also have differing heights, which (barely?) qualifies it as a 3D level, rather than 2D.
On each cell, there is potential to be a static sprite, which is centered on that cell. The sprite is rendered as a "cylindrical billboard" so that it's always facing the camera, but the base/bottom of the sprite is always rooted in the ground. Like this (not my video!) https://www.youtube.com/watch?v=2-queKkSVnw
(The point I'm making is that these cylindrical billboarded sprites render a bit differently than standard sprites which would be coplanar to the view plane on the vertical axis, as well)
Here's a simple representation, where the green stick is the player, and the pink squares are the billboarded sprites which are centered on their cells.
https://i.imgur.com/MjKe7Pf.png (Note: normally, the camera view would be from the player's eyes, and thus all the pink sprites should be aligned with the player's view plane)
I also have an "arm reach" vector which is a line segment that extends outward from the crosshair a certain distance, in other words a line segment. This line segment is slightly less than the width of one cell. So, worst case scenario, then, is that this line segment is intersecting with three cells at once, like this:
My question is how should I detect that the user's arm reach ray/segment is intersecting with a vertically billboarded sprite, knowing that the sprites may have transparent pixels?
I know that the first step would be to find which cells the ray/segment is overlapping, as shown in the previous pic as the green-highlighted cells. The next step (in my case) shall be a 3D distance check from the player to the sprite location. If too far, can't click it, no matter how you rotate the camera view (and distort the sprite). After that, I'm unsure. I assume I need to somehow first check if the ray is intersecting the billboarded sprite's "rectangle" (the pink rectangle) (I say rectangle in quotes because it will be a distorted rectangle in first person view, due to perspective angle), and then pixel-pick on the sprite and look for the transparent color, but I'm not sure how to do either of those steps.
Daggerfall somehow accomplished this. Interacting with a billboarded sprite (i.e. a treasure pile) requires you to actually click on a non-transparent pixel of that sprite, or the click doesn't count.
Any sources on related techniques would be greatly appreciated. Thank you for your time!