intersect_ray() result normal can be inverted or length 0.

[ghost]

Godot version:
0edcb8e

OS/device including version:
Win10 64-bit

Issue description:
intersect_ray can have unpredictable normal return values when the origin of the ray starts inside of geometry it can collide with.

https://docs.godotengine.org/en/3.2/classes/class_physicsdirectspacestate.html#class-physicsdirectspacestate-method-intersect-ray

Some cases it points to a random edge, other times completely inverted, and on occasion it might return Vector2(0, 0).

It would be nice if it were more robust, or at least didn't return results at all.

Steps to reproduce:
Cast a ray with it's starting point inside the collision geometry.

Minimal reproduction project:
Intersect Ray.zip

[Xrayez]

I confirm this to be somewhat a problem in my project too, but I'm not sure if this is by design. It might be computationally expensive to check whether a point lies inside those shapes.

Casting a continuous rays which reflects against colliders works:

But when a ray starts inside the collider, it's basically stuck:

There are also somewhat rare occasions where a ray gets stuck like that even if it wasn't initially "emitted" inside a collider, but I think that happens because I do some hacks to avoid continuous detection (see also #35344 (comment)):

# Minimize the chance of collider penetration
from = raycast.position + dir * safe_margin

which might lead to a situation where the next ray might indeed start inside a collider and not outside.

The same behavior persists with RayCast2D node.

I guess the workaround is to check whether there are any colliders at position by calling intersect_point method.

on occasion it might return Vector2(0, 0)

I think that happens when you cast a ray just exactly between internal adjacent convex shapes.

[ghost]

It's unexpected, and actually a bit difficult and limiting to work around this.

The wrinkle I've had is with whiskers on enemy steering. If they're too far outside their collision shape, they're likely to end up in a wall during a twist and turn.

In this example the goal is ahead, but a backwards contact with a wall would be fine to help create a separation force, but here it becomes an attractor, since the normal (white arrow) is inverted.

[ghost]

@pouleyKetchoupp This one may or may not be an issue, but if not, maybe it's still worth considering having some clarity or well defined behavior for what's supposed to happen. Only because sometimes one can get usable results, and other times not. It can mislead one to first think the code is working. In my case, I did some tests on some geometry that didn't quite capture what would later appear in the game. Initially it looked good, and for many weeks things appeared to be working, and I went on a bad development path.

The summary:

What's supposed to happen with rays (or even shapes) that have points originating inside geometry?

And what of compound geometries like many concave polygons created to represent a single convex polygon?

If things need to get ejected from a collection of shapes acting as one, the developer is probably only interested in the shared border. If it were a wall, it wouldn't be desirable to have something juggle around inside of the internal edges.

If convex polygons don't get treated as a whole in certain cases, it would be a very nice thing for the API to provide information somehow to figure out a solution in the game logic.