Add support for double precision floats

[aaronfranke]

This proposal is a summary and formalization of various past discussions about double support, especially issue #288 which this proposal directly supersedes.

Describe the project you are working on:

This proposal affects any game working with large scale environments in 3D, meaning, any environment larger than a few kilometers. This proposal is especially important for games taking place in the vastness of space. The problem technically also exists in 2D, but it is far less of an issue.

Describe the problem or limitation you are having in your project:

Any 3D game in Godot with large scale environments will begin to experience jitter once the player moves more than a few kilometers away from the world origin. The problem is most noticeable in FPS games, since objects tend to be close to the camera, and jitter is more clearly visible. This is caused by the limitations of single-precision floats. There are some workarounds for some use cases, but the only proper fix is one that is done on the engine level.

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

The core issue is that single-precision floating point numbers have a limited amount of precision, which is unsuitable for games that use large scales. Single-precision floats have 23 significant binary digits (they are 32-bit, 8 of the bits are used for the exponent and 1 bit is used for positive/negative). First-person shooter games depend on the world having better than about half a millimeter of precision. The formula 0.0005 * (2^23) shows us that errors big enough to notice appear approximately a few kilometers away from the world origin.

The solution, simply put, requires us to add more significant digits. Double-precision floats are 64-bit, with 52 of those bits being significant binary digits. This is 29 more significant binary digits than single-precision floats, which increases the maximum usable area by a factor of about half a billion, to about 2 Tm (2 billion km). We go from a fifth the length of Manhattan to an area greater than the orbital radius of Saturn, more than enough for 99.99% of games. (Of course, you don't have to use all that area up to see benefit, any game larger than a few kilometers will benefit from doubles).

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:

For now, the plan is for this to be a completely optional feature which is not enabled by default, to maintain high performance on older devices. Anyone who needs double support can compile their own version of the engine from source. The rest of this section describes the details of how this will work.

C++ has a keyword called typedef that allows aliasing of types. Godot already uses this for the real_t type used for vectors and many parts of the engine. Eventually, users will be able to compile the engine with real_t being aliased to double, which means that all vector math is done with doubles, including transformations and all physics code. Pull request #21922 is a stepping stone towards double support, fixing many of the issues that currently exist when trying to compile with doubles.

On the CPU, for the most part, doubles are equally as fast as single-precision floats. The x86 architecture does not have circuits for single-precision floats, the FPU elevates all floating-point types to an 80-bit extended precision format internally, and truncates the result. Doubles take up twice the amount of memory, which can be an issue for architectures not optimized for moving around pieces of 64-bit data (such as 32-bit architectures), but otherwise the total memory usage of the engine does not change very much. There is also the matter of SIMD vector instructions, designed to perform math in parallel. Godot does not currently use these, but if it did, full acceleration would require AVX2 (256-bit for 4 * 64-bit), which means Intel CPUs from 2013 or later, and AMD CPUs from 2015 or later. A Windows 11 compatible x86 CPU will have AVX2.

It's important to note that doubles cannot be used on the graphics card. Due to Nvidia intentionally crippling support for doubles on non-Quadro graphics cards, rendering has to be done with single-precision floats. The approach used by all games that use doubles is to do all of the CPU-side math in doubles, then take all coordinates and convert them to be relative to the camera, then pass this information to the GPU. The exact details of this will be left to @reduz to deal with.

If this enhancement will not be used often, can it be worked around with a few lines of script?:

No, it cannot be worked around in a few lines of script. However, let's explore what could be done.

A fair question to ask is how other games handle large scales.

• Most games don't. It's true that this feature is only truly needed for a small amount of games, as the majority of games take place on scales smaller than a few kilometers. For games that need somewhat large scales, sometimes maps are designed around this constraint, to be square and approximately 4 kilometers in radius, such as PlanetSide 2's Indar map.

• Kerbal Space Program (KSP) is a game created in Unity, which (like most engines) uses single-precision floats. The developers of KSP had to implement their own math types, doing a huge amount of calculations in user code. Even with all their effort, KSP struggled with floating-point issues for many years, and these issues came to be known as The Kraken. The ideal solution is for the engine to have first-class support.

• A commonly cited technique is origin shifting. This involves moving the world around the player such that the player is always near the world origin. This technique can work, but it comes with many of its own limitations. For example, it doesn't always work for multiplayer, where the server needs to have precision for all players at once. There are many tricks to make this work better, but this heavily complicates things to the point that it's both easier and more efficient to use doubles.

• Some games that use doubles for large scales include Star Citizen, Arma 3, Space Engineers, and Minecraft. Star Citizen uses a custom version of Amazon Lumberyard with double support added. Arma 3 uses their own in-house engine which they call "Real Virtuality" which uses doubles. Space Engineers and Minecraft both do not use an engine, but also, Minecraft in its early days (incorrectly) truncated the coordinates, which led to issues such as the jittering seen in Far Lands or Bust (explained here).

• Unreal added support for doubles with the release of Unreal 5 to support planetary-scale games. There's also Unigine, which is focused on being an engine for simulations, and Unigine can use doubles.

Is there a reason why this should be core and not an add-on in the asset library?:

This is by nature a core engine feature, and it cannot be an add-on. However, if anyone wishes to take the limited KSP approach, I do have this repo with some math types for C#.

[Jummit]

As a workaround you can use this plugin I made:
origin_shifter.zip
Add a OriginShifter node to the node you want to keep centered.
Set world_node to the root of everything that should be shifted.

[charliewhitfield]

Thank you! This is essential for @ivoyager going forward. We do origin shifting, dynamic adjustment of camera near and far, and other tricks. As described for Kerbal Space Program, however, it becomes a real monster trying to find and patch parts that don't work due to low precision. (But anyone is welcome to come over and ask me how to do it, in any case.)

[nonunknown]

I dont know for sure if this has something to do with this issue, but in my case, working with PS1 game files, ps1 hardware has a limited float precision so levels have to be integer size, lot of variables too, so when importing to godot and trying to mimic game movements, using kinematic's move_and_slide function is a very precise float, what about a ProjectSettings that user can set which precision he wants to its game? Like double_precision: (slider from 0...X)

[aaronfranke]

@nonunknown This can't be a slider in project settings due to it being a compile-time setting.

[MarcusElg]

Has there been any progress on this issue?

[HeadClot]

@MCrafterzz There is a PR Waiting to be reviewed for 4.0. You can find it here.

[Calinou]

Since Godot 4.0 is going to focus on 64-bit devices first and foremost, why not enable double precision by default and provide a way to build the engine with single precision instead? I'm not sure about the impact on Android devices though. (I presume popular iOS devices will be powerful enough in 2021 for this not to be an issue there.)

[aaronfranke]

If the performance difference is determined to be very small, we could indeed do that, yes.

I'm most concerned about phones, since 32-bit ARM chips only work with single-precision, which would mean the performance on 32-bit ARM devices would be very slow. 64-bit ARM can do doubles fine as far as I can tell, but I haven't actually tested/benchmarked it.

[lawnjelly]

On the CPU, for the most part, doubles are equally as fast as single-precision floats. The x86 architecture does not have circuits for single-precision floats, the FPU elevates all floating-point types to an 80-bit extended precision format internally, and truncates the result.

This is a bit misleading. Compilers on platforms that have minimum support for SIMD will preferentially use SIMD instructions to replace ye olde style floating point calculations. Try compiling a simple program with optimization and examine the assembly.

In practice equally or more relevant will be the extra memory, because cache hits / memory bandwidth has become more an more a bottleneck in modern CPUs. Sometimes dealing with twice the memory can simply take twice as long.

The speed of the operation / memory access is why even modern GPUs will still use 32 bit operations on 64 bit systems. Going further, mobile GPUs will use lower precision than 32 bit for the same reasons, as low as 10 bits.

That said I'm not against having a float / double compilation switch, especially if it is easy to do, it might be wise to temper expectations in terms of the relative performance of the two methods. Of course the real world effect depends on how much floating point calculations are bottlenecks in any particular game. You may find that in practice that only say 5% of CPU time is spent in floating point calculations, in which case a doubling of their time taken will only result in a 5% drop in performance.

For many reasons (including this) engines often use alternative approaches such as shifting the world origin:
https://docs.unrealengine.com/en-US/Engine/LevelStreaming/WorldBrowser/index.html

This can include approaches such as splitting the world into chunks (often addressed by integers) and using floats to reference the local area within these chunks.

Another approach is the use of fixed point. But these alternatives are considerably more involved that flipping from float to double.

Also note that aside from flipping float to double there may be some non-obvious complications, such as the values used for epsilons, and alignment, and files.

[aaronfranke]

I did mention SIMD in my post, SIMD instructions for processing whole Vector3 or Quats at a time need 256-bit instructions, which means Intel CPUs from 2013 or later, and AMD CPUs from 2015 or later. See also #290. Note that I'm not really concerned about cases of performing the same operation to more than one vector at once, since I don't think such cases are common in transform/physics math (though extremely common in your test case of arrays). By the way, your example output doesn't match the example code ("timing SIMD" vs "timing ranged").

some non-obvious complications, such as the values used for epsilons, and alignment.

Single-precision epsilons can still work for doubles, only breaking in the extreme cases. Alignment changing (and therefore the entire ABI) is expected.

[Ophiolith]

Hi all, seeing as this thread seems to be not entirely technically focused, I hope this comment isn't out of the scope of discussion for proposals. If it is, please do delete and I will raise it somewhere more appropriate. As proposals seem to be guided by user interest, I figured it might worthwhile talking about how as a hobbyist gamedev, I'd find the inclusion of double-precision floats beneficial, or at the very least appealing.

A lot of discussion on the issue centres around how any projects that would need it would inevitably be big open-world games or simulations, both of which are outwith the scope of indie developers; and that any such projects would be better off using world segmentation and co-ordinate shifting. While I understand both of these arguments, I feel that they somewhat miss the point of why double-precision floats would be beneficial.

Firstly, the assumption that large open worlds cannot be accomplished by indie devs is not true. There are already examples of extremely large environments made by small teams on the market that have been successful - For instance Kenshi, Outward, Astroneer, Space Engineers and the aforementioned Kerbal Space Program. While double-precision floats are not going to resolve the inherent logistical issues in making large open worlds, not having to use custom code or workarounds to achieve them is only going to be of benefit.

Secondly, simply not having to worry about world size, co-ordinate precision, jitter, z-fighting, or other related issues are benefits that aren't limited to open-world titles. Such issues can happen even in smaller environments, and giving developers the option to eliminate them, at cost to performance, may be worth the tradeoff.

My personal interest does not so much come from wanting to create huge, sprawling, AAA open world sandboxes, but simply being able to make large environments if I choose to do so without the added complications of having to endure precision falloff, or having to roll my own means to mitigate the issues this introduces. Seeing @reduz 's recent work towards scalable realtime global illumination for big open-world environments suggests there's at least developer intent to make Godot suitable for such games. As Godot focuses on solutions that are easy to manage by the end-user first and performance second, double-precision floats seem like a reasonable fit for this philosophy, provided it doesn't preclude using single-precision if need be.

[Calinou]

@Ophiolith We will definitely support double-precision floats in 4.0, but it will most likely be a compile-time option to avoid performance issues on slower hardware (especially mobile/Web platforms).

[MarcusElg]

@Ophiolith We will definitely support double-precision floats in 4.0, but it will most likely be a compile-time option to avoid performance issues on slower hardware (especially mobile/Web platforms).

Hmm this confuses me because looking at the pr it doesn't seam like reduz really want double precision floats. Also how would the compile time option work?

[Calinou]

Hmm this confuses me because looking at the pr it doesn't seam like reduz really want double precision floats.

He told me on IRC he's interested in having Godot support double-precision floats, but not by default for performance reasons.

Also how would the compile time option work?

As far as I know, it's more or less a matter of fixing REAL_T_IS_DOUBLE so it can be used reliably. This define already exists but it currently doesn't work as intended.

[MarcusElg]

Hmm this confuses me because looking at the pr it doesn't seam like reduz really want double precision floats.

He told me on IRC he's interested in having Godot support double-precision floats, but not by default for performance reasons.

Also how would the compile time option work?

As far as I know, it's more or less a matter of fixing REAL_T_IS_DOUBLE so it can be used reliably. This define already exists but it currently doesn't work as intended.

Ok so there won't be a option in project preferneces/settings? You would have to enable it from code?

[Calinou]

@MCrafterzz You would have to recompile the editor and project templates for this, as it's not technically possible to swap C++ types without recompiling the engine.

Since it's a relatively advanced use case, I don't think this will be too much of an issue. Still, nothing prevents a third party from distributing pre-built binaries with double precision support.

[MarcusElg]

@MCrafterzz You would have to recompile the editor and project templates for this, as it's not technically possible to swap C++ types without recompiling the engine.

Since it's a relatively advanced use case, I don't think this will be too much of an issue. Still, nothing prevents a third party from distributing pre-built binaries with double precision support.

Ok thanks for answering my questions. As long as it's clearly documented it shouldn't be a problem :D

[Ophiolith]

@Calinou Thank you for the response! I wasn't aware there was already an active push for a compile time option for this as part of 4.0. I'll be sure to test and give feedback when it's up and running :)

[mrjustaguy]

Wouldn't it be easier for the user to implement sth like example Vector3-double and Vector3-single and the like and have an option in project settings that is default at single? I don't know much about the inner workings of the engine so not sure if it is possible to make something like that and get the benefits of both worlds.. I mean my idea is to like have 2 classes with the same name just from 2 different modules, only the engine would load one module or the other depending on the setting.

[aaronfranke]

@mrjustaguy As stated above, it's not technically possible to swap C++ types without recompiling the engine.

[Megalomaniak]

@Calinou said:

Since it's a relatively advanced use case, I don't think this will be too much of an issue. Still, nothing prevents a third party from distributing pre-built binaries with double precision support.

Or similar to how currently you can choose between mono build of the engine or leaner gdscript only engine. Though I'd expect this to only happen if the Mono build becomes the official build and the other one is dropped so that there would be capacity to maintain an extra build. That is just speculation on my part though.

[Calinou]

Though I'd expect this to only happen if the Mono build becomes the official build and the other one is dropped so that there would be capacity to maintain an extra build.

Non-Mono builds are here to stay, they're smaller and don't require any system dependencies to be used (especially on Windows) 🙂

[aaronfranke]

@Megalomaniak Well, the build matrix isn't as big as you'd think, because there isn't much of a point of making 32-bit builds with doubles. So on Windows there would be 6 builds: 32-bit Single, 32-bit Single Mono, 64-bit Single, 64-bit Single Mono, 64-bit Double, 64-bit Double Mono, and on Mac/Linux there would be 4: Single, Single Mono, Double, Double Mono (since Mac/Linux official builds will be 64-bit-only for 4.0).

EDIT: It has since been pointed out to me that we will need to support 32-bit with doubles on WebAssembly if we want doubles on WebAssembly because there is no 64-bit WebAssembly.

EDIT: Mac builds would have 4 binaries but 8 build configurations if you count both Intel x86 and Apple Silicon ARM (Single x86, Single Mono x86, Double x86, Double Mono x86, Single ARM, Single Mono ARM, Double ARM, Double Mono ARM). If ARM Linux or ARM Windows ever take off then this would increase the combinations there too.

[Megalomaniak]

Right, but I meant pre-built double precision builds which far as I can tell from this issue/topic aren't going to be a thing? Just the support to build yourself with a compile time flag. I suppose there is always others building for those who don't want to deal with it themselves though.

[Ophiolith]

I think I'd honestly be okay with double-precision being relegated to a compile-time flag. It seems to be something the vast majority of indie developers would not have much use for, and may just end up with a lot of users downloading it by mistake and wondering why their memory footprint is insane and their game not being able to run on mobile devices.

[aaronfranke]

@Megalomaniak I'm not ruling out official double precision builds, but we'll cross that bridge when we come to it (and if the demand is high enough to make it worth it).

@Ophiolith It wouldn't actually increase the memory footprint very much, because only a small fraction of memory is used for floats. A lot of the memory is used for other things such as textures and other assets. The main concern is with reduced performance on old devices, especially 32-bit devices, but also older 64-bit devices without good vector instruction sets.

[mrjustaguy]

I can see use cases of double precision in indie games, but yea there aren't plenty.. Mainly for those that use procedural generation to make gigantic maps, and well, space games could benefit from it too in some cases..

[Gnollrunner]

I seemed to have been following this thread and forgot about it. Since it just popped up, here is my two cents. First I've never even used a game engine so I probably don't know all the ins and outs of this. However I have programmed in C++ with DirectX using double to do planet sized stuff. I would rather use a game engine but there is nothing free-ish that really supports this which is a bit surprising to me.

I think it should be very doable. I do a few things in my code which seems to work pretty well. First go strait to view space. Since you don't have double on the GPU (or not to any great degree) I think this works best. Once you go to world space with float you lose all your precision. If you go directly to view space you only lose it away from the camera where it doesn't matter. I generate World-View matrices in double on the CPU and then truncate to float before sending them down to the GPU.

Second you of course need a very good LOD system. However I suppose this might be considered part of the application instead of the engine. In my code I'm using voxels and marching cubes with LOD transitions, which happens at run time, so it takes care of that, at least for terrain, but this is kind of specific. In any case you need to do something to prevent Z fighting.

Finally I take the projection stuff out of the matrix and do that in a post step. I found not doing this causes major instability. I basically scale everything down by a large power of 2 to get Z inside the box required by DirectX. The idea here is that it will only change the exponent bits on Z and not the precision bits going from view space to projection space. This seemed to fix a few issues.

These are just some ideas. Maybe this is all obvious but I thought I'd throw it out there since it works OK. So anyway if you guys decide to do this, I'll probably switch to Godot and hopefully contribute something.

[albinaask]

@Gnollrunner I think your projection trick may also be solved by the use of logarithmic depth buffers

[fire]

For adjacent work with voxel and level streaming. Zylann's work is notable. https://github.com/Zylann/godot_voxel/tree/godot4 https://github.com/Zylann/solar_system_demo

Here's Zylann's Solar System Demo.

[isral]

@fire
Is Godot v4.0 alpha1 already support double precision?

[fire]

Let me ask @aaronfranke if he has the exact syntax.

[Calinou]

Is Godot v4.0 alpha1 already support double precision?

There is some limited support but it's not complete or production-ready yet.

[aaronfranke]

Here's an update on the status of this proposal:

I have not spent much time working on double support recently, but Zylann has been doing some great work recently fixing the most critical bugs such as binary resources not loading, the remote inspector not working, and the is_equal_approx tolerance being incorrect. You can view some of his double support PRs here.

Here's a video Zylann made that shows the current status (it includes merging the one PR linked above that hasn't been merged yet): https://www.youtube.com/watch?v=g5wwa5W5_Cw

It appears that mostly everything except rendering is working. From watching the cubes fall and interact we can see that they are moving at a smooth rate, so the physics system seems to be correctly using doubles.

The next step is that we need to have double support in rendering. Since using doubles on the GPU is not an option, this means we need a camera-relative rendering system to process positions relative to the camera on the CPU before we pass this information to the GPU. I'd like to invite @Gnollrunner @clayjohn @lawnjelly and @reduz to take a look at this as these people all have rendering experience and are interested in double support.

Aside from that, there are still some non-rendering bugs to fix that we know about, in particular Zylann's PR about fixing is_equal_approx tolerance breaks some of the unit tests. So progress is not completely blocked by rendering.

In its current state, the float=64 option is working well enough that users who want large world support may be better off using it compared with float=32, even though there are still issues and limitations.

[HeadClot]

Bit of a question for you @aaronfranke you mentioned a "camera-relative rendering system" would this system work with VR and AR as well? Just curious.

[aaronfranke]

@HeadClot Yes. To be clear, the system just needs to adjust the camera's coordinates as seen by the GPU enough that they are within the limits of 32-bit floats there. Unless someone's eyes are 10km apart, it should 100% be doable for XR.

EDIT: To clarify further, this means that in games with extra cameras and viewports, it's not feasible to have two cameras on opposite sides of a planet sized object with precise rendering. AFAIK.

[HeadClot]

@HeadClot Yes. To be clear, the system just needs to adjust the camera's coordinates as seen by the GPU enough that they are within the limits of 32-bit floats there. Unless someone's eyes are 10km apart, it should 100% be doable for XR.

Good to hear :)

[albinaask]

Awesome work!! to be clear @aaronfranke , is the float=64 something that is passed as a scons argument, a project setting or a #define thing?

[aaronfranke]

@albinaask It's a scons argument for compiling the engine.

If you're interested in seeing how it works, this argument sets a define called REAL_T_IS_DOUBLE which you can search in the codebase for.

[fire]

godotengine/godot#58516 is related.

[3top1a]

Just a thought - is float=16, float=128 or even float=256 possible?

[Calinou]

Just a thought - is float=16, float=128 or even float=256 possible?

No, only 32 and 64 are allowed. C++ does not have primitive 16-bit, 128-bit or 256-bit float types.

[aaronfranke]

@3top1a C++ does have a few additional types available to us, float_t, double_t, and long double. None are fixed sizes across platforms. The first two just mean "at least as big as float/double but can be bigger if the hardware prefers that", which isn't very useful to us if we're looking for a specific size.

The interesting one is long double. The type long double must be at least as big as double, but it is usually bigger. On x86 systems, this is an 80-bit type representing the underlying extended precision type, on RISC-V this uses the Q extension for 128-bit, and on ARM it would be 64-bit. Most compilers will respect this except for MSVC which just always treats it the same as double (so you need MinGW to properly use long double on Windows). This could be implemented as float=80, but it would actually be 128-bit on RISC-V (with Q, else 64-bit) and 64-bit on ARM. Even so, 80-bit isn't a huge improvement over 64-bit (you go from a few trillion meters to a few quadrillion meters), so there's not much point.

There is also __float128, which isn't available in the C++ language, but it exists in GCC. It will always be 128-bit, but on platforms without 128-bit support it will use emulation, so it would be extremely slow.

[Gnollrunner]

EDIT: To clarify further, this means that in games with extra cameras and viewports, it's not feasible to have two cameras on opposite sides of a planet sized object with precise rendering. AFAIK.

I think from a low level graphics API standpoint there is no reason why this too shouldn't work. I guess it depends on how everything is set up and I'm certainly not an game engine expert. The way I'm doing it you set transformations for every object on every frame. I write matrices straight to a upload buffer sequentially and just kind of scroll though them as you render your objects. This may be an issue if you have a huge number of different objects, but with say 5K meshes it hasn't been so far and I'm using a somewhat old computer and graphics card.

[aaronfranke]

For discussion about resolving the rendering issues with double precision, check out this issue: godotengine/godot#58516

[akien-mga]

I think the bulk of this proposal has been pretty much implemented by now, though there are still issues left to resolve.

To keep a good overview over what's left to do, I would suggest opening bugs reports, and if relevant more detailed feature proposals, which can both be targeted at 4.0. WDYT?

[YuriSizov]

I'll close this per @akien-mga's comment above. Feel free to open improvement proposals on top of the implemented feature and submit bug reports.