Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Programmable camera distortion #3039

Merged
merged 15 commits into from
Dec 17, 2020
Merged

Programmable camera distortion #3039

Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
4fa2107
API for distortion parameters
saihv Sep 24, 2020
08ee44c
Initial version of camera distortion material and API
saihv Sep 25, 2020
cf8d7fb
Update PythonClient, remove distortion postprocess at EndPlay()
saihv Sep 25, 2020
42f5aec
Remove irrelevant files
saihv Sep 25, 2020
d736207
Refactor - move distortion params to PIPCamera
saihv Sep 25, 2020
4aaa6e3
Remove unneeded includes
saihv Sep 25, 2020
eab194e
Fix includes
saihv Sep 25, 2020
db5e99e
Add missing material instancing, API to retrieve distortion params
saihv Oct 3, 2020
ed31286
Fix docstring
saihv Oct 3, 2020
e41becf
Revert change for compatibility
saihv Oct 7, 2020
e5967a0
Add placeholders for Unity compilation
saihv Dec 9, 2020
57f458e
Revert irrelevant changes
saihv Dec 9, 2020
5b3e1cd
Fix typo
saihv Dec 10, 2020
ded679f
Fix returntype Unity
saihv Dec 11, 2020
2667c3a
Fix unimpl Unity methods
saihv Dec 15, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion AirLib/include/api/RpcLibClientBase.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,14 +103,14 @@ class RpcLibClientBase {
CollisionInfo simGetCollisionInfo(const std::string& vehicle_name = "") const;

CameraInfo simGetCameraInfo(const std::string& camera_name, const std::string& vehicle_name = "") const;
void simSetDistortionParam(const std::string& camera_name, const std::string& param_name, float value, const std::string& vehicle_name = "");
void simSetCameraPose(const std::string& camera_name, const Pose& pose, const std::string& vehicle_name = "");
void simSetCameraFov(const std::string& camera_name, float fov_degrees, const std::string& vehicle_name = "");
// This is a backwards-compatibility wrapper over simSetCameraPose, and can be removed in future major releases
void simSetCameraOrientation(const std::string& camera_name, const Quaternionr& orientation, const std::string& vehicle_name = "");

msr::airlib::Kinematics::State simGetGroundTruthKinematics(const std::string& vehicle_name = "") const;
msr::airlib::Environment::State simGetGroundTruthEnvironment(const std::string& vehicle_name = "") const;

std::vector<std::string> simSwapTextures(const std::string& tags, int tex_id = 0, int component_id = 0, int material_id = 0);

// Recording APIs
Expand Down
1 change: 1 addition & 0 deletions AirLib/include/api/VehicleSimApiBase.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,7 @@ class VehicleSimApiBase : public msr::airlib::UpdatableObject {
virtual CameraInfo getCameraInfo(const std::string& camera_name) const = 0;
virtual void setCameraPose(const std::string& camera_name, const Pose& pose) = 0;
virtual void setCameraFoV(const std::string& camera_name, float fov_degrees) = 0;
virtual void setDistortionParam(const std::string& camera_name, const std::string& param_name, float value) = 0;

virtual CollisionInfo getCollisionInfo() const = 0;
virtual int getRemoteControlID() const = 0; //which RC to use, 0 is first one, -1 means disable RC (use keyborad)
Expand Down
1 change: 0 additions & 1 deletion AirLib/include/api/WorldSimApiBase.hpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,6 @@ class WorldSimApiBase {
virtual bool setObjectPose(const std::string& object_name, const Pose& pose, bool teleport) = 0;
virtual bool runConsoleCommand(const std::string& command) = 0;
virtual bool setObjectScale(const std::string& object_name, const Vector3r& scale) = 0;

virtual std::unique_ptr<std::vector<std::string>> swapTextures(const std::string& tag, int tex_id = 0, int component_id = 0, int material_id = 0) = 0;
virtual vector<MeshPositionVertexBuffersResponse> getMeshPositionVertexBuffers() const = 0;

Expand Down
6 changes: 6 additions & 0 deletions AirLib/src/api/RpcLibClientBase.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,12 @@ void RpcLibClientBase::simSetCameraFov(const std::string& camera_name, float fov
pimpl_->client.call("simSetCameraFov", camera_name, fov_degrees, vehicle_name);
}

void RpcLibClientBase::simSetDistortionParam(const std::string& camera_name, const std::string& param_name, float value, const std::string& vehicle_name)
{
pimpl_->client.call("simSetDistortionParam", camera_name, param_name, value, vehicle_name);
}


msr::airlib::Kinematics::State RpcLibClientBase::simGetGroundTruthKinematics(const std::string& vehicle_name) const
{
return pimpl_->client.call("simGetGroundTruthKinematics", vehicle_name).as<RpcLibAdapatorsBase::KinematicsState>().to();
Expand Down
4 changes: 4 additions & 0 deletions AirLib/src/api/RpcLibServerBase.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -243,6 +243,10 @@ RpcLibServerBase::RpcLibServerBase(ApiProvider* api_provider, const std::string&
return RpcLibAdapatorsBase::CameraInfo(camera_info);
});

pimpl_->server.bind("simSetDistortionParam", [&](const std::string& camera_name, const std::string& param_name, float value, const std::string& vehicle_name) -> void {
getVehicleSimApi(vehicle_name)->setDistortionParam(camera_name, param_name, value);
});

pimpl_->server.bind("simSetCameraPose", [&](const std::string& camera_name, const RpcLibAdapatorsBase::Pose& pose,
const std::string& vehicle_name) -> void {
getVehicleSimApi(vehicle_name)->setCameraPose(camera_name, pose.to());
Expand Down
26 changes: 26 additions & 0 deletions PythonClient/airsim/client.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -482,6 +482,32 @@ def simGetCameraInfo(self, camera_name, vehicle_name = ''):
# TODO: below str() conversion is only needed for legacy reason and should be removed in future
return CameraInfo.from_msgpack(self.client.call('simGetCameraInfo', str(camera_name), vehicle_name))

def simSetDistortionParams(self, camera_name, distortion_params, vehicle_name = ''):
"""
Set camera distortion parameters

Args:
camera_name (str): Name of the camera, for backwards compatibility, ID numbers such as 0,1,etc. can also be used
distortion_params (dict): Dictionary of distortion param names and corresponding values
{"K1": 0.0, "K2": 0.0, "K3": 0.0, "P1": 0.0, "P2": 0.0}
vehicle_name (str, optional): Vehicle which the camera is associated with
"""

for param_name, value in distortion_params.items():
self.client.call('simSetDistortionParam', param_name, value, vehicle_name)

def simSetDistortionParam(self, camera_name, param_name, value, vehicle_name = ''):
"""
Set single camera distortion parameter

Args:
camera_name (str): Name of the camera, for backwards compatibility, ID numbers such as 0,1,etc. can also be used
param_name (str): Name of distortion parameter
value (float): Value of distortion parameter
vehicle_name (str, optional): Vehicle which the camera is associated with
"""
self.client.call('simSetDistortionParam', str(camera_name), param_name, value, vehicle_name)

def simSetCameraPose(self, camera_name, pose, vehicle_name = ''):
"""
- Control the pose of a selected camera
Expand Down
Binary file added Unreal/Plugins/AirSim/Content/HUDAssets/CameraDistortion.uasset
View file
Open in desktop
Binary file not shown.
Binary file added Unreal/Plugins/AirSim/Content/HUDAssets/DistortionParams.uasset
View file
Open in desktop
Binary file not shown.
71 changes: 49 additions & 22 deletions Unreal/Plugins/AirSim/Source/PIPCamera.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,6 @@
#include "Camera/CameraComponent.h"
#include "Engine/TextureRenderTarget2D.h"
#include "Engine/World.h"
#include "Materials/MaterialInstanceDynamic.h"
#include "ImageUtils.h"

#include <string>
Expand All @@ -23,6 +22,15 @@ APIPCamera::APIPCamera()
UAirBlueprintLib::LogMessageString("Cannot create noise material for the PIPCamera",
"", LogDebugLevel::Failure);

static ConstructorHelpers::FObjectFinder<UMaterial> dist_mat_finder(TEXT("Material'/AirSim/HUDAssets/CameraDistortion.CameraDistortion'"));
if (dist_mat_finder.Succeeded())
{
distortion_material_static_ = dist_mat_finder.Object;
}
else
UAirBlueprintLib::LogMessageString("Cannot create distortion material for the PIPCamera",
"", LogDebugLevel::Failure);

PrimaryActorTick.bCanEverTick = true;

image_type_to_pixel_format_map_.Add(0, EPixelFormat::PF_B8G8R8A8);
Expand Down Expand Up @@ -66,6 +74,7 @@ void APIPCamera::BeginPlay()
Super::BeginPlay();

noise_materials_.AddZeroed(imageTypeCount() + 1);
distortion_materials_.AddZeroed(imageTypeCount() + 1);

//by default all image types are disabled
camera_type_enabled_.assign(imageTypeCount(), false);
Expand Down Expand Up @@ -208,6 +217,18 @@ void APIPCamera::EndPlay(const EEndPlayReason::Type EndPlayReason)
noise_material_static_ = nullptr;
noise_materials_.Empty();

if (distortion_materials_.Num()) {
for (unsigned int image_type = 0; image_type < imageTypeCount(); ++image_type) {
if (distortion_materials_[image_type + 1])
captures_[image_type]->PostProcessSettings.RemoveBlendable(distortion_materials_[image_type + 1]);
}
if (distortion_materials_[0])
camera_->PostProcessSettings.RemoveBlendable(distortion_materials_[0]);
}

distortion_material_static_ = nullptr;
distortion_materials_.Empty();

for (unsigned int image_type = 0; image_type < imageTypeCount(); ++image_type) {
//use final color for all calculations
captures_[image_type] = nullptr;
Expand Down Expand Up @@ -312,12 +333,12 @@ void APIPCamera::setupCameraFromSettings(const APIPCamera::CameraSetting& camera
false);
break;
}

setDistortionMaterial(image_type, captures_[image_type], captures_[image_type]->PostProcessSettings);
setNoiseMaterial(image_type, captures_[image_type], captures_[image_type]->PostProcessSettings, noise_setting);
}
else { //camera component
updateCameraSetting(camera_, capture_setting, ned_transform);

setDistortionMaterial(image_type, camera_, camera_->PostProcessSettings);
setNoiseMaterial(image_type, camera_, camera_->PostProcessSettings, noise_setting);
}
}
Expand Down Expand Up @@ -423,30 +444,36 @@ void APIPCamera::updateCameraPostProcessingSetting(FPostProcessSettings& obj, co
}
}

void APIPCamera::setDistortionMaterial(int image_type, UObject* outer, FPostProcessSettings& obj)
{
UMaterialInstanceDynamic* distortion_material = UMaterialInstanceDynamic::Create(distortion_material_static_, outer);
distortion_materials_[image_type + 1] = distortion_material;
obj.AddBlendable(distortion_material, 1.0f);
}

void APIPCamera::setNoiseMaterial(int image_type, UObject* outer, FPostProcessSettings& obj, const NoiseSetting& settings)
{
if (!settings.Enabled)
return;

UMaterialInstanceDynamic* noise_material_ = UMaterialInstanceDynamic::Create(noise_material_static_, outer);
noise_materials_[image_type + 1] = noise_material_;


noise_material_->SetScalarParameterValue("HorzWaveStrength", settings.HorzWaveStrength);
noise_material_->SetScalarParameterValue("RandSpeed", settings.RandSpeed);
noise_material_->SetScalarParameterValue("RandSize", settings.RandSize);
noise_material_->SetScalarParameterValue("RandDensity", settings.RandDensity);
noise_material_->SetScalarParameterValue("RandContrib", settings.RandContrib);
noise_material_->SetScalarParameterValue("HorzWaveContrib", settings.HorzWaveContrib);
noise_material_->SetScalarParameterValue("HorzWaveVertSize", settings.HorzWaveVertSize);
noise_material_->SetScalarParameterValue("HorzWaveScreenSize", settings.HorzWaveScreenSize);
noise_material_->SetScalarParameterValue("HorzNoiseLinesContrib", settings.HorzNoiseLinesContrib);
noise_material_->SetScalarParameterValue("HorzNoiseLinesDensityY", settings.HorzNoiseLinesDensityY);
noise_material_->SetScalarParameterValue("HorzNoiseLinesDensityXY", settings.HorzNoiseLinesDensityXY);
noise_material_->SetScalarParameterValue("HorzDistortionStrength", settings.HorzDistortionStrength);
noise_material_->SetScalarParameterValue("HorzDistortionContrib", settings.HorzDistortionContrib);

obj.AddBlendable(noise_material_, 1.0f);
UMaterialInstanceDynamic* noise_material = UMaterialInstanceDynamic::Create(noise_material_static_, outer);
noise_materials_[image_type + 1] = noise_material;

noise_material->SetScalarParameterValue("HorzWaveStrength", settings.HorzWaveStrength);
noise_material->SetScalarParameterValue("RandSpeed", settings.RandSpeed);
noise_material->SetScalarParameterValue("RandSize", settings.RandSize);
noise_material->SetScalarParameterValue("RandDensity", settings.RandDensity);
noise_material->SetScalarParameterValue("RandContrib", settings.RandContrib);
noise_material->SetScalarParameterValue("HorzWaveContrib", settings.HorzWaveContrib);
noise_material->SetScalarParameterValue("HorzWaveVertSize", settings.HorzWaveVertSize);
noise_material->SetScalarParameterValue("HorzWaveScreenSize", settings.HorzWaveScreenSize);
noise_material->SetScalarParameterValue("HorzNoiseLinesContrib", settings.HorzNoiseLinesContrib);
noise_material->SetScalarParameterValue("HorzNoiseLinesDensityY", settings.HorzNoiseLinesDensityY);
noise_material->SetScalarParameterValue("HorzNoiseLinesDensityXY", settings.HorzNoiseLinesDensityXY);
noise_material->SetScalarParameterValue("HorzDistortionStrength", settings.HorzDistortionStrength);
noise_material->SetScalarParameterValue("HorzDistortionContrib", settings.HorzDistortionContrib);

obj.AddBlendable(noise_material, 1.0f);
}

void APIPCamera::enableCaptureComponent(const APIPCamera::ImageType type, bool is_enabled)
Expand Down
8 changes: 8 additions & 0 deletions Unreal/Plugins/AirSim/Source/PIPCamera.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@
#include "common/AirSimSettings.hpp"
#include "NedTransform.h"

#include "Materials/MaterialParameterCollectionInstance.h"
#include "Materials/MaterialInstanceDynamic.h"
#include "PIPCamera.generated.h"


Expand Down Expand Up @@ -51,6 +53,9 @@ class AIRSIM_API APIPCamera : public ACameraActor
UTextureRenderTarget2D* getRenderTarget(const ImageType type, bool if_active);

msr::airlib::Pose getPose() const;

UPROPERTY() UMaterialParameterCollection* distortion_param_collection_;
UPROPERTY() UMaterialParameterCollectionInstance* distortion_param_instance_;

private: //members
UPROPERTY() TArray<USceneCaptureComponent2D*> captures_;
Expand All @@ -60,7 +65,9 @@ class AIRSIM_API APIPCamera : public ACameraActor
//TMap<int, UMaterialInstanceDynamic*> noise_materials_;
//below is needed because TMap doesn't work with UPROPERTY, but we do have -ve index
UPROPERTY() TArray<UMaterialInstanceDynamic*> noise_materials_;
UPROPERTY() TArray<UMaterialInstanceDynamic*> distortion_materials_;
UPROPERTY() UMaterial* noise_material_static_;
UPROPERTY() UMaterial* distortion_material_static_;

std::vector<bool> camera_type_enabled_;
FRotator gimbald_rotator_;
Expand All @@ -79,6 +86,7 @@ class AIRSIM_API APIPCamera : public ACameraActor
bool auto_format, const EPixelFormat& pixel_format, const CaptureSetting& setting, const NedTransform& ned_transform,
bool force_linear_gamma);
void setNoiseMaterial(int image_type, UObject* outer, FPostProcessSettings& obj, const NoiseSetting& settings);
void setDistortionMaterial(int image_type, UObject* outer, FPostProcessSettings& obj);
static void updateCameraPostProcessingSetting(FPostProcessSettings& obj, const CaptureSetting& setting);
static void updateCameraSetting(UCameraComponent* camera, const CaptureSetting& setting, const NedTransform& ned_transform);
};
9 changes: 9 additions & 0 deletions Unreal/Plugins/AirSim/Source/PawnSimApi.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
#include "NedTransform.h"
#include "common/EarthUtils.hpp"

#include "Materials/MaterialParameterCollectionInstance.h"
#include "DrawDebugHelpers.h"

PawnSimApi::PawnSimApi(const Params& params)
Expand Down Expand Up @@ -426,6 +427,14 @@ void PawnSimApi::setCameraFoV(const std::string& camera_name, float fov_degrees)
}, true);
}

void PawnSimApi::setDistortionParam(const std::string& camera_name, const std::string& param_name, float value)
{
UAirBlueprintLib::RunCommandOnGameThread([this, camera_name, param_name, value]() {
APIPCamera* camera = getCamera(camera_name);
camera->distortion_param_instance_->SetScalarParameterValue(FName(param_name.c_str()), value);
}, true);
}

//parameters in NED frame
PawnSimApi::Pose PawnSimApi::getPose() const
{
Expand Down
1 change: 1 addition & 0 deletions Unreal/Plugins/AirSim/Source/PawnSimApi.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,7 @@ class PawnSimApi : public msr::airlib::VehicleSimApiBase {
virtual msr::airlib::CameraInfo getCameraInfo(const std::string& camera_name) const override;
virtual void setCameraPose(const std::string& camera_name, const Pose& pose) override;
virtual void setCameraFoV(const std::string& camera_name, float fov_degrees) override;
virtual void setDistortionParam(const std::string& camera_name, const std::string& param_name, float value) override;
virtual CollisionInfo getCollisionInfo() const override;
virtual int getRemoteControlID() const override;
virtual msr::airlib::RCData getRCData() const override;
Expand Down
5 changes: 2 additions & 3 deletions Unreal/Plugins/AirSim/Source/SimMode/SimModeBase.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,8 +65,7 @@ ASimModeBase::ASimModeBase()

}
else
loading_screen_widget_ = nullptr;

loading_screen_widget_ = nullptr;
}

void ASimModeBase::toggleLoadingScreen(bool is_visible)
Expand Down Expand Up @@ -201,7 +200,7 @@ void ASimModeBase::initializeTimeOfDay()
sky_sphere_ = sky_spheres[0];
static const FName sun_prop_name(TEXT("Directional light actor"));
auto* p = sky_sphere_class_->FindPropertyByName(sun_prop_name);
UObjectProperty* sun_prop = Cast<UObjectProperty>(p);
FObjectProperty* sun_prop = Cast<UObjectProperty>(p);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change breaks UE 4.24, which is okay IMO, just need to update docs to 4.25
It still produces warnings on 4.25, so it should be fixed completely like
FObjectProperty* sun_prop = CastField<FObjectProperty>(p)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! I will revert this, because officially we still wish to support older UE versions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, that's also good, however I think the latest PRs like the segmentation one and even this has asset modifications which were done on 4.25 I assume, and wouldn't work in 4.24. Need to test all this though
While trying to maintain compatibility with 4.24, it might be good to update the docs to recommend 4.25, since some features could stop working on 4.24 and also raise support questions. Currently docs say 4.22 is supported, which I'm pretty sure is not the case

UObject* sun_obj = sun_prop->GetObjectPropertyValue_InContainer(sky_sphere_);
sun_ = Cast<ADirectionalLight>(sun_obj);
if (sun_)
Expand Down
2 changes: 0 additions & 2 deletions Unreal/Plugins/AirSim/Source/SimMode/SimModeBase.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@
#include "PawnSimApi.h"
#include "common/StateReporterWrapper.hpp"
#include "LoadingScreenWidget.h"

#include "SimModeBase.generated.h"

DECLARE_DYNAMIC_MULTICAST_DELEGATE(FLevelLoaded);
Expand Down Expand Up @@ -124,7 +123,6 @@ class AIRSIM_API ASimModeBase : public AActor

protected:
int record_tick_count;

UPROPERTY() UClass* pip_camera_class;
UPROPERTY() UParticleSystem* collision_display_template;

Expand Down