Tutorial
The goal of the Trolley Mod is to create a simple, effective means of setting up trolley problems in the CARLA simulator. These can be used to record human decisions when confronted with ethical dilemmas on the road, which later could teach the AI of a self-driving car what is the socially accepted decision when confronted with a similar situation.
In the case of the mod, the trolley problems are set up such that there are two victims or group of victims on each side of the road in CARLA’s Unreal Engine 4 (UE4) maps. When playing the Trolley Mod, the car will accelerate automatically, and the user can only swerve left or right. Thus, the human playing the simulation has no choice but to hit something. Upon conclusion of the collision, the choices that the driver made are recorded into a text file for later analysis. That way, raw data will exist for uses such as teaching an AI how to make decisions, compiling sociological information, etc. From this point forward, each individual Trolley problem will be called a scenario. A sequence of scenarios is what constitutes a Trolley Mod simulation. The following text will demonstrate an example implementation of a Trolley Mod simulation to give the user an idea on where to begin.
First, it is important to understand how the Trolley Mod spawns object models into a level in the Unreal Engine 4. It takes an input text file named TestVictims.txt, which contains a list of tuples; the syntax for these tuples are discussed in greater detail later. Each tuple specifies a victim—such as a pedestrian, a car, or some other object—and provides any other ancillary data that describes the object. This text is parsed so the there is a reference to a UE4 Actor class in memory. The remaining text is converted into whatever data type is appropriate—such as integers—to represent the ancillary data. Other functions then take this parsed information and spawns the desired Actor into the level opened in the UE4 editor. For more information on navigating the editor, view this link.
There are three types of objects that can be spawned automatically with the Trolley Mod: pedestrians, vehicles, and prop objects. These correspond to the Walker, the CarlaWheeledVehicle, and StaticActor classes respectively. The first two are native to the CARLA plugin, while StaticActor is a product of the Trolley Mod. The mod adjusts these classes so each class instance stores new information pertaining to it. All three classes share two variables:
- TestNum: an integer that tracks what scenario is being run.
- GroupMemberNames: a delineated string that lumps together the names of all victims in the scenario that the class instance is a part of. It is formatted in the form of Victim1^Victim2^Victim3...
Walker Actors have additional information:
- Age (integer)
- Gender (string)
- GroupNumber (integer): tracks on the map if a Walker is part of a group. If it equals 0 or isn’t included in an input string, the Walker is considered solitary. Otherwise, all Walkers that share the same nonzero GroupNumber is considered to be part of the same group. For example, if three Walkers on the map share GroupNumber = 1, then all three of those Walkers are considered part of the same group. An important note: proximity has nothing to do with whether or not the script interprets a Walker as part of a group—only the GroupNumber.
- GroupSize (integer): an integer that simply records the size of the group the Walker resides in. If it isn’t part of a group, the size is 1.
- SpecialTrait (Name): a UE4 Name that stores a user-specified trait that makes the Walker unique, such as being pregnant, disabled, etc. It is stored using the Tags array native to all UE4 Actors; this page is a good reference for basic information of Tags. It can be left blank if desired.
The input text file should be in the following format: TestNumber|Victim_1|Victim_2
Note that there are no spaces whatsoever in the text. For each victim specifically, the following format is used:
- Pedestrians (Walker classes): ClassName,Gender,Age,GroupNumber,SpecialTrait
- If the Walker has no special trait: ClassName,Gender,Age
- If the Walker has a special trait: ClassName,Gender,Age,0,SpecialTrait
- To spawn a Walker group, chain them together in the string using “^” like so: ClassName,Gender,Age,GroupNumber,SpecialTrait^ClassName,Gender,Age,GroupNumber,SpecialTrait. There is no limit to the number of Walkers that can be spawned in a cluster using this method. However, to ensure all of these Walkers are treated as a group, make sure they share the same GroupNumber, otherwise the output data will be affected!
- Vehicles (CarlaWheeledVehicle classes): ClassName
- Objects (StaticActor class): ClassName,StaticMesh
- ClassName is "StaticActor" for all cases.
- StaticMesh affects the kind of object that will be spawned in. To elaborate, the StaticActor class is a child of the AStaticMeshActor class, which contains a UStaticMesh component. StaticMesh specifies the appearance and physics that the StaticActor will gain. For example, to spawn in a traffic cone the input text may be StaticActor,Prop_ConstructionCone. Or, if a trash can is needed, StaticActor,Prop_Trashcan. See the “StaticMeshes list” page for the compilation of currently stored static meshes.
Now, for an example assume that three scenarios need to be created with a male and a pregnant female pedestrian, a female pedestrian with a car, a group of three pedestrians with a car, and finally a male pedestrian with a traffic cone:
1|WalkeMaleShortsShirtShoesVar1_C,Male,30|WalkerFemaleDressShoesVar1_C,Female,25,0,Pregnant 2|WalkerFemaleSkinnyjeansTshirtBootsVar1_C,Female,21|AudiA2_C 3|WalkeMaleShortsShirtShoesVar1_C,Male,30,1,Deaf^WalkerFemaleDressShoesVar1_C,Female,25,1^WalkerFemaleSkinnyjeansTshirtBootsVar1_C,Female,21,1|AudiA2_C 4|WalkeMaleShortsShirtShoesVar1_C,Male,45|StaticActor,Prop_ConstructionCone
Each numbered line of the text file represents a scenario, where each scenario is separated by a new line. To reiterate, do not use any spaces in the text.
It is important that all values of the victim are spelled correctly, because there are instances when the names must resolve class references in the Unreal Engine 4. If the text has a syntax error, then its Actor may not spawn into the level. A useful reference to ascertain all of the available generated classes can be created using the GetVictimTypes Blutility; this will print a text file called VictimsList.txt with all of the correct ClassNames for Walkers and CarlaWheeledVehicles. To ensure the victim names are correct, simply copy and paste the ClassNames from the VictimsList file into your scenario’s input file. For more information on UE4 Blutilities in general, use this link. More information on Blueprints can be found here.
Once the TestVictims.txt input file is complete, ensure that it is saved in the .../UnrealEngine_4.18/carla/Unreal/CarlaUE4 directory, which is the CARLA simulation’s home directory. The scenarios should now be ready for generation into a map.
CARLA comes with two maps—Town1 and Town2—which is sufficient for the Trolley Mod to perform. It is highly advisable not to modify these maps without making copies of them first. To do so, navigate to the “File” tab in the upper left corner of the Editor, click it, and then select “Open level...”. After the level is open, click the “File” tab again and select “Save current as...”. Rename the level and then save; there will now be a copy of the original map under this new name.
Assuming the Town1 and Town2 maps were unmodified before saving a copy, there should be numerous Actors already in the new map comprising of the classes WalkerStartSpawnPoint, WalkerSpawnPoint, and PlayerStart. To ensure the Trolley Mod works correctly, all Actors from these classes should be deleted from the map. The most efficient way of doing so is using the World Outliner in the Editor.
For example, to delete all of the PlayerStart Actors quickly, type “PlayerStart” into the text box of the World Outliner and a reference to each PlayerStart Actor will appear inside the Outliner. All that’s left is to do is select the upper most PlayerStart, scroll down the Outliner to where the last PlayerStart resides, and then hold SHIFT and click. This should select all PlayerStart Actors in the list. Finally, simply press the DELETE key to remove the Actors and repeat the process for the WalkerStartSpawnPoint and WalkerSpawnPoint Actors.
Once the map is clear, the scenarios are ready to be spawned in. Run the ScenarioGen Blutility by double-clicking it and the victims will be placed into the map (assuming no syntax errors in TestVictims.txt). Currently, the script can automatically spawn in about 36 scenarios in the Town1 map and 12 scenarios in Town2. Because the script avoids spawning scenarios within a certain radius of each other, there will be noticeable gaps in the map where the user can fit in other scenarios manually if desired. Instructions to do so are described in the next section.
The following images show what ScenarioGen will create in the Town1 map if the input text from Section 2 is used:
1|WalkeMaleShortsShirtShoesVar1_C,Male,30|WalkerFemaleDressShoesVar1_C,Female,25,0,Pregnant
2|WalkerFemaleSkinnyjeansTshirtBootsVar1_C,Female,21|AudiA2_C
3|WalkeMaleShortsShirtShoesVar1_C,Male,30,1,Deaf^WalkerFemaleDressShoesVar1_C,Female,25,1^WalkerFemaleSkinnyjeansTshirtBootsVar1_C,Female,21,1|AudiA2_C
4|WalkeMaleShortsShirtShoesVar1_C,Male,45|StaticActor,Prop_ConstructionCone
After the ScenarioGen Blutility is run and the user still desires to add other scenarios in locations the script avoided, this can be done by manually placing Actors into the level manually. For a scenario to be “added” to the data retrieving script, a Target must be placed into the map first.
Target Actors are children of the TargetPoint class; their purpose in the simulation is to pinpoint the location of the next scenario in the map, so the user’s car can automatically teleport from one scenario to the next or specify a certain scenario to run alone, depending on the set preferences (see Section 6). To place one in the map, type “Target” into the search bar under the “Modes” window in the UE4 Editor. An icon with “Target” should appear below the bar.
To place it into the level, click it and drag it directly into level viewport where it is desired. Make sure there is plenty of space around the Target, otherwise there might be issues when running that scenario.
For proper functionality, it is important make sure the Target is facing the right direction, because that is where the user’s vehicle will face when beginning that particular scenario. After placing the Target, select it in the viewport, click the viewport’s rotation tool (or press the E key).
Then, click and drag the mouse to point the Target in the direction the player will need to face. Follow the same procedure for placing in the desired types of victims.
All that remains is for the user to set the variable values for each of the placed Actors. Select the Actor in the viewport, and in the “Details” window all the information about the Actor will be displayed and can be modified there.
StaticActors will by default use the “Prop_Trashcan” static mesh, but if the user wants that Actor to simulate another object all that is required is to click on the drop-down menu under the “Static Mesh” section when it is selected, and then type in or search for the desired mesh.
Finally, take care to ensure the TestNum values for the manually placed scenarios don’t conflict with the ones created by the ScenarioGen Blutility.
The Trolley Mod provides the necessary functions and assets to set up scenarios, but it is up to the user on how to implement them in the Level Blueprint (this is what’s actually run when the CARLA game is playing). Since each user will want to define a simulation differently, the Trolley Mod only provides a functional framework. However, this section will provide a demonstration on setting up a basic Level Blueprint that utilizes all the major aspects of the Trolley Mod and then outputs a file of the simulation’s results.
To open the Level Blueprint, click on the button near to top of the editor viewport called “Blueprints...” and then “Open Level Blueprint.” The following images demonstrate what a typical Level Blueprint for the Trolley Mod may look like; if needed, they can be opened in an external application for more detailed viewing. For more information on Blueprints, click this link.
To begin, the user must specify the following variables:
- Output (string array): this stores the output data and feeds it into a file-writing function once the simulation is complete.
- Locations (TargetPoint array): stores all the locations in the map where the user’s scenarios will take place. This allows the simulation to conduct scenarios one after another by moving the player to next point in the Locations array, and therefore the next scenario.
- TestNumber (integer): this is a running counter tracking the number of scenarios that have been completed.
- PlayerRotation (Rotator): stores the player’s initial orientation in space so the player’s car can be reset to that orientation if necessary.
- HitDisplay (HitResult Widget): will display a result on the screen after the user hits a victim.
- StartDisplay (StartMenu Widget): this is a simple user interface that greets the player before the simulation starts, allowing the specification of features such as the save file name and whether to launch the simulation for a single scenario or for all scenarios.
The Event Tick node is called every frame when the simulation is running. Its only purpose is to make the car Pawn controlled by the player to automatically accelerate, forcing the player to decide quickly on what victim to collide with.
From the Event BeginPlay node, the Blueprint first calls the InitValues function in order to initialize and set the variables defined above. Then, it binds the player’s Pawn to a hit event—in other words, whenever the player collides with any object in the map, it will generate an event that calls the OnActorHit_Event_0 node. This is where most of the heavy lifting occurs.
After the OnActorHit_Event_0 node is called, the simulation immediately unbinds the player Pawn from the node to prevent too many hit events from being recorded. Next—assuming the player hit one of the intended victims—the Blueprint displays text describing what the player collided with using the SetDisplay function. If the player did not hit an intended target, the simulation refers to the PlayerRotation variable to reset where the car is facing and rebinds hit events to it.
Otherwise, the CollisionHandler function analyzes the Actor hit by the player, extracts its data to the Output variable, and then prepares the simulation to move on to the next scenario. This continues until all the scenarios in the Locations array have been visited, after which the simulation writes the data to a text file. It’s also possible for the player to specify only one scenario to complete using the start menu, and consequently the simulation will terminate after the specified scenario is run.
This section demonstrates the implementation of the Level Blueprint from Section 5. First, the in-game HUD class used by CARLA had been changed from “CarlaHUD” to “HUD”; this is because CarlaHUD displays numerical data on the side of the screen which can ruin some of the immersion for the player.
To duplicate this change, navigate to the Blueprints button in the top bar of the UE4 Editor Viewport. Click it, and hover over the “GameMode: Edit CarlaGameMode...” drop-down tab in the “Project Settings” section. In the corresponding submenu, hover over the next drop-down tab called “HUD: CarlaHUD” in the “Game Mode Classes” section. In the submenu of this tab, hover over “Select HUD class” and finally click “HUD.”
To test out the simulation, click the Play button in the top bar of the Editor. A menu screen will then pop up:
When the menu appears, press the SHIFT+F1 keys to use the mouse and interact with it. Notice that there are two buttons corresponding to the two modes that can the simulation can run: Continuous and Single. Continuous mode runs all scenarios present on the map, one after another. Single mode allows only one scenario to be run, which is the one that possesses a test number matching the value entered by the player in the box under “Enter test number.”
By default, the output file is called “test.txt” as indicated by the textbox under “Enter file name(.txt).” However, the user can rename this file by typing in the box. Beside this is a checkbox used to indicate if the user wants to overwrite the output file in case the name already exists in the save directory. It is important to note that if the overwrite option isn’t checked and a duplicate file already exists, data from the simulation will not be written.
In keeping with the regular CARLA controls, the A and D keys steer the car left and right. After colliding with a victim, the screen will display the result (i.e., “You killed a pedestrian!”), and then either proceeds to the next scenario or terminates depending on what was selected in the start menu. Once all scenarios are completed, the data will be written to the output file and be placed in the .../UnrealEngine_4.18/carla/Unreal/CarlaUE4 directory.