Revealing Dangerous Behaviors in Self-Driving Vehicles Due to Switching between Different Scenarios

(1)

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to

Switching between Different Scenarios

A Search-based Approach

Aleksander Rødseth Kjelstrup

Thesis submitted for the degree of

Master in Programming and System Architecture:

Software 60 credits

Department of Informatics

Faculty of Mathematics and Natural Sciences

UNIVERSITY OF OSLO

(2)

(3)

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to

Switching between Different Scenarios

A Search-based Approach

Aleksander Rødseth Kjelstrup

(4)

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to Switching between Different Scenarios

http://www.duo.uio.no/

Printed: Reprosentralen, University of Oslo

(5)

Abstract

With the continuous research and development of autonomous vehicles, there is an unmet demand of streamlining the testing process. As the outfitting and testing of autonomous vehicles is expensive and infeasible when vehicles are required to drive billions of miles to demonstrate their reliability, one can instead opt to use simulation software and execute scenarios to test the vehicle systems. Additionally, due to the multitude of environmental conditions that autonomous vehicles must operate under, it is of upmost importance that vehicle safety is maintained.

During its operation, vehicles are continuously executing scenarios to change lanes and react to traffic conditions. To be able to reveal flawed systems, one has to repeatedly test the system until it fails. Due to the complexity of autonomous vehicle systems, it is difficult to determine the exact cause of failure without extensive domain knowledge. Even with this, it is possible to encounter the oracle problem where it is not certain that the autonomous vehicle is at fault. With the use of simulations, one can test autonomous systems in specified scenarios at a rapid pace. When these vehicles switch between scenarios, it is important that safety is maintained, however this is not always possible.

This thesis explores the use of search-based algorithms to reveal flawed system configurations when testing ego vehicles, specifically testing their nature when switching to different scenarios.

From the results, it is shown that ego danger is greatly increased by the application of the genetic algorithm. In the first scenario group out of all executed scenarios, the results were 54.75%, 61.23% and 65.9%

respectively for scenarios in which at least one collision occurred. For the second group the results were 48.58%, 57.39% and 67.96%. As for the results regarding ego configurations which experienced at least one collision, out of all that were tested, the first group scored 80.0%, 84.39%

and 87.69% respectively. Where as the second group scored 72.42%, 81.05%

and 89.61%. The result indicates that while danger is attained through the use of the genetic algorithm, it does not guarantee that each scenario will experience a collision. As there is a relationship between scenario and system dependency, which indicates how a scenario is potentially useless with a ego configuration which cannot be adequately tested due to its nature.

(6)

List of Figures

2.1 Potential of traffic accident avoidance by adopting autonom-

ous vehicles [57] . . . 5

2.2 The system-of-systems of an autonomous vehicle [47] . . . . 7

2.3 Example of one-point crossover . . . 12

2.4 Actuation system of an autonomous vehicle [46] . . . 14

2.5 Internal architecture of the traffic manager [51] . . . 16

4.1 Program flow . . . 22

4.2 Representation of an individual Ego . . . 25

4.3 Town01 of scenario_1 inFollowLeadingVehicle . . . 28

4.5 Top down view scenario_6 of FollowLeadingVehicle . . . 29

4.4 Top down view of scenario_1 execution . . . 30

4.6 The introduced obstacle of the second group . . . 31

6.1 Ego parameters that are prone to collisionsFLV, 30 DTV. . . 39

6.2 Safe parameters versus unsafe parametersFLV, 30 DTV . . . 39

6.7 Ego parameters that are prone to collisionsFLVWO, 30 DTV 50 6.8 Safe parameters versus unsafe parametersFLVWO, 30 DTV 50 6.9 Ego parameters that are prone to collisionsFLVWO, 20 DTV 53 6.10 Safe parameters versus unsafe parametersFLVWO, 20 DTV 53 6.11 Ego parameters that are prone to collisionsFLVWO, 15 DTV 56 6.12 Safe parameters versus unsafe parametersFLVWO, 15 DTV 56 6.13 Scenario results from each DTV in both groups . . . 57

6.14 Configuration results from each DTV in both groups . . . 57

6.15 Average fitness across generationsFLV, 30 DTV. . . 59

6.18 Average fitness across generationsFLVWO, 30 DTV. . . 61

6.19 Average fitness across generationsFLVWO, 20 DTV . . . 62

6.20 Average fitness across generationsFLVWO, 15 DTV . . . 62

(9)

List of Tables

4.1 Display of data from a scenario at a 100 frames interval . . . 24

4.2 Scoring of an individual Ego DTV . . . 27

5.1 Parameter constraints for the initial experiment . . . 34

5.2 Example of an overly cautious vehicle . . . 34

6.1 Scenario results forFLV, 30 DTV . . . 38

6.2 Configuration results forFLV, 30 DTV . . . 38

6.3 Results ofGeneration 1inFLV, 30 DTV . . . 38

6.5 Results forFLV, 20 DTV . . . 41

6.9 Results forFLV, 15 DTV . . . 45

6.13 Results forFLVWO, 30 DTV . . . 49

6.14 Configuration results forFLVWO, 30 DTV . . . 49

6.15 Results ofGeneration 1inFLVWO, 30 DTV . . . 49

6.25 Percentage of safe configurations across experiments. . . 57

7.1 Scenario results from experiments of each group . . . 63

7.2 Configuration results from experiments of each group . . . . 63

7.3 Behavioural difference with binary representation . . . 70

7.4 CARLA usability in the approach . . . 74

(10)

Preface

I want to thank my family and friends for motivating me and helping me through the past year, with both proof reading and friendly support. I also want to thank my supervisor Shaukat Ali of Simula Research Laboratory for putting up with my sporadic updates, always being available for guidance and being a pleasure to work with. Thesis work aside, this past year has been a very challenging endeavor mentally and the support I received from everyone kept me going.

(11)

Chapter 1

Introduction

This chapter introduces the motivation of the thesis and the project plan, before introducing the problem statement and the proposed research questions. Followed by the contribution of the work.

1.1 Motivation

It is a challenging endeavour to commit and dedicate oneself to writing a master thesis. Therefore, the selection of the topic should be something that can be enjoyable or in other ways motivating to work with for the student.

In this case, it starts with the fascination of new and emerging technologies in the field of autonomous vehicles, as well as related topics such as ethical questions, privacy, and the potential danger such technologies can lead to. With very limited experience in this area, the choice fell on the testing aspect of autonomous vehicles. Specifically, presenting a testing approach to support the field of testing methods and strategies.

The thesis will be a part of an ongoing project at Simula Research Laboratory, focusing on the design, development, and testing of self- driving vehicles.

1.2 Planning the project and initial work

There were three fundamental items of priority at the beginning of the project. The first task was to find a suitable software to be able to run the simulations needed to realize the thesis approach. The second task was to get familiarized with the underlying concepts and theory surrounding the domain of self-driving vehicles and its technology. The last fundamental task was to get familiar with the chosen software and understand the documentation to be able to create the scripts needed to create and execute the experiments.

It should be noted that due to the ongoing COVID-19 pandemic, there were some aspects of the original project plan that had to be reworked.

This effectively changed the approach of the thesis. However, the physical limitations due to the pandemic occurred fairly early in the overall thesis

(12)

work, so little progress was lost and changing course did not cause a significant setback.

The selection of the simulation software would determine the possibilities and limitations of the thesis. Therefore, it was important to consider its availability, complexity, and whether or not it could be run out-of-the-box.

This phase of the project was also intertwined with reading and getting familiarized with the concepts of autonomous technology and regarding testing in general as well as scenario-based testing of autonomous vehicles. It is no understatement to say that the scope of the thesis effectively changed with the deepening understanding of the functionality, accessibility, and limitations of the simulation software. Original ideas or concepts had to be reworked to fit the constraints of the selected software.

The result of the work is three distinct scripts which will run the simulation with a specified scenario sequence, initialise the ego vehicle with parameters determined by an evolutionary algorithm, and lastly scrape the data from each scenario for a given vehicle which in turn is used for fitness evaluation. In succession, the scripts allow us to evolve the vehicle behavior in the simulation.

1.3 Problem statement

It is important that self-driving cars do not cause physical damage to either its occupants or external entities. Currently, the development of such vehicles are, as they should be, confined to strict standards regarding their safety of operation. However, identifying faults regarding the system behavior is difficult due to the sheer complexity of autonomous driving systems. Engineers are working with a black box in which they have to trace how a system configuration may have caused the self-driving vehicle to swerve or why it did not register the shifting traffic lights. Testing autonomous driving systems can be a tedious and time-costly endeavour.

As such, it is necessary to develop new tools to help mediate the testing processes.

During autonomous driving the vehicle is continuously switching between scenarios due to the nature of the vehicle path planner. The vehicle experiences such scenarios as initiating a lane change, making a right turn, or exiting a roundabout. With the use of search-based algorithms and simulation software, I will present an approach to testing the behavior of self-driving vehicles when switching between such scenarios. Where ego behavior will be evolved towards a more dangerous behavior which can mediate the ability to identify system flaws.

(13)

1.4 Research questions

The first two research questions were developed with the verification of the ego behavior in mind, the final research question was added when reviewing results from the initial experiments.

• RQ1: How does the danger affect the ego vehicle between scenarios within a scenario group?

• RQ2: How does danger affect the ego vehicle between scenario groups?

• RQ3: How are the ego configurations related to ego danger?

1.5 Contribution of the work

The work presented in this thesis explores a different approach to scenario-based testing, where a search-based method is used to evolve dangerous behavior in ego vehicles with the goal of testing danger in and between scenarios. The work displays the relationship between system configurations and scenarios with a great increase in experienced danger, showing the effectiveness of a search-based strategy. While the use of evolutionary algorithms has been used in previous research regarding autonomous vehicles. I have not discovered any other study focused on evolving dangerous behavior to study its effect when switching between scenarios.

(14)

Chapter 2

Background

This section informs on the background information related to the work of this thesis. It begins by introducing the concept of self-driving vehicles as well as integral components of their automated systems which enable their autonomous behavior. Afterwards, there is a discussion regarding the benefits following their adoption. Further on, there is the question of ethical impact and public perception which affects the image of the technology while considering the current public opinion. Following, this is a brief view on the preparedness of countries regarding infrastructure and legislation and potential issues with the deployment of autonomous vehicles. To further expand on the research area and current workflow, there is a section on how testing is done in practice. Then there is a brief discussion of related works and earlier proposed ideas and concepts which can be taken into consideration regarding the work of the thesis. Lastly, there is the introduction of the software that has been utilized, as well as an in-depth explanation of the program flow with a brief explanation of the approach.

2.1 Self-driving vehicles

The introduction of self-driving vehicles brings a promise beyond their autonomous capability. They can significantly reduce average vehicle ownership per household via vehicle sharing [43], which in turn can affect overall traffic congestion. The adoption of the technology can also produce a faster and more reliable way of transportation by way of methods such as route optimization and GPS tracking [22]. Other areas of improvement such as reduced emissions, traffic congestion, and the increase in fuel efficiency lead to better fuel economy and therefore less pollution [28].

Following the approval from regulators and the public acceptance of the concept, it is estimated that autonomous and near-autonomous vehicles will bring a significant economic impact of 200 billion USD to 1.9 trillion USD per year by 2025 [10]. Current obstacles in the wide adoption of autonomous vehicles, besides the development, include establishing necessary regulatory frameworks, supporting infrastructure, legal and ethical questions as well as public support.

(15)

Figure 2.1: Potential of traffic accident avoidance by adopting autonomous vehicles [57]

One of the selling points of self-driven vehicles, besides the economic benefit, is the promised increase in reliability and safety when it comes to travelling by automotive vehicles. Wachenfield and Winner [57] described how the introduction of self-driving vehicles can prevent accidents that are prone to happen, since most present-day accidents are caused by human drivers. As figure 2.1 shows with consideration to the severity of an accident. While new accidents are introduced by adopting self-driving vehicles, they can reportedly reduce traffic fatalities by up to 90% [9]. As seen in figure 2.1, there is a big reduction in accidents with a lower degree of severity. A higher degree of safety is also crucial when it garners a public positive attitude to the technology.

2.1.1 Public opinion

One should take note that these vehicles are bound not only by the current technology and financial investments in the area. They are also bound by the public perception and the image of what self-driving vehicles are capable of, both for good and harm. With the adoption of autonomous vehicles, one can expect a reduced number of severe accidents in day-to- day traffic, leading to fewer fatalities and minor accidents overall. With this argument alone, one would expect that corporations, governments, and the general public alike would all massively support the technology.

While it is late to call it an emerging technology, self-driving vehicles are still in its early stages. Especially when it comes to public perception, given that the use of autonomous vehicles is not the norm in day-to-day traffic. As such, there is a lack of experience when acting and reacting to autonomous vehicles in traffic for most drivers. With this comes some uncertainty in the public eye. Most of whom are inexperienced or have a lack of knowledge regarding the area of the technology itself.

Studies have shown that the public perception of self-driving vehicles can be polarized [32]. Where some groups widely accept or hold positive attitudes towards self-driving vehicles, other groups may fear or have other negative attitudes towards them. However, a direct experience with

(16)

these vehicles has shown to change the public attitude to be more positive of the technology. When considering the attitude towards autonomous technology between groups, there has been observed a more positive attitude in rural areas when compared to urban areas [19]. Especially when considering the possibility of public transportation as this can increase reachability and and general road access in rural communities. As it is, the current public opinion is varied, public perception may be negatively skewed as most people have not interacted with the technology yet. On the other hand, results from other studies(N=1,722)has shown a more positive reception to the idea of autonomous vehicles [44].

When asked about their opinion regarding the technology, the respondents from Asia (China, India, Japan) were more positive toward the idea of self-driving vehicles than the western respondents (U.S, U.K). Where the U.S had the highest result, with 16.4% of respondents having a negative impression. Furthermore, when asked about the benefits expected from autonomous vehicles, the majority reported that benefits such as fewer crashes, reduced severity of crashes, and shorter travel time were likely.

Which implies that the public generally accepts the superior safety which is introduced with the wide use of autonomous vehicles. On the other hand, when regarding concerns while riding in an autonomous vehicle with SAE¹ level 3 self-driving technology. Most results reported in being slightly concerned or more. Where only 14.6% (U.K) of respondents reported that they would not be concerned. With the rest of the respondents (China, India, Japan, U.S, Australia) reporting lower.

2.1.2 Autonomous vehicle systems

Generally speaking, there are several levels of automation in vehicles.

Advanced driver assistance systems (ADAS) that support drivers with functions such as driving or parking, are commonly classified between between SAE level 1 or 2 [23], depending on their degree of automation.

Autonomous vehicles at higher levels consist of both software and hardware, as well as sensors to track their surrounding environment in order to be capable of performing a higher degree of autonomous driving.

They are reliant on their automated driving systems (ADS) and are defined as an autonomous system at or above SAE level 3, depending on the degree of autonomy.

The systems and equipment in an ADS includes, but are not limited to; path planner, lidar, visual sensors, and actuators to realize decisions on different parts of the vehicle like throttling or braking. Each component has its own purpose, be it the visual sensor which is to detect other vehicles, pedestrians, or obstacles or the path planner which is responsible to determine the path which is to be followed by the autonomous vehicle.

In general, a path planner calculates a number of different possible paths at each time step. These calculations are based on a set of cost functions with different weights, where the best path is chosen and followed until the next

1SAE International, a standards developing organization

(17)

Figure 2.2: The system-of-systems of an autonomous vehicle [47]

(18)

time-step. As such, the vehicle ADS is continuously computing the best option on the current route. The path planner weights are often calculated to conform to different aspects such as safety, comfort, compliance, and other internal or external limitations. It must also be stated that this function is not uniform for all autonomous vehicles across the automotive industry. While the idea remains the same, that is, to choose the best possible path, the function itself and its weights varies.

The software itself exists as a system-of-systems [47] (see Figure 2.2) with many complex parts and modules that rely on one another to operate.

The ADS has to be able to properly identify street signs, pedestrians, traffic lights, and road markings as well as identifying and distinguishing between multiple types of vehicles. Figure 2.2 shows a broader picture of a multi-vehicle system where several autonomous vehicles will give each other feedback. The communication module aside, the figure highlights the complexity of a single autonomous vehicle and the sub-systems necessary for optimal autonomy.

2.2 Urban deployment of autonomous vehicles

While the research and development of autonomous vehicles is a hot topic for the industry, one must also consider the question of urban implications.

An autonomous vehicle should always follow the traffic rules, no matter where in the world the vehicle is. This is given, but the current set of traffic rules depends on the country and no defined international standard defines them. While there have been conventions and agreements on this topic [41], which forms the base of many national traffic laws. There are still loopholes and uncertainties such as what constitutes a driver in an autonomous vehicle [56]. What or who is the driver; the passenger, the system itself, or the manufacturer? With the potential wide use of autonomous vehicles, one has to take into consideration the existing urban environments such as infrastructure and the local driving culture. This must also be extended to the drivers of human-operated vehicles (SAE levels 1 and 0) and their reaction to the early adoption of the technology. Driving around in an environment with more autonomous vehicles than non-autonomous is not something that has been experienced before. Therefore, it is hard to predict how normal drivers will react to potential changes in driving culture.

The accessibility of autonomous vehicles is largely dependent on clear lane markings, traffic lights, and traffic cohesion. Where some countries have invested in more bicycle-friendly cities with clear markings and lane division in traffic. Other countries and cultures have developed a driving culture which involves having both vehicles and cyclists sharing the lane with motorcycles or scooters weaving in and out of traffic in between vehicles. When deployed into environments as previously described, the behavioral difference between the ADS will become very noticeable. As such, if the infrastructure, driving culture or urban environment in general hinders the vehicle, then it is better to not adopt them as the vehicles will not reach their full potential and might instead cause problems in traffic.

(19)

Other problems that can arise when adopting the technology are the effects of harsh weather on the system and sensors of the vehicle. Vehicles will experience more darkness in the winter and so the perception of the vehicle can be reduced. This can lead to mislabeling or not identifying vehicles and pedestrians. Mislabeling or even labeling too late can lead to accidents [40]. This is especially dangerous considering the possible harsh weather that can arise, such as heavy snowfall or icy roads. Finally, the autonomy of the vehicle is only as strong as its software and its ability to detect and identify its surroundings. If the vehicle is driving along an old decrepit road with little signage and lighting, then the vehicle will surely not perform at the quality one might expect. A wide adoption of autonomous vehicles cannot be implemented when all system testing occurs in urban environments that are modeled after a few countries’ city layout and local climate. As [42] explains, one must take the climate, urban environment, existing infrastructure, and environmental quality in to consideration when testing and developing autonomous systems. Of course, testing should be done to assess the functionality and behavior of the ADS, but there must also be tests where the local infrastructure is lacking. Building on this, an assessment [49] of current readiness for autonomous vehicles indicates four areas of importance.

• Policy and legislation

• Technology and innovation

• Infrastructure

• Consumer acceptance

It is suggested that national and local transport strategies are revised to account for a future use of autonomous vehicles. As such, countries should introduce more EV charging points, higher quality AV road infrastructure, and better mobile communication technologies. Furthermore, governments should help drive the adoption of AV-technology by funding and supporting AV-pilot projects, as this would increase the development speed as well as ensuring that the technology can be steered by newly adopted policies.

2.2.1 Privacy in an autonomous world

As with any modern technology with an internet connection, there is always a concern regarding privacy, this is no different with autonomous vehicles. However, the wide usage of the internet, smartphones, email, and apps has led to the introduction and adaption of new laws, which increases privacy protection [45]. As such, it is no wild guess that this would not also be the case for autonomous vehicles when they become widely adopted.

Even so, one must consider the implications of the possibility of having your own vehicle position logged and stored on servers owned by the

1Electric Vehicle

(20)

vehicle manufacturer. The implications also apply to theInternet-of-Things (IoT) [59] of smart vehicles.

While access can be limited, it is no doubt that smart-vehicles can be hacked, especially when vehicle users show a lack of concern regarding whether or not they will use a smart-vehicle app which can be hacked.

[34]. This is a major concern when it comes to autonomous vehicles, when hackers successfully infiltrate an autonomous vehicle, they can get control over critical behavior-related functions [24]. If they can be accessed and controlled remotely, there is a considerable risk when it comes to the safety of all passengers and pedestrians in the vicinity of the vehicle. Furthermore, hacked systems can reveal patterns such as driving behavior and vehicle usage, which can pinpoint the home, place of work, or other locations frequently visited by the vehicle owner [31], [48].

Such data is very valuable and can be used in many ways such as either location tracking with criminal intent or gathering data with the intent to create user-specific advertisements [8]. To prevent such possibilities, the manufacturers and developers of both vehicles and accompanying applications must increase the security of their systems.

2.2.2 Ethical considerations

As previously mentioned in section 2.1, safety is a critical part of the acceptance of self-driving vehicles. With this in mind, the autonomous vehicle has to be as approachable and useful as possible for the general public. Therefore, the argument of increased safety is probably the strongest reasoning for making its wide use and adoption a reality. As studies have shown that over 90% of car accidents are caused by human error [21]. When one instead opts to ride in a self-driving vehicle, the possibility of human error is all but eliminated, instead the passengers must place all trust in the vehicle ADS.

This in of itself is a different area of trust that needs to be gained with the use of self-driving vehicles. Specifically, a more ethical approach to how ADS should behave in different traffic scenarios. There are many things to consider regarding ADS behavior in dangerous situations.

• Should it divert all of its efforts into minimizing overall damage, if so, at what cost?

• Should the life of passengers be prioritized over the life of pedestrians?

• How do you quantitatively differentiate damage?

• Are the broken bones of a passenger worse than the broken bones of an innocent bystander?

From a bystander point of view, you would like to minimize the overall damage, but as a passenger would you not prioritize your own well- being over the other vehicles and their passengers? As [36] points out people have inconsistent or paradoxical attitudes when answering ethical

(21)

questions regarding their own safety either in- or out of vehicle. While people favor self-driving vehicles that minimize harm, they also want the self-driving vehicles to favor themselves.

Liability and safety are urgent concerns regarding the technology [35]

and the question of who should be liable for collisions has implications for future legislation [39]. Whether the moral agent of ethical dilemmas should be the vehicle, the programmers behind the ADS, or the owner of the vehicle is up for discussion [37]. While the development of the technology is moving forward, the ethical question remains and is not something that can be determined by a few lines of code. As such, another hurdle which self-driving cars must clear, besides increased vehicular safety and solutions to ethical questions, are the legal and regulatory frameworks. The introduction of new legislation that encompasses the AV-technology can be very difficult given the rapid pace at which the technology develops.

As such, issuing relevant regulatory frameworks, standards, or legislation will encounter issues regarding their relevancy [2]. Furthermore, the widespread adoption of self-driving vehicles is held back until adapted frameworks and regulations are in place. These issues act together like a tug of war. Where the rapid development of autonomous technology renders previously issued standards as potentially useless and redundant, and the deployment and usage of autonomous vehicles are delayed by the lack of frameworks and legislation.

2.3 Genetic Algorithm

The use of search-based algorithms has been proven to be useful for autonomous vehicle research since performing an exhaustive search is impossible due to the large number of environmental conditions [4], [18].

Given the possible search space that arises when conducting experiments with self-driving vehicles, one wants the ability to efficiently find a solution to a contextual problem. With genetic algorithms (GA) [14] a problem with a large search space can be efficiently searched by applying evolutionary ideas in practice. Therefore, using search-based algorithms such as the genetic algorithm helps in reducing the time complexity one encounters in an exhaustive search, by applying a core concept from the theory of evolution. Namely, the survival of the fittest. While implementations of the genetic algorithm differentiate, they generally follow the steps seen in [33] which are divided into two parts.

• Initialisation

– Initialise the first generation by creatingNrandom solutions.²

• Learning

– Until the new population hasNindividuals

1. Create a new population, which is initially empty

2Each solution represents an individual in the possible search space

(22)

Figure 2.3: Example of one-point crossover

2. Select two solutions from the current population, using a specified way of selection

3. Recombine the parents in order to produce an offspring 4. Mutate the offspring

5. Add the offspring to the new population

To simulate evolutionary ideas in practice, one first has to abstract the problem at hand. This mainly revolves around how a population consisting of individuals should be represented. These individuals are possible solutions that exist within the search space of our problem. The individuals mimic the concept of survival by having a fitness score which is computed from the effectiveness of their solution. As such, the individuals in a given population have their fitness evaluated by the criteria of the algorithm, which depends on the context of the problem. With this concept, it is possible to differentiate and rank individuals based on their evolutionary fitness. Individuals with traits that are desirable will get a higher fitness score than individuals with traits that are undesirable. Following the evaluation of each individual in a population, the next step is to select some of the individuals to be the breeders for the next generation. This is usually done with the individuals that achieved the highest scores in a generation.

There are several ways to do this, such as:

• Tournament selection

Where two individuals from a population are selected either at random or from an elitist criterion and compare their fitness scores with each other. Where the best individual is selected as a parent.

• Truncation selection

Where a selected fraction of an existing population is selected, say f = 0.5. Such that the best 50% individuals in a population are put into a mating pool and randomly shuffled to make pairs.

After selection, the parents are recombined to create offspring, this is the genetic part of the algorithm. With the individuals being represented as chromosomes with genes as in figure 2.3. We can apply genetic operators such as crossover and mutation to create new individuals. The way this works is by selecting one or more crossover points and having these points determine where the gene from a parent comes from. If a selected crossover

(23)

point is half the length of a parent chromosome, then the offspring will inherit 50% of their genes from each parent. After the crossover, we mimic the probability of mutation from evolution. By applying a small chance per gene in the offspring, we have the ability to escape possible convergence in a local optima³. The mutation gives a chance to possibly generate a much better solution than any individual in the current or previous populations.

The mutation rate is usually kept low to maintain a degree of control over the rate of the evolving power of the individuals. Selecting the proper mutation rate can be heavily dependent on the context and purpose of the algorithm, especially with consideration of the possible search space.

2.4 Simulation software

While the development and research on autonomous vehicle software is widespread, there is a limit to their complexity. Open-source simulators such as TORCS [12] lack the complexity of urban driving such as pedestrian actors or traffic rules and adhere more to track racing rather than urban city driving. As [38] suggests, the simulation software in use should have a compatible interface with access to data from ego and non-ego vehicles as well as other environmental objects. More importantly, the simulation software should have a detailed physics engine and sensor capabilities to mimic real-life scenarios as much as possible.

Therefore, the software used in this thesis is the open-source project CARLA (Car Learning to Act) [11] which is an open simulator for urban driving. The software has been built to support the training, prototyping, and validation of autonomous vehicles. The simulation includes urban environments, a multitude of vehicle models as well as buildings, street signs, pedestrians, and more. CARLA is built in C++ and runs in Epic Games’ Unreal Engine 4 (UE4) [54]. The simulation is set up as a server-client system where the client API is implemented and used in Python. Python scripts are used as the interface to control the simulation.

Commands are sent by the client and the server returns sensory data. As such, the user is able to steer the simulation and affect different settings such as weather, vehicle physics, velocity, and adherence to traffic rules.

2.4.1 CARLA

The software simulates a dynamic world in which the user can control one or several agents through the API. While the simulation can never replace real-world testing for autonomous vehicles, the software has taken steps to accommodate differences. Non-player vehicles and their kinetic parameters as well as vehicle engine power, vehicle weight, acceleration rate, and other vehicle-based statistics vary from car model to car model. CARLA introduces three approaches to use in the study of autonomous driving. Firstly, a modular pipeline that consists of a vision- based perception module, a rule-based path planner, and a maneuver

3The best possible solution within a small neighborhood of possible solutions

(24)

Figure 2.4: Actuation system of an autonomous vehicle [46]

controller. The second approach is a deep network that maps input to driving commands, which has been trained via imitation learning. The last approach is also a deep network, but is trained using reinforcement learning instead.

This thesis makes use of the first approach, the modular pipeline, given the performance of the approach in goal-oriented navigation tasks. The pipeline divides the driving task into three sub-systems:

1. Perception 2. Planning

3. Continuous control.

The perception system uses semantic segmentation [1] to determine road limits, hazards, lanes, and other dynamic objects. Additionally, a classification model [7] is used to determine the proximity to intersections.

The planning module uses a rule-based state machine with predefined policies that have been tuned for urban environments, to execute different driving strategies. The control of the ego agent is being performed by a Proportional-Integral-Derivative (PID) controller [53] that actuates the throttle, steering and brake. The controller functions as the actuator system as shown in figure 2.4

The PID-controller is a control loop where feedback is used to correct the controller. In the loop, it will continuously calculate an error value between a set target value and its current process value. The corrections are calculated on the proportions, integrals, and derivatives in question.

In other words, given continuous feedback, the PID-controller will adjust itself to achieve the wanted behavior. The functionality of the PID controller is essential in how the simulated ego vehicle will behave in a given scenario. It acts on the current velocity, target velocity, and angular

(25)

deviation of the vehicle in question. From these values, the simulation will run two PID calculations, one for its longitudinal output and one for its lateral output. The longitudinal value represents the vehicle throttle and braking power. Whereas the lateral values represent the steering power of the vehicle.

2.4.2 Carla Traffic Manager

To properly simulate realistic urban traffic conditions, CARLA makes use of the Traffic Manager module. This allows users to customize vehicle and traffic behavior according to their needs. The traffic manager is controlled by the client and its execution is divided into several stages with their own operations. The logic shown in figure 2.5 can be divided as follows:

1. Store and update the simulation state

The world is scanned to keep track of existing actors (vehicles, walkers and other entities). TheAgent Lifecycle and State Management retrieves the positional data and stores the agents’ velocity, position and bounding box, with more.

2. Calculate the movement of vehicles in vehicle registry

Each vehicle has its movement calculated separately. Which again are separated into four stages of their own. With synchronization barriers all calculations are completed for each vehicles before moving on to the next stage.

(a) Localization:A near-future list of waypoints is generated (b) Collision:Bounding boxes are extended to identify any collision

hazard

(c) Traffic Light: Identifies hazards that can affect the vehicle with influence from traffic lights, stop signs or junctions

(d) Motion Planner: When a path has been defined (list of waypoints) the movement is calculated using aPID Controller 3. Apply commands in the simulation

When every action has been calculated, the final stage will apply the commands in the simulation

(26)

Figure 2.5: Internal architecture of the traffic manager [51]

(27)

Chapter 3

Literature review

This chapter will briefly discuss testing practices and introduce the concepts, tools, and approaches of earlier works in the field of autonomous vehicles. These works are used as inspiration and general guidance for the work of the thesis.

3.1 Testing practices

To validate the functionality and design of an ADS, there is the option to run tangible tests within either a controlled environment such as testing tracks or out in real traffic. Additionally, one can opt to use Hardware- in-the-loop (HiL) or X-in-the-loop [60] (XiL) to test real-life components either in- or out of simulation. These approaches allow more flexibility in simulating parts of the system as shown in [6]. These options are also more cost-effective as testing autonomous vehicles on a large scale is not only costly, seeing that one has to acquire, design, and outfit a car with the necessary instruments. It is also a very tedious and labour intensive process [13], with physical limitations and other uncontrollable constraints such as weather, traffic, and the durability of the vehicle and vehicle components themselves. There is also the question of real-life implications if anything should go wrong during the testing, which occurred in 2018 when an autonomous vehicle operated by Uber struck and killed a pedestrian [58].

Besides the manpower and economic costs that come with outfitting a single vehicle to be able to operate autonomously, a limitation is that a single vehicle is not sufficient to collect the vast amount of data needed.

As the development of self-driving vehicles also requires billions [26]

of miles driven to demonstrate their reliability. The alternative is to utilize simulations to validate driving strategies. This option is not only affordable, it is also a more efficient and safe method regarding adapting real-life scenarios. The use of simulations also makes it easier and more adaptive regarding mass-testing one or more vehicles in complex situations [20]. Many real-life scenarios are too dangerous and costly to validate as the result may damage one or more of the vehicles that are used. As such, there are edge cases which are hard to test without destroying expensive equipment. This goes especially for the self-driving vehicle (known as the

(28)

Egovehicle) whose ADS is being tested. Running a test which results in a crash for the ego vehicle when it is equipped with expensive instruments is a costly decision and its debatable whether the collected data outweighs the economical costs.

Instead, it is recommended to run simulated scenarios to validate the behavior, using one or more of the aforementioned techniques. Scenarios in this context can be defined as a dynamic description of both the environment and the components as well as sub-systems of the autonomous vehicle, over a period of time [30]. Given the possible conditions available in a scenario, the interactions between the environment and the autonomous vehicle can be hard to predict. This introduces a challenge regarding scenario-based testing due to non-determinism. As there are no unique correct ADS behaviors for any given scenario, which makes it harder to evaluate the correctness of the results [27]. This does not, however, de- value the benefits of simulated testing. Seeing that customized scenarios can be deeply complex and as such be very cost-effective regarding testing and validating a wide range of differently configured ADS.

With that in mind, the automotive industry has access to both monetary funds to equip and modify vehicles for autonomous behavior. Even so, the process of developing and testing self-driving vehicles is tedious. Given that the underlying system-of-systems to a degree exists as a black box [55]

to the engineers working on vehicles. It is not easy to point out where or what went wrong within the system after a test. A testing engineer and a systems engineer who work on developing the tests or the system itself may not have extensive knowledge of what the other part is specifically doing. A test run of an ego vehicle can be a scenario in which the ego vehicle will overtake a leading vehicle in a traffic scenario. The engineers will determine the weights of the functions which will determine the system behavior, but the execution of the scenario can lead to unexpected results which are not easily detectable given the complexity of the system.

This could be anything from a miscalculation in a function, a false-positive label in the vehicle’s perception system, to possible faulty equipment in the ego vehicle. In cases like these, the use of deep neural networks are shown to be useful when detecting erroneous behavior in vehicle ADS [50]. While this approach does not determine which part failed, one must take into account the complexity of it all. By utilizing previous works, frameworks, or approaches to testing, one can specify tests based on vehicle parts, systems or behavior independently.

Furthermore, as [46] describes, conventional testing of a system and its subsystems falls short due to the number of interactions and environmental conditions that the system must perform under. Even with the introduction of automated testing of the system-of-systems, some conditions might be missed. Furthermore, it might not be possible to verify if a system will operate as expected with the vast number of conditions that the vehicle is exposed to. Which leads to another problem regarding the testing of an ADS. Whether its in- or out of simulation, there is the possibility of the oracle problem [3]. Stating that its not always clear whether the ADS is at fault for a failed criterion or not. This aside, regarding solving the problem

(29)

of testing systems with a large number of environmental conditions, constraints, and variables, one can apply search-based algorithms [29].

It has previously been shown that with a search-based testing approach it is possible to solve ADS scenario collisions with an alternative ADS configuration [5], this inspired the approach and the chosen method for the thesis.

3.2 Other works

In earlier works [4] it is brought forth in the simulated testing of scenario collisions between an ADS and other vehicles, that one cannot claim it has been a failure of operation of the system. Without complete access to the specification of the ADS or extensive knowledge of the domain. The author of the aforementioned work proposes the term avoidable collision, which depends on neither domain knowledge nor specification. A collision is deemed avoidable if there exists a different configuration of the ADS in which the resulting test does not crash the vehicle. Given the vast search space for possible configurations, it is not feasible to conduct an exhaustive search to find an avoidable collision. Therefore, the use of evolutionary algorithms, more specifically the genetic algorithm, is a favorable choice to find a possible ADS configuration which is said to solve the scenario.

Regarding the scenarios themselves, they are often derived from engineers with domain knowledge [16]. Given the sheer number of possible scenarios, it is infeasible to generate one for each possible test case. With this in mind, when selecting a scenario to test, there is the possibility of generating a “best” possible scenario as suggested in [18]. By way of measuring the fitness of a parameterized scenario, one can deduct a scenario which can reveal flaws in the ADS. This is a useful tool regarding reducing the overhead which comes when testing self-driving vehicles and their ADS. With the works previously mentioned, one can increase the efficiency of scenario-based testing. This could be solved by limiting the selected scenarios to those that are deemed the best and using evolutionary algorithms to test the different ADS configurations.

By utilizing the proposed tools, frameworks, and approaches from other works, one can increase the quality of results and test cases when used with traffic simulations. The advancement of autonomous technology requires cost-effective optimization in different areas. From producing individual parts or equipment to having proper software and reliable ADS.

With the need to perform millions or billions of miles of testing, one needs to have access to a vast testing catalogue. As repeatedly testing on a limited number of scenarios is not sufficient to detect all possible flaws. There must be a broad selection of scenario types to be able to validate the different systems. While test-case generation and tailored test cases can be used to fill the catalogue with different scenario types. One must take into account that its not feasible to test the systems with the same scenario forever. It is revealed that re-using test scenarios is a bad idea, since a "good" scenario is system-dependent [15]. A good scenario for one configuration of an ADS

(30)

might be totally useless for another. An approach has been suggested by [17], which enables us to determine when a certain scenario type has been exhausted to the point of not being cost-effective. By defining the test-end criterion as a question, if there is enough real-life data collected to cover all scenario types or not.

As previously mentioned, one must take into account that repeating the same test-cases and scenarios over and over can lead to the degradation in the usefulness of scenario cases. As [17] explains, if we run an iteration of a lane-change scenario with a preconfigured ADS. The system behaves as expected, but during the testing revealed that the ADS is dangerously close to other vehicles when performing the lane change. Following this, the ADS is reconfigured to maintain a better minimum safety distance to other vehicles and the scenario is run again. Now there is a possibility of the ego vehicle to not perform a lane change at all, and the lane-change scenario has essentially become useless for further testing. While naively reusing the same test scenarios is a bad idea, since a "good" scenario is system- dependent. Making use of test case generation [52] can help identifying system flaws and other critical behaviors and be a possible solution to overusing the same scenarios.

This thesis follows up on the idea of solvable configurations by utilizing another approach.

• Is it possible to reduce the search space of ADS configurations by utilizing genetic algorithms?

– By utilizing genetic algorithms, is it possible to reveal the dangerous ADS configurations of an ego vehicle?

From the results, we can determine whether or not it is feasible to evolve dangerous behavior to reveal system flaws. As such, the thesis proposes an approach to scenario-based testing, by using search-based algorithms, namely, the genetic algorithm, to verify and therefore eliminate dangerous configurations for an ego vehicle. The essence of the topic boils down to whether or not utilizing search-based algorithms on an ego vehicle with the intent to increase danger, will yield results that can prove to be valuable in scenario-based testing of autonomous vehicles.

(31)

Chapter 4

Program flow

This chapter presents my contribution by introducing and defining the three main scripts used to achieve the results. An overview of the program flow is presented, followed by detailed descriptions of each of the three scripts as well as the preexisting script from the developers of CARLA.

Lastly, the scenario groups used in the thesis are introduced and discussed.

4.1 Project code

The project code consists of several scripts written in Python. There are four scripts which are the main actors for running the simulated scenarios and achieving the results. The script responsible for setting up the scenarios themselves are written by the developers of CARLA and exists as a separate module known as ScenarioRunner. This module provides users with a relatively user-friendly way to engage in intricate predefined or custom made scenarios. The remaining three scripts were all created for the thesis in order to facilitate interaction between the ScenarioRunner script, the CARLA server and the genetic algorithm. Following the naming convention of the scenario module, these were namedEgoRunner, MetricRunnerandGA-Runnerrespectively.

The program flow is described in figure 4.1. At startup the first population is initialised, after following this we move on to the loop in the script. Here theScenarioRunnerandEgoRunnerare initiated, which executes and records each scenario in the scenario queue of an individual. With the data from each recording stored on the server, the next step accesses each scenario-recording individually to extract the wanted metrics, this is the job of the MetricRunner. This returns us to the GA portion of the script where we can now calculate an exact fitness score per individual based on their performance in the scenarios. This is done for each individual in a population for a specified number of generations.

(32)

Figure 4.1: Program flow 4.1.1 ScenarioRunner

TheScenarioRunnermodule provides traffic scenario definition and execution via the CARLA simulator. Written in Python, the module acts as an interface with CARLA by allowing access to define and run different predefined scenarios or custom ones using the OpenSCENARIO [25] standard.

There are several predefined scenarios and scenario groups existing with the module. As the focus of the thesis lies within the domain of scenario- based testing the choice was made to use the predefined scenarios for the experiments. The module also includes a metric module specifically made in order to extract data from previously recorded scenarios. This metric module is the basis of the aforementionedMetricRunnerscript.

4.1.2 MetricRunner

The purpose of theMetricRunneris to retrieve the necessary data from each recording associated with a scenario execution. Therefore, each scenario that has been executed by an individual ego will have their data analyzed and stored. This is a key part of the script as it would be impossible to determine a fitness function or in other ways score the behavior of an ego vehicle without access to proper metrics. The chosen metrics used to determine the danger of a ego vehicle in a given scenario are as follows:

• Ego velocity

• Ego distance to scenario vehicle over time

• Ego DTV parameter value¹

(33)

• Ego distance to lane center over time (DTLC)

• Number of collisions involving the ego vehicle

The purpose of these metrics is to help determine whether the ego is exhibiting a dangerous behavior or not. The velocity metric shows the resulting data from the longitudinal PID-controller for each frame in a scenario, in other words, its resulting velocity in relation to the braking and throttling power. An ego vehicle with a high target speed will increase velocity over time until it is reached or another parameter or object hinders it. With this, it is possible to determine whether the vehicle in question was accelerating when it should have been braking to avoid a collision.

One of the primary indicators of the danger level in a scenario is the distance to an opposing scenario vehicle over time. This is calculated from the ego vehicle center of mass to the opposing vehicle center of mass. Such that an actor within 4.6 - 4.4 meters (depending on the vehicle model) of the ego vehicle in a given scenario equals a collision. Over time, we can see whether the ego vehicle acts in a safe manner and maintains a good distance to the opposing scenario vehicle or if it acts sporadically.

The next important metric is the ego vehicle distance to the lane center (DTLC). As it is hard for heavy objects to enter sharp turns at high speeds, the ego vehicle can swerve in and out of the lane and potentially lead to a crash. The DTLC metric is also useful to relate to the ego vehicle behavior in a scenario in general. If we have a resulting scenario with no collisions but we have swerved on the road, we can deduct that the ego vehicle avoided a collision by moving out of the lane. While an ego vehicle can avoid a collision, the act of swerving out of the lane could be considered dangerous.

Lastly, we have a metric to calculate the number of collisions in a given scenario. Since the recordings from the simulation exist as a packet of data relative to a frame, a single collision may be indicated over several frames. With this in mind, a collision is defined as separate if there’s an indication of another collision at least 5 in-game seconds from the previous collision and that the ego vehicle in question has moved at least 4 meters away from the last indicated collision. This helps to eliminate any potential false positives where the ego vehicle may be out of bounds, but is trying to accelerate and is hitting other objects in its path. Furthermore, a collision should not be counted if, following a stand-still after a crash, the ego vehicle accelerates and immediately collides with another entity. As such, a defunct ego vehicle which accelerates into a wall non-stop would not have its "collisions" counted.

For each scenario queue, there will be an individual instance of the ego vehicle, the aforementioned metrics are recorded for each scenario in the queue and calculated and stored for future fitness scoring. Table 4.1 shows an example of data that is retrieved and stored, note that the original data is stored at 1 frame intervals.

1This is explained further in section 4.1.4

(34)

Frames Velocity DTV DTLC Collision

0 0.00 25.89 0.00 0

100 6.41 10.88 0.00 0

200 5.28 4.68 0.04 0

300 0.31 4.66 -0.01. 1

400 0.02 4.64 -0.12 1

Table 4.1: Display of data from a scenario at a 100 frames interval The results from the table show a periodic increase in velocity over time, followed by a sudden braking maneuver and a collision, as can be seen by the low DTV and of course, the indication of a collision in the rightmost column. The DTLC metric indicates that the collision happened head-on and that the ego vehicle did not steer away from the actor that it collided into. Note here that the DTLC values represent a right or left turn with a positive or negative value.

With access to these metrics, we can get quantitative data from which a fitness score can be calculated. This is an essential step as the proper evolutionary behavior relies on being able to determine which individuals are fit for being used to repopulate.

4.1.3 EgoRunner

TheEgoRunnerscript is responsible for initialising an ego vehicle and being on stand-by until the scenario has loaded and is ready to be executed. This is done in several parts.

1. Connect to CARLA server

2. Load the selected map for a scenario 3. Initialise the ego vehicle

(a) Set up an agent which uses the modular pipeline to control the vehicle

(b) Give the agent a destination set as the end of the street, repeatedly

(c) Spawn the vehicle on the map

4. At each frame, the ego agent runs a calculation step similar to the traffic manager (see Figure 2.5) in order to determine behavior 5. At scenario completion, despawn vehicle and clean up. Prepare for

the next scenario in the queue

The ego vehicle is prepared and spawned with the GA-determined parameters. These are target speed and distance-to-vehicle or DTV for short. The target speed indicates the target velocity for the PID-controller,

(35)

Figure 4.2: Representation of an individual Ego

whereas the DTV parameter indicates the ego vehicle’s distance to an opposing vehicle before it considers braking to avoid a collision. Additionally, the representation shown in figure 4.2 displays the chromosome representation for an instance of a ego vehicle. The scenario queue differentiates between individuals and is not taken into account when calculating fitness.

The use of the queue is to be able to determine the behavior and the experienced danger across different scenarios, and each gene in the queue chromosome determines whether the scenario with a specified index should be executed or not.

While the scenarios are being run, the rendering of the simulation server in Unreal Engine is disabled. This is done to perform complex calculations at a higher frequency and avoid unnecessary workloads on the GPU by rendering maps and vehicles. This is necessary keeping in mind that there are a great number of scenarios to be run for a given execution of theGA-Runner.

4.1.4 GA-Runner

The GA-Runner is the primary script for the project, it initialises a population of a predetermined size with random values for its first generation, after this it starts the ScenarioRunner and the EgoRunner respectively. The GA-Runner completes the loop of the program flow as shown in figure 4.1. To properly explain the inner workings of the program, the following sections are explained in detail:

• The individual

• The selection

• The fitness Calculation

(36)

The individual

To properly run the genetic algorithm, we have to genetically represent the individuals to be able to apply methods such as mutation and crossover.

In the case of the representation of an ego vehicle, we have two things to consider.

1. Each individual ego has a parameter for target speed and DTV, these are our first two genes

2. Any remaining gene represent a scenario within a scenario group As shown in figure 4.2 in order to represent this individual genetically, the representation of target speed and DTV are integers while the scenario queue is represented with 1s and 0s to determine whether a scenario should be executed or not. To properly apply crossover and mutation to the target speed and DTV parameters, they are converted to and from binary, such that crossover and mutation applies to the binary representation of the integers. To avoid unnecessary computation and unwanted results, the binary values were limited to the constraints set in the form of a floor and ceiling values for both target speed and DTV. These constraints were set in place as there’s a limit to how unrealistic behavior we can expect from the configuration of an ego vehicle. As such, the target speed floor and ceiling were set to 5 km/h and 120 km/h, respectively. For the DTV value, the floor was 0 meters and the initial ceiling was set at 30 meters.

The selection

To select the best individuals in a population, an elitist tournament selection was chosen. This led to the two best parents from the previous generation to be selected as breeders for the next. The reason for this choice was the wish to limit the number of generations needed to achieve the wanted behavior, by always choosing the best individuals as parents, we could facilitate dangerous behavior faster. To avoid a homogeneous generation of offspring, a mutation rate of 0.1 per gene was applied, as well as a random point crossover. This ensured that while the offspring would come from the best parents of a generation, their chances of having the same genes would be slim. Especially regarding the target speed and DTV being converted to and from binary at these stages. Another note is that the crossover and mutation takes into account the constraints regarding the upper and lower limits of both target speed and possible DTV. As such, no offspring would be initialised with values outside of the floor and ceiling values.

The fitness calculation

The most important step in the genetic algorithm is to determine the success of each individual in a population. The parameters and selection choices for the genetic algorithm are paramount to achieving results within a search space. Although the parameters can vary on a case-to-case basis,

(37)

there are some that are kept relatively consistent. In this case, the mutation rate, which is set at a 0.1 chance per gene. The fitness of the data is derived from the execution of an individual ego in a scenario queue and was primarily computed with three variables in mind.

• Ego deviation from lane center

• Ego collisions

• Ego DTV

The deviation of an ego vehicle in a given scenario is calculated as the distance from an ego vehicle’s center of mass and the waypoint in the middle of the current lane. This value is calculated on a frame-to-frame basis, where a deviation is counted if the distance exceeds 1 meter from the lane center. With this we get how much a vehicle deviated from the lane and for how long.

The ego collisions were calculated by comparing an ego vehicle’s location at the indicated collision versus its previously known location where it indicated a collision, as previously mentioned. Given the possibility of false positive collisions, to verify the accuracy of a collision, there was implemented a function which would give us a count of confirmed collisions. A collision would be confirmed if it fulfilled three criteria.

1. The registered collision is more than 5 simulated seconds since last one

2. The registered collision is between 3 and 5 meters of the registered collision object

3. The registered collision is more than 5 meters away from the previously registered collision

Collisions that fulfill these criteria are registered as confirmed collisions and are used in the fitness calculation. Lastly, the DTV value of an ego vehicle would either penalize or reward the individual depending on its value.

DTV Points

0 10

10 0

20 -10

30 - 20

Table 4.2: Scoring of an individual Ego DTV

Conforming to the set default setting in the simulation, a DTV of 10 would equal to a reward of 0 points. Whereas a DTV of would reward a

(38)

total of 10 points and a DTV of 20 would penalize the individual with -10 points. This part of the fitness calculation has the goal of narrowing down the DTV parameter of the PID controller. As a lower value, here indicates a shorter indicated braking distance for the vehicle in question. Which in turn will increase the danger of scenarios. Additionally, the ability to draw a conclusion from the ego vehicle performance regarding safety is a challenge of its own [57]. As such, while the metrics chosen have the ability to represent dangerous behavior, they are by no means a uniform representation of scenario danger.

4.2 Scenario groups

The ScenarioRunner module comes with predefined scenarios which are separated into scenario groups. Each scenario is unique in its composition, meaning that while the overall theme of a scenario group is the same, each scenario is executed on different parts of a map with different roads, lanes, and junctions. With this in mind, we can test the proposed approach and its effectiveness and differences between groups. Keeping in mind that we’d like to keep some differences between groups while also not being completely opposite, two scenario groups were chosen.

• Group 1: FollowLeadingVehicle (FLV)

• Group 2: FollowLeadingVehicleWithObstacle (FLVWO) 4.2.1 Group 1: FollowLeadingVehicle

Figure 4.3: Town01 of scenario_1 inFollowLeadingVehicle

This group was chosen to represent a relatively safe environment for the ego vehicle. The group consists of a total of 11 scenarios, which all occur at different locations and different maps. As such, an individual instance of the ego vehicle may partake in anyN number of scenarios in

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to Switching between Different Scenarios

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to

Switching between Different Scenarios

A Search-based Approach

Aleksander Rødseth Kjelstrup

Thesis submitted for the degree of

Master in Programming and System Architecture:

Software 60 credits

Department of Informatics

Faculty of Mathematics and Natural Sciences

UNIVERSITY OF OSLO

Revealing Dangerous Behaviors in Self-Driving Vehicles Due to

Switching between Different Scenarios

A Search-based Approach

Aleksander Rødseth Kjelstrup

Abstract

Contents

List of Figures

List of Tables

Preface

Chapter 1

Introduction

1.1 Motivation

1.2 Planning the project and initial work

1.3 Problem statement

1.4 Research questions

1.5 Contribution of the work

Chapter 2

Background

2.1 Self-driving vehicles

2.2 Urban deployment of autonomous vehicles

2.3 Genetic Algorithm

2.4 Simulation software

Chapter 3

Literature review

3.1 Testing practices

3.2 Other works

Chapter 4

Program flow

4.1 Project code

4.2 Scenario groups