The objective of the thesis is the development of the system's lifelong learning of ethical behaviors, within controlled boundaries, integrated in a co-construction process with human feedback to guide the learning. The learning system will be composed of several interacting agents [5]: each agent is responsible for controlling/recommending actions based on moral preferences of the user with which it interacts ; and each agent also interacts with the other agents of the system (e.g. those interacting with other users of the transportation system). Thus the thesis focuses on the combination of multi-objective and multi-agent reinforcement learning (MOMARL), to consider the decision of other learning agents and the multiple moral values (objectives) of the users.

Very few works have tackled both MORL and MARL [6] and no work in the Machine Ethics field considers multi-objective, multi-agent [6-7], and human-centered approach [16]. The main contribution of the thesis would be to propose a multi-objective multi-agent learning algorithm, able to identify sets of optimal policies, considering different trade-offs for the conflicting objectives and multiple agents, while operating within specified boundaries. A proof-of-concept approach, developed in a previous project [4], allows an artificial agent and a human user co-identifying conflicting objectives and possible trade-offs. A first approach in this thesis could be to extend this work to consider multiple agents in the same environment, and address trade-offs that could involve more than one agent, by proposing joint actions instead of unilateral actions. Most MARL approaches [8-9] require sharing information with other agents (Centralized Learning Decentralized Execution paradigm), which impairs privacy. In addition to the `by design’ ethical considerations, by improving the number and the quality of the found trade-offs with a joint policy, the contribution will have to abide by `in design’ ethical considerations. One such important aspect will be to preserve privacy during the data sharing among agents. For this part, intrinsically motivated social learning [13-14] will be considered.

Another aspect concerns the “dilemmas” situations, where several moral values are in conflict, and no single decision allows satisfying all of them at the same time: each choice will lead to regret. We argue that these situations cannot be “autonomously” settled by machines only, at least not how humans would like (expect) them to be settled. Thus another contribution of the thesis will be to propose an intelligible MOMARL approach taking into account several (more than three) objectives, and to be able to identify and settle situations of dilemmas, especially those requiring human intervention. First, as the number of “dilemmas” situations could be too high to be efficiently presented to end users, an exploration guided process based on intrinsically motivated reinforcement learning (e.g., curiosity models, learning progress,...) [10] will be investigated. Then, to allow the intelligible presentation of the alternatives to users, a refinement process will also be studied to classify dilemmas and involve the users through non-invasive HCI. We also propose to leverage human preferences to decide how to settle some dilemmas. To this end, the system must be able to use human feedback as a reward [11] or to learn models of users preferences, investigating approaches that learn preferences/profiles with few/no a priori data [12] and then adapt them through non-invasive HCI.