For an agent, it is very challenging to autonomously elaborate and make use of a visual representation of its open environment. It is necessary to introduce new categories for unseen objects, and recognize already seen objects to refine and make evolve the representation, which is even more difficult when undergoing uncertainty and variability under uncontrolled operation conditions. The autonomy of the agent in classification, its adaptation to a changing environment as well as continual representation building are important properties of such a dynamic machine learning system. However, in some of the existing systems, especially with neural networks, the acquired representation tends to gradually drift away from initial observations which causes catastrophic forgetting. Another challenge comes from the decision whether incoming observations are new or belong to already seen objects. One has to detect novelty and introduce a new concept or class if necessary, and maintain the already acquired representation, i.e. recognize the object and make use of new observations appropriately, which is part of the more general problem of finding a meaningful and robust representation under the stability-plasticity dilemma.

In this thesis, we address these challenges by adopting a state-of-the-art continual unsupervised representation learning model based on a specific Variational Auto-Encoder (VAE) architecture that we extended and applied on sequences of images showing different objects. Assuming that, in a realistic environment, there is some continuity in the sequence of object observations, the first contribution is an algorithm that robustly detects when object class changes occur in the image stream based on the dynamic evolution of the observation likelihood. It is shown that the thus obtained clusters in the representation space are more consistent with real objects and therefore facilitate the classification of known objects. The second contribution replaces the first approach and further increases the autonomy of the model by introducing an algorithm based on the Hotelling t-squared statistical hypothesis test that is able to continuously detect class changes and simultaneously decides if the currently observed object is novel or already present in the learnt representation. Extensive experimental results on several image benchmarks show that this self supervised continual learning approach is able to build representations that favor the autonomy of the agent by minimising supervision while maintaining a high object recognition accuracy compared to the state of the art.