As data in real-world applications continuously evolve, the ability of artificial intelligence systems to learn incrementally while preserving previously acquired knowledge has become increasingly critical. However, deploying continual learning (CL) methods in practice is impeded by blurred task boundaries, severe data imbalance, and the high computational demands and data privacy concerns associated with large models. This thesis addresses these challenges through three core contributions, thus enhancing the feasibility and robustness of CL in dynamic environments. First, to manage blurred task boundaries, where data distributions often overlap, we propose a novel Distribution-Shift Incremental Learning (DS-IL) scenario. In this framework, an entropy-guided learning approach effectively leverages these overlaps to mitigate catastrophic forgetting without maintaining large memory buffers. In real-world scenarios, data imbalance is a common challenge that can significantly hinder the performance of learning systems. To address this issue, our second contribution introduces a Memory Selection and Contrastive Learning (MSCL) strategy. By actively sampling representative instances and coupling them with current data in a contrastive loss, the model better balances underrepresented classes and domains. This approach not only preserves crucial historical information but also maintains robust performance under significantly skewed data distributions. Finally, to alleviate the computational overhead of continually training diffusion models, particularly relevant in scenarios with data privacy constraints or prohibitive storage costs, we introduce a Multi-Mode Adaptive Generative Distillation (MAGD) framework. Using generative distillation, noisy intermediate representations, and exponential moving averages, this method enables efficient continual updates while preserving high-quality image generation and classification performance. Collectively, these contributions form a comprehensive framework for scalable, memory-efficient, and computationally tractable continual learning. Through effective knowledge retention, dynamic adaptation to imbalanced data, and resource-efficient generative replay, this thesis expands the applicability of CL methods to a wider range of real-world settings.