1.Background
In recent years, with the rapid popularization of mobile smart devices and the widespread application of IoT devices, the problem of device energy consumption and thermal management has become an urgent and important challenge to be solved. Dynamic voltage frequency scaling (DVFS) technology, as a key method for controlling power consumption and heat, balances performance and energy consumption by adjusting the operating frequency of the device's processor. However, traditional DVFS methods usually rely on fixed rules or heuristic strategies, which are difficult to adapt to complex and ever-changing operating scenarios. Reinforcement learning, as a technology that can perform optimized decision-making through interactive learning, has shown great potential in DVFS problems. However, current reinforcement learning-based frequency scaling algorithms generally have significant insufficient generalization problems. Specifically, these algorithms need to be retrained for different hardware devices and operating scenarios, resulting in high development and deployment costs, making it difficult to meet actual application needs. Therefore, this project focuses on building reinforcement learning frequency scaling algorithms with stronger generalization capabilities, hoping to achieve rapid migration and efficient adaptation of algorithms in multiple scenarios and devices, and promote the intelligent development of edge devices in energy consumption management.
2.Research Module Description
2.1 Network Structure Search with Generalization
In response to the insufficient generalization ability of reinforcement learning algorithms, this module introduces neural architecture search (NAS) technology to explore neural network structures with good generalization ability. NAS technology, through an automated design process combined with reinforcement learning or evolutionary algorithms, efficiently searches for the optimal network structure in a vast architecture space, greatly reducing the design cost of manual intervention. In this project, the application focus of NAS technology is to find a general network structure that can handle both device diversity and cope with the complexity of scenarios.
Specifically, first, define a search space containing multiple potential network structures, which can cover mainstream model structures such as convolutional neural networks, recurrent neural networks, and Transformers, while introducing lightweight design strategies (such as pruning, quantization, etc.) to adapt to the limitations of edge devices on computing resources. Secondly, design efficient search strategies, such as reinforcement learning-based proxy methods or gradient-based differentiable NAS methods, and find architectures with good performance in different scenarios by evaluating the performance of network structures in multiple virtual hardware scenarios. Finally, the searched structure is offline trained and verified, and its performance on unseen devices and scenarios is analyzed through generalization tests to ensure that it meets actual application needs.
Through the above process, this module builds a more adaptable and scalable network structure for the frequency scaling algorithm, laying a solid foundation for subsequent algorithm optimization and migration.
2.2 Meta-Learning-Based Initial Model Optimization
In the actual application of reinforcement learning frequency scaling algorithms, each new device or new scenario may exhibit different characteristics, making it difficult for the original model to directly adapt. Therefore, this module introduces the model-agnostic meta-learning algorithm (MAML) in meta-learning technology, and optimizes the initial model parameters, so that the model has the ability to quickly adapt to new tasks.
The core idea of the MAML algorithm is to train a general initial model so that it can quickly adapt to new tasks through a small amount of data and gradient updates. In this project, the application process of MAML mainly includes the following steps: First, build a task set based on multiple hardware devices and scenarios. These tasks cover the diversity of device processing capabilities, load conditions, and environmental factors. Then, in the training phase, the MAML algorithm is used to learn initial parameters on these task sets, so that the model adapts to each task faster and performs better. MAML achieves parameter adjustment through a two-layer optimization process: the outer layer optimization is responsible for adjusting global initial parameters to improve overall adaptability, and the inner layer optimization fine-tunes the parameters based on specific tasks to improve task performance.
In order to further enhance the generalization of the model, this module also combines technologies such as data augmentation and adversarial training to simulate various interference factors in real scenarios and improve the robustness and migration ability of the model. The initial model optimized by MAML does not need to be trained from scratch in new devices or scenarios, and only a small amount of data is needed to complete rapid adaptation, which greatly reduces deployment costs and time.
2.3 Online Model Update Based on TFLite
Although the research in the first two parts can significantly improve the initial generalization ability of the algorithm, in practical applications, the operating environment of edge devices may change dynamically, such as fluctuations in device load, the introduction of new applications, etc., making it possible for the model to need further adaptive optimization during operation. Therefore, this module introduces the online training function of TensorFlow Lite (TFLite) to realize real-time model updates on the edge device side.
The implementation process of online update mainly includes the following stages. First, integrate a lightweight model training framework on the device side, and perform efficient incremental training through the support of TFLite. During the operation of the device, real-time data is continuously collected, such as the current processor utilization, frequency settings, power consumption, frame rate, etc., as training samples for the reinforcement learning algorithm. Secondly, based on the collected data, the model parameters are incrementally updated using pre-designed optimization strategies. In this process, distributed training or progressive learning techniques are used to reduce the impact of online updates on device performance. Finally, by continuously monitoring the model performance, the training frequency and update strategy are dynamically adjusted to ensure the stability and efficiency of online training.
In addition, in order to prevent overfitting or catastrophic forgetting problems during the online update process, this module designs a rollback mechanism to restore to the previous version when the model performance significantly decreases. At the same time, by introducing model compression techniques, such as weight pruning and quantization, the resource occupation of online training is further reduced, ensuring the practical feasibility of online updates. Through TFLite's online update capability, this module provides the algorithm with the ability to dynamically adapt to environmental changes, enhancing the long-term stability and applicability of the model.
3.Summary
This project focuses on the generalization problem of reinforcement learning frequency scaling algorithms, and proposes a highly adaptive frequency scaling solution for edge devices by combining neural architecture search, meta-learning, and online training. Neural architecture search technology provides the algorithm with a lightweight and general network structure, meta-learning optimizes the initial parameters of the model, enabling it to quickly adapt to new tasks, and online training technology realizes the adaptive improvement of the algorithm in the actual operating environment. The organic combination of the three not only improves the generalization performance of the algorithm, but also provides theoretical and practical support for its efficient deployment in diverse edge devices. This research is of great significance to the application of reinforcement learning in the field of energy consumption control and thermal management, and also provides new ideas for solving the problem of generalization.