Résultats de recherche pour « Offline Reinforcement Learning »

Abstract Conventional online reinforcement learning (RL) systems interact with their environments to gather data, aiming to develop an optimal policy that maximizes a predefined cumulative reward. Nevertheless, the applicability of online RL is limited under conditions where cost and safety are critical concerns. Offline RL emerges as a solution by leveraging previously collected datasets to derive an effective policy without further interacting with the environment. A significant hurdle in offline RL is the propensity to overestimate the values of actions not represented in the data (out-of-distribution, or OOD actions), often due to insufficient exploration of the state-action space. Prior works tend to add to the algorithm complexity to improve the performance, however, this paper halves the algorithm computation as well as gaining performance improvement.In this paper, we delve into the inaccuracies associated with isolated critic training, devoid of policy enhancement typically driven by actor training. We hypothesize that the dataset's inherent Q-values (behavior Q-values) may be more accurate than those derived from actor-critic cycles, which are susceptible to overestimation and volatility.Upon validating the dataset Q-values' reliability, we propose pre-training models to capture these values before commencing offline training, potentially streamlining algorithmic efficiency across various models and reducing computational demands.Moreover, by harnessing precise dataset Q-value estimations, we advocate for a conservative approach to offline training, which demonstrably mitigates value overestimation.Our methodologies have the potential to be generalized among offline RL and may show a new perspective on the ways of optimizing the offline learning.

Reinforcement learning (RL) has been a promising essence in future 5G-beyond and 6G systems. Its main advantage lies in its robust model-free decision-making in complex and large-dimension wireless environments. However, most existing RL frameworks rely on online interaction with the environment, which might not be feasible due to safety and cost concerns. Another problem with online RL is the lack of scalability of the designed algorithm with dynamic or new environments. This work proposes a novel, resilient, few-shot meta-offline RL algorithm combining offline RL using conservative Q-learning (CQL) and meta-learning using model-agnostic meta-learning (MAML). The proposed algorithm can train RL models using static offline datasets without any online interaction with the environments. In addition, with the aid of MAML, the proposed model can be scaled up to new unseen environments. We showcase the proposed algorithm for optimizing an unmanned aerial vehicle (UAV) 's trajectory and scheduling policy to minimize the age-of-information (AoI) and transmission power of limitedpower devices. Numerical results show that the proposed fewshot meta-offline RL algorithm converges faster than baseline schemes, such as deep Q-networks and CQL. In addition, it is the only algorithm that can achieve optimal joint AoI and transmission power using an offline dataset with few shots of data points and is resilient to network failures due to unprecedented environmental changes.

Abstract Bus bunching is a self-reinforcing phenomenon in which a delayed bus boards more passengers, incurs further delay, and is caught by its follower. Holding a bus at a stop to regulate headways is a widely studied corrective measure, yet most evaluations rely on a single shaped objective rather than independent passenger and service outcomes. This study asks whether a fixed operational log can support network-wide holding policies without simulator interaction, and whether offline-to-online fine-tuning yields sufficient improvement to justify its computational cost. We constructed a microscopic Simulation of Urban Mobility (SUMO) benchmark with 12 Changsha bus lines and 3.37 million logged holding-only transitions. We compared five transportation rules and three offline learners: behavior cloning (BC), conservative Q-learning (CQL), and robust ensemble Soft Actor-Critic (RE-SAC). We also compared online Soft Actor-Critic (SAC) with warm-start reinforcement learning (WSRL) and reinforcement learning with prior data (RLPD), two offline-to-online methods, under separate interaction budgets. Every reported checkpoint was evaluated across ten common 18 000-s SUMO episodes that ran to natural completion. WSRL achieved the highest selected-checkpoint return (−710 544 ± 17 162). The mean passenger waiting time was 301 ± 4 s and the mean total travel time was 1 266 ± 18 s. However, no single controller was best on every passenger and service outcome, underscoring the need for evaluation beyond shaped return. The study contributes a reproducible benchmark and a comparison design with explicitly separated interaction budgets. The findings are limited to the simulated setting, as observed operating trajectories were unavailable for external calibration.

Offline reinforcement learning (RL) learns policies for solving sequential decision problems directly from offline datasets. Most existing works focus on countering out-of-distribution (OOD) behavior to improve decision-making. This work investigates the fundamental Bellman inconsistency problem that leads to suboptimal decision-making, which is an essential factor causing suboptimality in offline settings. An offline RL algorithm without regularization and pessimism (RFORL) is proposed. The RFORL algorithm constrains uncertainties constructed by the inconsistency among Bellman estimations under an ensemble of learned Qfunctions. Compared to existing offline RL methods, the proposed RFORL achieves exponential convergence and allows learning the best policy from the given offline dataset, without adopting regularization techniques or pessimism. We empirically demonstrate that the Bellman inconsistency constraint mechanism is effective in improving policy learning. Consequently, RFORL outperforms existing offline RL approaches on most tasks of the D4RL benchmark.